Compare commits

...

42 commits

Author SHA1 Message Date
Norm 1414e709e3 Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-06-17 21:35:18 -04:00
Norm b9e7b100a2 Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-06-12 19:52:57 -04:00
Floatingghost a8243d2d51 Convert rich media backfill to oban task 2024-06-11 13:16:10 -04:00
Norm a41545a781 Merge branch 'pool-timeouts' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-06-10 10:17:55 -04:00
Norm c7b4e34cf9 Merge branch 'websocket_fullsweep' of https://akkoma.dev/Oneric/akkoma into akko.wtf 2024-06-09 20:09:51 -04:00
Norm e3f90afc62 Revert "TEMP: add cleanup worker logging"
This reverts commit e41d35180e.

No longer needed anymore.
2024-06-09 16:57:04 -04:00
Norm f02d4d094d Merge branch 'pool-timeouts' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-06-09 16:28:45 -04:00
Norm 7c2c11fdd8 Merge branch 'pool-timeouts' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-06-09 15:33:03 -04:00
Norm baee4acde1 Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-06-07 14:23:58 -04:00
Norm cdf05077fe Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-06-01 23:27:34 -04:00
Norm deb64d113e Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-05-31 11:09:14 -04:00
Norm e41d35180e TEMP: add cleanup worker logging 2024-05-30 21:08:25 -04:00
Norm fe8395c2cd Revert "TEMP: add logging for cleanup worker"
This reverts commit d89d189bd3.
2024-05-30 20:47:04 -04:00
Norm d89d189bd3 TEMP: add logging for cleanup worker 2024-05-30 20:46:05 -04:00
Norm d6592053e9 Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-05-27 23:32:37 -04:00
Norm 387c368b8c Fix Exiftool stderr being read as an image description
Fixes: #773
2024-05-23 14:40:25 -04:00
Norm 6288682173 Pull security updates from upstream develop 2024-05-22 15:00:18 -04:00
Norm f5a8f7ba2d Merge remote-tracking branch 'oneric/prune-batch' into akko.wtf 2024-05-14 22:56:23 -04:00
Oneric c127d48308 dbprune/activites: prune array activities first
This query is less costly; if something goes wrong or gets aborted later
at least this part will arelady be done.
2024-05-15 03:45:50 +02:00
Oneric 40ae91a45c dbprune: allow splitting array and single activity prunes
The former is typically just a few reports; it doesn't make sense to
rerun it over and over again in batched prunes or if a full prune OOMed.
2024-05-15 03:45:50 +02:00
Oneric 3c319ea732 dbprune: use query! 2024-05-15 01:43:53 +02:00
Oneric 91e4f4f885 dbprune: add more logs
Pruning can go on for a long time; give admins some insight into that
something is happening to make it less frustrating and to make it easier
which part of the process is stalled should this happen.

Again most of the changes are merely reindents;
review with whitespace changes hidden recommended.
2024-05-15 01:43:53 +02:00
Oneric 7e03886886 dbprune: shortcut array activity search
This brought down query costs from 7,953,740.90 to 47,600.97
2024-05-15 01:43:53 +02:00
Oneric 1caac640da Test both standalone and flag mode for pruning orphaned activities 2024-05-15 01:43:53 +02:00
Oneric b03947917a Also allow limiting the initial prune_object
May sometimes be helpful to get more predictable runtime
than just with an age-based limit.

The subquery for the non-keep-threads path is required
since delte_all does not directly accept limit().

Again most of the diff is just adjusting indentation, best
hide whitespace-only changes with git diff -w or similar.
2024-05-15 01:43:53 +02:00
Oneric 3258842d0c Log number of deleted rows in prune_orphaned_activities
This gives feedback when to stop rerunning limited batches.

Most of the diff is just adjusting indentation; best reviewed
with whitespace-only changes hidden, e.g. `git diff -w`.
2024-05-14 23:45:10 +02:00
Oneric ff684ba8ea Add standalone prune_orphaned_activities CLI task
This part of pruning can be very expensive and bog down the whole
instance to an unusable sate for a long time. It can thus be desireable
to split it from prune_objects and run it on its own in smaller limited batches.

If the batches are smaller enough and spaced out a bit, it may even be possible
to avoid any downtime. If not, the limit can still help to at least make the
downtime duration somewhat more predictable.
2024-05-14 23:45:10 +02:00
Oneric f5b5838c4d refactor: move prune_orphaned_activities into own function
No logic changes. Preparation for standalone orphan pruning.
2024-05-14 23:45:10 +02:00
Norm 2007b1c586 Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-05-11 14:47:40 -04:00
Norm 5a90aa50f1 Revert "temp add logging for collection fetching"
This reverts commit 9486abca22.
2024-04-29 13:51:09 -04:00
Norm 36f2422650 Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-04-27 11:33:02 -04:00
Norm 6ed176ba45 Merge remote-tracking branch 'upstream/develop' into akko.wtf 2024-04-20 03:03:33 -04:00
Norm 9486abca22 temp add logging for collection fetching 2024-04-06 11:37:09 -04:00
Norm 1a3624f45f Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-04-06 11:36:42 -04:00
Oneric 4f44d08816 [TEST] Avoid accumulating stale data in websockets
For some but not all instances (likely depending on usage patterns)
those [i’m guessing, to be tested] process end up accumulating
stale binary data in such a way it’s not included into young garbage
collection cycles. At the same time, full cycles are barely ever
triggered making it seem like a memory leak.
To avoid this, make full sweeps more frequent for only the affected
processes.

TODO: actually test this theory + fix

ref:
  https://www.erlang.org/doc/man/erlang#ghlink-process_flag-2-idp226
  https://blog.guzman.codes/using-phoenix-channels-high-memory-usage-save-money-with-erlfullsweepafter
    (showed up in search results and inspired this)
  https://git.pleroma.social/pleroma/pleroma/-/merge_requests/4060
    (different patch due to different socket implementation)
2024-04-04 17:32:05 +02:00
Norm 47896ae170 Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-03-30 11:01:48 -04:00
Norm c648f4af9d Merge remote-tracking branch 'upstream/develop' into akko.wtf 2024-02-24 15:40:23 +00:00
Oneric bb327870f7 Test both standalone and flag mode for pruning orphaned activities 2024-02-20 19:32:01 -05:00
Oneric 4fcf2cbf85 Also allow limiting the initial prune_object
May sometimes be helpful to get more predictable runtime
than just with an age-based limit.

The subquery for the non-keep-threads path is required
since delte_all does not directly accept limit().

Again most of the diff is just adjusting indentation, best
hide whitespace-only changes with git diff -w or similar.
2024-02-20 19:32:01 -05:00
Oneric 92e6839d46 Log number of deleted rows in prune_orphaned_activities
This gives feedback when to stop rerunning limited batches.

Most of the diff is just adjusting indentation; best reviewed
with whitespace-only changes hidden, e.g. `git diff -w`.
2024-02-20 19:32:01 -05:00
Oneric c4923b6ed8 Add standalone prune_orphaned_activities CLI task
This part of pruning can be very expensive and bog down the whole
instance to an unusable sate for a long time. It can thus be desireable
to split it from prune_objects and run it on its own in smaller limited batches.

If the batches are smaller enough and spaced out a bit, it may even be possible
to avoid any downtime. If not, the limit can still help to at least make the
downtime duration somewhat more predictable.
2024-02-20 19:32:01 -05:00
Oneric ba14196856 refactor: move prune_orphaned_activities into own function
No logic changes. Preparation for standalone orphan pruning.
2024-02-20 19:32:01 -05:00

View file

@ -18,6 +18,8 @@ defmodule Pleroma.Web.MastodonAPI.WebsocketHandler do
@timeout :timer.seconds(60)
# Hibernate every X messages
@hibernate_every 100
# Tune garabge collect for long-lived websocket process
@fullsweep_after 20
def init(%{qs: qs} = req, state) do
with params <- Enum.into(:cow_qs.parse_qs(qs), %{}),
@ -59,6 +61,10 @@ defmodule Pleroma.Web.MastodonAPI.WebsocketHandler do
"#{__MODULE__} accepted websocket connection for user #{(state.user || %{id: "anonymous"}).id}, topic #{state.topic}"
)
# process is long-lived and can sometimes accumulate stale data in such a way it's
# not freed by young garbage cycles, thus make full collection sweeps more frequent
:erlang.process_flag(:fullsweep_after, @fullsweep_after)
Streamer.add_socket(state.topic, state.oauth_token)
{:ok, %{state | timer: timer()}}
end