Compare commits

..

54 commits

Author SHA1 Message Date
67f3ee45b4 Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-10-31 19:01:11 -04:00
ef22f3a92b Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-10-28 19:39:50 -04:00
cb7f6fb3d6 Update hashtag prune to account for followed hashtags 2024-10-25 11:09:20 -04:00
370894d153 Merge remote-tracking branch 'oneric/custom-source' into akko.wtf 2024-10-20 01:07:57 -04:00
7a16e12029 Update the nsfwCensorImage fix from the update-nsfwCensorImage PR 2024-10-20 01:07:01 -04:00
e4332d06b5 update nsfwCensorImage suggestion in config/description.exs
Some checks are pending
ci/woodpecker/pr/build-amd64 Pipeline is pending
ci/woodpecker/pr/build-arm64 Pipeline is pending
ci/woodpecker/pr/docs Pipeline is pending
ci/woodpecker/pr/lint Pipeline is pending
ci/woodpecker/pr/test Pipeline is pending
Turns out this is also used to set the default values in adminfe.
However, this URL may break with newer Akkoma-FE versions.
Instead, set this to blank so that it falls back to the default
NSFW cover image set at build time on Akkoma-FE.
2024-10-20 00:59:52 -04:00
f98d5f3f20 Teach admin-fe about custom source URLs
Some checks are pending
ci/woodpecker/pr/build-amd64 Pipeline is pending
ci/woodpecker/pr/build-arm64 Pipeline is pending
ci/woodpecker/pr/docs Pipeline is pending
ci/woodpecker/pr/lint Pipeline is pending
ci/woodpecker/pr/test Pipeline is pending
Matching  AkkomaGang/akkoma-fe#421
2024-10-18 00:47:43 +02:00
cd151999cb update nsfwCensorImage suggestion in config/description.exs
Turns out this is also used to set the default values in adminfe, so
update this to the correct URL for the NSFW banner so that it doesn't
break if you reset this value.
2024-10-17 02:39:14 -04:00
345e3ed04c Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-10-16 20:52:11 -04:00
17a0b6c546 Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-10-13 22:25:24 -04:00
b15698a772 Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-09-15 19:19:04 -04:00
d6ac8cb457 Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-06-23 15:02:59 -04:00
1414e709e3 Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-06-17 21:35:18 -04:00
b9e7b100a2 Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-06-12 19:52:57 -04:00
a8243d2d51 Convert rich media backfill to oban task 2024-06-11 13:16:10 -04:00
a41545a781 Merge branch 'pool-timeouts' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-06-10 10:17:55 -04:00
c7b4e34cf9 Merge branch 'websocket_fullsweep' of https://akkoma.dev/Oneric/akkoma into akko.wtf 2024-06-09 20:09:51 -04:00
e3f90afc62 Revert "TEMP: add cleanup worker logging"
This reverts commit e41d35180e.

No longer needed anymore.
2024-06-09 16:57:04 -04:00
f02d4d094d Merge branch 'pool-timeouts' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-06-09 16:28:45 -04:00
7c2c11fdd8 Merge branch 'pool-timeouts' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-06-09 15:33:03 -04:00
baee4acde1 Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-06-07 14:23:58 -04:00
cdf05077fe Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-06-01 23:27:34 -04:00
deb64d113e Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-05-31 11:09:14 -04:00
e41d35180e TEMP: add cleanup worker logging 2024-05-30 21:08:25 -04:00
fe8395c2cd Revert "TEMP: add logging for cleanup worker"
This reverts commit d89d189bd3.
2024-05-30 20:47:04 -04:00
d89d189bd3 TEMP: add logging for cleanup worker 2024-05-30 20:46:05 -04:00
d6592053e9 Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-05-27 23:32:37 -04:00
387c368b8c Fix Exiftool stderr being read as an image description
Fixes: #773
2024-05-23 14:40:25 -04:00
6288682173 Pull security updates from upstream develop 2024-05-22 15:00:18 -04:00
f5a8f7ba2d Merge remote-tracking branch 'oneric/prune-batch' into akko.wtf 2024-05-14 22:56:23 -04:00
c127d48308 dbprune/activites: prune array activities first
Some checks failed
ci/woodpecker/pr/test Pipeline is pending
ci/woodpecker/pr/build-arm64 Pipeline is pending
ci/woodpecker/pr/lint Pipeline failed
ci/woodpecker/pr/test/1 unknown status
ci/woodpecker/pr/test/2 unknown status
ci/woodpecker/pr/test/3 unknown status
ci/woodpecker/pr/test/4 unknown status
ci/woodpecker/pr/build-amd64 unknown status
ci/woodpecker/pr/docs unknown status
This query is less costly; if something goes wrong or gets aborted later
at least this part will arelady be done.
2024-05-15 03:45:50 +02:00
40ae91a45c dbprune: allow splitting array and single activity prunes
The former is typically just a few reports; it doesn't make sense to
rerun it over and over again in batched prunes or if a full prune OOMed.
2024-05-15 03:45:50 +02:00
3c319ea732 dbprune: use query! 2024-05-15 01:43:53 +02:00
91e4f4f885 dbprune: add more logs
Pruning can go on for a long time; give admins some insight into that
something is happening to make it less frustrating and to make it easier
which part of the process is stalled should this happen.

Again most of the changes are merely reindents;
review with whitespace changes hidden recommended.
2024-05-15 01:43:53 +02:00
7e03886886 dbprune: shortcut array activity search
This brought down query costs from 7,953,740.90 to 47,600.97
2024-05-15 01:43:53 +02:00
1caac640da Test both standalone and flag mode for pruning orphaned activities 2024-05-15 01:43:53 +02:00
b03947917a Also allow limiting the initial prune_object
May sometimes be helpful to get more predictable runtime
than just with an age-based limit.

The subquery for the non-keep-threads path is required
since delte_all does not directly accept limit().

Again most of the diff is just adjusting indentation, best
hide whitespace-only changes with git diff -w or similar.
2024-05-15 01:43:53 +02:00
3258842d0c Log number of deleted rows in prune_orphaned_activities
This gives feedback when to stop rerunning limited batches.

Most of the diff is just adjusting indentation; best reviewed
with whitespace-only changes hidden, e.g. `git diff -w`.
2024-05-14 23:45:10 +02:00
ff684ba8ea Add standalone prune_orphaned_activities CLI task
This part of pruning can be very expensive and bog down the whole
instance to an unusable sate for a long time. It can thus be desireable
to split it from prune_objects and run it on its own in smaller limited batches.

If the batches are smaller enough and spaced out a bit, it may even be possible
to avoid any downtime. If not, the limit can still help to at least make the
downtime duration somewhat more predictable.
2024-05-14 23:45:10 +02:00
f5b5838c4d refactor: move prune_orphaned_activities into own function
No logic changes. Preparation for standalone orphan pruning.
2024-05-14 23:45:10 +02:00
2007b1c586 Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-05-11 14:47:40 -04:00
5a90aa50f1 Revert "temp add logging for collection fetching"
This reverts commit 9486abca22.
2024-04-29 13:51:09 -04:00
36f2422650 Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-04-27 11:33:02 -04:00
6ed176ba45 Merge remote-tracking branch 'upstream/develop' into akko.wtf 2024-04-20 03:03:33 -04:00
9486abca22 temp add logging for collection fetching 2024-04-06 11:37:09 -04:00
1a3624f45f Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-04-06 11:36:42 -04:00
4f44d08816 [TEST] Avoid accumulating stale data in websockets
For some but not all instances (likely depending on usage patterns)
those [i’m guessing, to be tested] process end up accumulating
stale binary data in such a way it’s not included into young garbage
collection cycles. At the same time, full cycles are barely ever
triggered making it seem like a memory leak.
To avoid this, make full sweeps more frequent for only the affected
processes.

TODO: actually test this theory + fix

ref:
  https://www.erlang.org/doc/man/erlang#ghlink-process_flag-2-idp226
  https://blog.guzman.codes/using-phoenix-channels-high-memory-usage-save-money-with-erlfullsweepafter
    (showed up in search results and inspired this)
  https://git.pleroma.social/pleroma/pleroma/-/merge_requests/4060
    (different patch due to different socket implementation)
2024-04-04 17:32:05 +02:00
47896ae170 Merge branch 'develop' of https://akkoma.dev/AkkomaGang/akkoma into akko.wtf 2024-03-30 11:01:48 -04:00
c648f4af9d Merge remote-tracking branch 'upstream/develop' into akko.wtf 2024-02-24 15:40:23 +00:00
bb327870f7 Test both standalone and flag mode for pruning orphaned activities 2024-02-20 19:32:01 -05:00
4fcf2cbf85 Also allow limiting the initial prune_object
May sometimes be helpful to get more predictable runtime
than just with an age-based limit.

The subquery for the non-keep-threads path is required
since delte_all does not directly accept limit().

Again most of the diff is just adjusting indentation, best
hide whitespace-only changes with git diff -w or similar.
2024-02-20 19:32:01 -05:00
92e6839d46 Log number of deleted rows in prune_orphaned_activities
This gives feedback when to stop rerunning limited batches.

Most of the diff is just adjusting indentation; best reviewed
with whitespace-only changes hidden, e.g. `git diff -w`.
2024-02-20 19:32:01 -05:00
c4923b6ed8 Add standalone prune_orphaned_activities CLI task
This part of pruning can be very expensive and bog down the whole
instance to an unusable sate for a long time. It can thus be desireable
to split it from prune_objects and run it on its own in smaller limited batches.

If the batches are smaller enough and spaced out a bit, it may even be possible
to avoid any downtime. If not, the limit can still help to at least make the
downtime duration somewhat more predictable.
2024-02-20 19:32:01 -05:00
ba14196856 refactor: move prune_orphaned_activities into own function
No logic changes. Preparation for standalone orphan pruning.
2024-02-20 19:32:01 -05:00
2 changed files with 21 additions and 4 deletions

View file

@ -1184,7 +1184,7 @@
logoMask: true, logoMask: true,
minimalScopesMode: false, minimalScopesMode: false,
noAttachmentLinks: false, noAttachmentLinks: false,
nsfwCensorImage: "/static/img/nsfw.74818f9.png", nsfwCensorImage: "",
postContentType: "text/plain", postContentType: "text/plain",
redirectRootLogin: "/main/friends", redirectRootLogin: "/main/friends",
redirectRootNoLogin: "/main/all", redirectRootNoLogin: "/main/all",
@ -1194,7 +1194,9 @@
showInstanceSpecificPanel: false, showInstanceSpecificPanel: false,
subjectLineBehavior: "email", subjectLineBehavior: "email",
theme: "pleroma-dark", theme: "pleroma-dark",
webPushNotifications: false webPushNotifications: false,
backendCommitUrl: "",
frontendCommitUrl: ""
} }
], ],
children: [ children: [
@ -1285,7 +1287,7 @@
type: {:string, :image}, type: {:string, :image},
description: description:
"URL of the image to use for hiding NSFW media attachments in the timeline", "URL of the image to use for hiding NSFW media attachments in the timeline",
suggestions: ["/static/img/nsfw.74818f9.png"] suggestions: [""]
}, },
%{ %{
key: :postContentType, key: :postContentType,
@ -1398,6 +1400,18 @@
label: "Stop Gifs", label: "Stop Gifs",
type: :boolean, type: :boolean,
description: "Whether to pause animated images until they're hovered on" description: "Whether to pause animated images until they're hovered on"
},
%{
key: :backendCommitUrl,
label: "Backend Commit URL",
type: :string,
description: "URL prefix for backend commit hashes"
},
%{
key: :frontendCommitUrl,
label: "Frontend Commit URL",
type: :string,
description: "URL prefix for frontend commit hashes"
} }
] ]
}, },

View file

@ -346,7 +346,10 @@ def run(["prune_objects" | args]) do
DELETE FROM hashtags AS ht DELETE FROM hashtags AS ht
WHERE NOT EXISTS ( WHERE NOT EXISTS (
SELECT 1 FROM hashtags_objects hto SELECT 1 FROM hashtags_objects hto
WHERE ht.id = hto.hashtag_id) WHERE ht.id = hto.hashtag_id
UNION
SELECT 1 FROM user_follows_hashtag ufht
WHERE ht.id = ufht.hashtag_id)
""" """
|> Repo.query!() |> Repo.query!()