Prune Objects --keep-threads option #350

Merged
floatingghost merged 4 commits from ilja/akkoma:prune_objects_whithout_breaking_threads into develop 2 months ago
ilja commented 4 months ago

This adds an option to the prune_objects mix task.
The original way deleted all non-local public posts older than a certain time frame.
Here we add a different query which you can call using the option --keep-threads.

We query from the activities table all context id's where
1. the newest activity with this context is still old
2. none of the activities with this context is is local
3. none of the activities with this context is bookmarked
and delete all objects with these contexts.

The idea is that posts with local activities (posts, replies, likes, repeats...) may be interesting to keep.
Besides that, a post lives in a certain context (the thread), so we keep the whole thread as well.

Caveats:

  • Quotes have a different context. Therefore, when someone quotes a post, it's possible the quoted post will still be deleted. fixed in #379
  • Although undocumented (in docs/docs/administration/CLI_tasks/database.md/#prune-old-remote-posts-from-the-database), the 'normal' delete action still kept old remote non-public posts. I added an option to keep this behaviour, but this also means that you now have to explicitly provide that option. This could be considered a breaking change!
  • Note that this removes from the objects table, but not from the activities. See #427 for that.

Some statistics from explain analyse:
(cost=1402845.92..1933782.00 rows=3810907 width=62) (actual time=2562455.486..2562455.495 rows=0 loops=1)
Planning Time: 505.327 ms
Trigger for constraint chat_message_references_object_id_fkey: time=651939.797 calls=921740
Trigger for constraint deliveries_object_id_fkey: time=52036.009 calls=921740
Trigger for constraint hashtags_objects_object_id_fkey: time=20665.778 calls=921740
Execution Time: 3287933.902 ms


TODO

  1. Question: Is it OK to keep it like this in regard to quote posts? If not (ie post quoted by local users should also be kept), should we give quotes the same context as the post they are quoting? (If we don't want to give them the same context, I'll have to see how/if I can do it without being too costly)
  2. Question: the "original" query only deletes public posts (this is undocumented, but you can check the code). This new one doesn't care for scope. From the docs I get that the idea is that posts can be refetched when needed. But I have from a trusted source that Pleroma can't refetch non-public posts. I assume that's the reason why they are kept here. I see different options to deal with this
    1. We keep it as currently implemented and just don't care about scope with this option
    2. We add logic to not delete non-public posts either (I'll have to see how costly that becomes)
    3. We add an extra --keep-non-public parameter. This is technically speaking breakage (you didn't have to provide a param before for this, now you do), but I'm inclined to not care much because it wasn't documented nor tested in the first place.
  3. See if we can do the query using Elixir
  4. Test on a bigger DB to see that we don't run into a timeout
  5. Add docs
This adds an option to the prune_objects mix task. The original way deleted all non-local public posts older than a certain time frame. Here we add a different query which you can call using the option --keep-threads. We query from the activities table all context id's where 1. the newest activity with this context is still old 2. none of the activities with this context is is local 3. none of the activities with this context is bookmarked and delete all objects with these contexts. The idea is that posts with local activities (posts, replies, likes, repeats...) may be interesting to keep. Besides that, a post lives in a certain context (the thread), so we keep the whole thread as well. Caveats: * ~~Quotes have a different context. Therefore, when someone quotes a post, it's possible the quoted post will still be deleted.~~ fixed in https://akkoma.dev/AkkomaGang/akkoma/pulls/379 * Although undocumented (in docs/docs/administration/CLI_tasks/database.md/#prune-old-remote-posts-from-the-database), the 'normal' delete action still kept old remote non-public posts. I added an option to keep this behaviour, but this also means that you now have to explicitly provide that option. **This could be considered a breaking change!** * ~~Note that this removes from the objects table, but not from the activities.~~ See https://akkoma.dev/AkkomaGang/akkoma/pulls/427 for that. Some statistics from explain analyse: (cost=1402845.92..1933782.00 rows=3810907 width=62) (actual time=2562455.486..2562455.495 rows=0 loops=1) Planning Time: 505.327 ms Trigger for constraint chat_message_references_object_id_fkey: time=651939.797 calls=921740 Trigger for constraint deliveries_object_id_fkey: time=52036.009 calls=921740 Trigger for constraint hashtags_objects_object_id_fkey: time=20665.778 calls=921740 Execution Time: 3287933.902 ms *** **TODO** 1. [x] **Question:** Is it OK to keep it like this in regard to quote posts? If not (ie post quoted by local users should also be kept), should we give quotes the same context as the post they are quoting? (If we don't want to give them the same context, I'll have to see how/if I can do it without being too costly) * See https://akkoma.dev/AkkomaGang/akkoma/pulls/379 2. [x] **Question:** the "original" query only deletes public posts (this is undocumented, but you can check the code). This new one doesn't care for scope. From the docs I get that the idea is that posts can be refetched when needed. But I have from a trusted source that Pleroma can't refetch non-public posts. I assume that's the reason why they are kept here. I see different options to deal with this 1. ~~We keep it as currently implemented and just don't care about scope with this option~~ 2. ~~We add logic to not delete non-public posts either (I'll have to see how costly that becomes)~~ 3. We add an extra --keep-non-public parameter. This is technically speaking breakage (you didn't have to provide a param before for this, now you do), but I'm inclined to not care much because it wasn't documented nor tested in the first place. 3. [x] See if we can do the query using Elixir 4. [x] Test on a bigger DB to see that we don't run into a timeout 5. [x] Add docs
ilja added 1 commit 4 months ago
ci/woodpecker/pr/woodpecker Pipeline is pending Details
b201d625d7
Prune Objects --keep-threads
ilja added 1 commit 3 months ago
ci/woodpecker/pr/woodpecker Pipeline is pending Details
a3629b698b
Build prune_objects --keep-threads query with Ecto
ilja added 1 commit 3 months ago
ci/woodpecker/pr/woodpecker Pipeline is pending Details
e5dd375384
Add --keep-non-public option
ilja added 1 commit 3 months ago
Owner

Is it OK to keep it like this in regard to quote posts? If not (ie post quoted by local users should also be kept), should we give quotes the same context as the post they are quoting?

hm, decent point

i might err on giving quotes the same context?

though that might then cause them to be caught by context fetching

>Is it OK to keep it like this in regard to quote posts? If not (ie post quoted by local users should also be kept), should we give quotes the same context as the post they are quoting? hm, decent point i _might_ err on giving quotes the same context? though that might then cause them to be caught by context fetching
ilja closed this pull request 3 months ago
ilja reopened this pull request 3 months ago
ilja force-pushed prune_objects_whithout_breaking_threads from bf60996c6d to 9ba6d1bb08 3 months ago
ilja changed title from WIP: [I have questions] Prune Objects --keep-threads option to Prune Objects --keep-threads option 3 months ago
ilja force-pushed prune_objects_whithout_breaking_threads from 9ba6d1bb08 to f1739ac17b 3 months ago
Owner

that's quite the query you've got going there, splitting into 2PRs was probably the right move

this one looks fine though! thanks a lot~

that's quite the query you've got going there, splitting into 2PRs was probably the right move this one looks fine though! thanks a lot~
floatingghost merged commit 7695010268 into develop 2 months ago
floatingghost deleted branch prune_objects_whithout_breaking_threads 2 months ago
qbism referenced this issue from a commit 2 months ago
ci/woodpecker/pr/woodpecker Pipeline is pending
The pull request has been merged as 7695010268.
Sign in to join this conversation.
Loading…
There is no content yet.