AkkomaGang/akkoma
AkkomaGang/akkoma
15
65
Fork 92
Code Issues 176 Pull requests 24 Projects 2 Releases 14 Packages Wiki Activity

Prune old Update activities #683

Merged
floatingghost merged 2 commits from Oneric/akkoma:db-prune-old-updates into develop 2024-02-19 13:59:34 +00:00
Conversation 2 Commits 2 Files changed 4 +97 -6
Oneric commented 2024-02-10 01:04:34 +00:00
Member
Copy link

Plus fixing a typo in the test factory and expanding the prune_task test.

Followup to IRC question/discussion; as far as I can tell already processed Update activities serve no purpose anymore and just bloat the DB (conceptually this holds true for all types of updates and i had closer look at updates of: users, notes and for poll vote updates no activity ever showed up in the DB to begin with for me).

The last commit merging the different, simple prunes seems sensible to me, but I admit I didn’t do any actual perf measurements; if you’re unsure about it, feel free to just drop the last commit during merge.

Plus fixing a typo in the test factory and expanding the `prune_task` test. Followup to IRC question/discussion; as far as I can tell already processed `Update` activities serve no purpose anymore and just bloat the DB (conceptually this holds true for all types of updates and i had closer look at updates of: users, notes and for poll vote updates no activity ever showed up in the DB to begin with for me). The last commit merging the different, simple prunes seems sensible to me, but I admit I didn’t do any actual perf measurements; if you’re unsure about it, feel free to just drop the last commit during merge.
Oneric force-pushed db-prune-old-updates from 8cc04c9e53 to 4fa1183f6a 2024-02-10 01:14:19 +00:00 Compare
floatingghost commented 2024-02-10 08:55:19 +00:00
Owner
Copy link

seems good, I'll try it out locally to make sure nothing explodes soon

seems good, I'll try it out locally to make sure nothing explodes soon
👍 1
floatingghost reviewed 2024-02-17 12:03:51 +00:00
lib/pleroma/activity/pruner.ex Outdated
@ -31,3 +15,1 @@
from(a in Activity,
where: fragment("?->>'type' = ?", a.data, "Remove") and a.inserted_at < ^before_time
where:
floatingghost commented 2024-02-17 12:03:51 +00:00
Owner
Copy link

ok the one thing that does mildly concern me here is the size of this query - the reason i split things up before is to make queries as short and self-contained as possible, so reduce the chance of things timing out

for example, on my production database, running a select over these types takes ~16s

whereas just running for one type at a time takes less than a second

probably due to index searching being very easy

ok the one thing that does mildly concern me here is the size of this query - the reason i split things up before is to make queries as short and self-contained as possible, so reduce the chance of things timing out for example, on my production database, running a select over these types takes ~16s whereas just running for one type at a time takes less than a second probably due to index searching being very easy
Oneric commented 2024-02-17 16:08:38 +00:00
Author
Member
Copy link

I guess the query optimiser isn’t smart enough to also use the context index for sets, oh well

Thanks for testing perf on real world system, i now dropped the last commit (and rebased; no changes to the first two commits necessary)

I guess the query optimiser isn’t smart enough to also use the context index for sets, oh well Thanks for testing perf on real world system, i now dropped the last commit *(and rebased; no changes to the first two commits necessary)*
Oneric marked this conversation as resolved
Oneric force-pushed db-prune-old-updates from 4fa1183f6a to 1a7839eaf2 2024-02-17 16:06:20 +00:00 Compare
floatingghost commented 2024-02-19 13:59:28 +00:00
Owner
Copy link

nothing exploded, so i think that is a good sign

nothing exploded, so i think that is a good sign
floatingghost merged commit 3e24210e9f into develop 2024-02-19 13:59:34 +00:00
floatingghost deleted branch db-prune-old-updates 2024-02-19 13:59:34 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
approved, awaiting change

   
bug

Something is not working

  
configuration

This requires config changes

  
documentation

This is about human-readable docs

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
extremely low priority

   
feature request

Something new?

  
Fix it yourself

Support has been attempted

  
help wanted

Need some help

  
invalid

Something is wrong

  
mastodon_api

   
needs docs

   
needs tests

   
not a bug

   
planned

   
pleroma_api

   
privacy

   
question

More information is needed

  
static_fe

   
triage

   
wontfix

This won't be fixed

No labels
approved, awaiting change
bug
configuration
documentation
duplicate
enhancement
extremely low priority
feature request
Fix it yourself
help wanted
invalid
mastodon_api
needs docs
needs tests
not a bug
planned
pleroma_api
privacy
question
static_fe
triage
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: AkkomaGang/akkoma#683
No description provided.
Delete branch "Oneric/akkoma:db-prune-old-updates"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?