Tweak users database indexes and drop exclude_visibilities #1019

Merged

Oneric merged 3 commits from Oneric/akkoma:db-index-tweaks into develop 2025-11-27 19:43:24 +00:00

Conversation 1 Commits 3 Files changed 11 +132 -44

Oneric commented 2025-11-24 19:51:03 +00:00

Owner

Copy link

Acting on the remaining insights from #764. Not all parts implemented yet, but can also be merged piecewise.

• converting eligible user indexes to partial indexes to reduce space usage and make them more efficient
• convert the activity visibility index to a proper table colum; there are several issues with this being an index:
  • it is computed by an SQL function activity_visibility which duplicates existing elixir code but is not aware of all actually existing visibility modes (miscategorises local as well as recently removed modes)
    (we can remove this function when the index is gone)
  • it depends on another index to compute its entries in bearable time; this indiex interdependence is causing issues during backup restore (see next point) and prevents using parallel index restore
  • the drawback is that we’ll need to manually ensure this field keeps in sync iwth other changes, BUT since we don't allow addressing to be updated after post creation anyway this should be simple (there’s nothing to do besides the initial computation)
• after the above, remove aa_users_ap_id_COALESCE_follower_address_index
  For our queries, it is redundant with users_ap_id_index. The only use it can serve is that from its haky renaming to start with aa_ it is more likely to be restored before the activity visibility index needs it

After this is all done, it might be good to (reset statistics and) collect new data about index issues / db health and SQL statement outliers from a diverse set of instances.

Acting on the remaining insights from https://akkoma.dev/AkkomaGang/akkoma/issues/764. Not all parts implemented yet, but can also be merged piecewise. - [X] converting eligible user indexes to partial indexes to reduce space usage and make them more efficient - [ ] convert the activity visibility index to a proper table colum; there are several issues with this being an index: - it is computed by an SQL function `activity_visibility` which duplicates existing elixir code but is not aware of all actually existing visibility modes *(miscategorises `local` as well as recently removed modes)* (we can remove this function when the index is gone) - it depends on another index to compute its entries in bearable time; this indiex interdependence is causing issues during backup restore *(see next point)* and prevents using parallel index restore - the drawback is that we’ll need to manually ensure this field keeps in sync iwth other changes, BUT since we don't allow addressing to be updated after post creation anyway this should be simple *(there’s nothing to do besides the initial computation)* - [ ] after the above, remove `aa_users_ap_id_COALESCE_follower_address_index` For our queries, it is redundant with `users_ap_id_index`. The only use it can serve is that from its haky renaming to start with `aa_` it is more likely to be restored _before_ the activity visibility index needs it After this is all done, it might be good to (reset statistics and) collect new data about index issues / db health and SQL statement outliers from a diverse set of instances.

Oneric added 1 commit 2025-11-24 19:51:03 +00:00

db: restrict indexes to local where appropriate ... 6865248dbc

This reduces both the size the indexes take on the disk
and the overhead of maintaining it since it now only
needs to be updated for some entries.

For the email index restriction queries needed to be updated,
the last_active_at index is used only by one query which already
also restricted by "local" and for the remaining restrictions
no query adaptations are needed.

Oneric force-pushed db-index-tweaks from 6865248dbc to 799e3400f5 2025-11-24 19:53:32 +00:00 Compare

Oneric force-pushed db-index-tweaks from 799e3400f5 to 18b3b5d65b 2025-11-27 17:58:25 +00:00 Compare

Oneric force-pushed db-index-tweaks from 18b3b5d65b to ba0e396ade 2025-11-27 17:59:46 +00:00 Compare

Oneric force-pushed db-index-tweaks from ba0e396ade to 453ab11fb2 2025-11-27 19:22:34 +00:00 Compare

Oneric commented 2025-11-27 19:33:53 +00:00

Author

Owner

Copy link

As it turns out, activity_visibility is only really used for

• the to-be-removed thread_containment
• the deprecated and in Mastodon removed direct timeline (which is consistently buggy in large instances)
• to implement the exclude_visibilities parameter which is very likely unused

So rather than going ahead with promoting visibility to a full column, let’s tentatively drop support for exclude_visibilities now and migrate akkoma-fe to the conversations API (which imho is better anyway) and then just completely drop activity_visibility etc

As it turns out, `activity_visibility` is only really used for - the to-be-removed `thread_containment` - the deprecated and in Mastodon removed `direct` timeline *(which is consistently buggy in large instances)* - to implement the `exclude_visibilities` parameter which is very likely unused So rather than going ahead with promoting visibility to a full column, let’s tentatively drop support for `exclude_visibilities` now and migrate akkoma-fe to the conversations API (which imho is better anyway) and then just completely drop `activity_visibility` etc

Oneric changed title from Tweak and cleanup database indexes to Tweak users database indexes and drop exclude_visibilities 2025-11-27 19:43:15 +00:00

Oneric merged commit 22d1b08456 into develop 2025-11-27 19:43:24 +00:00

Oneric deleted branch db-index-tweaks 2025-11-27 19:43:24 +00:00

Oneric referenced this pull request from a commit 2025-11-27 19:43:25 +00:00

Merge pull request 'Tweak users database indexes and drop exclude_visibilities' (#1019) from Oneric/akkoma:db-index-tweaks into develop

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

approved, awaiting change

  

broken setup

Issue arose from a misconfiguration by the admin, custom patches, manual db edits or similar

  

bug

Something is not working

  

cannot reproduce

  

configuration

This requires config changes

  

documentation

This is about human-readable docs

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

extremely low priority

  

feature request

Something new?

  

Fix it yourself

Support has been attempted

  

help wanted

Need some help

  

invalid

Something is wrong

  

mastodon_api

  

needs change/feedback

Issue or PR is stalled on response from the author or a third party

  

needs docs

  

needs tests

  

not a bug

  

not our bug

Bug in a dependency of ours (meaning we can't just fix it), or in another AP implementation or client interfacing with Akkoma (which needs to be fixed on their end)

  

planned

  

pleroma_api

  

privacy

  

question

More information is needed

  

static_fe

  

triage

  

wontfix

This won't be fixed

No labels

approved, awaiting change

broken setup

bug

cannot reproduce

configuration

documentation

duplicate

enhancement

extremely low priority

feature request

Fix it yourself

help wanted

invalid

mastodon_api

needs change/feedback

needs docs

needs tests

not a bug

not our bug

planned

pleroma_api

privacy

question

static_fe

triage

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

AkkomaGang/akkoma!1019

No description provided.