Commit graph

51 commits

Author SHA1 Message Date
ilja
57eef6d764 prune_objects can prune orphaned activities who reference an array of objects
E.g. Flag activities have an array of objects

We prune the activity when NONE of the objects can be found

Note that the cost of finding and deleting these is ~4x higher than finding and deleting the non-array ones

Only string:
Delete on activities  (cost=506573.48..506580.38 rows=0 width=0)

Only Array:
Delete on activities  (cost=3570359.68..4276365.34 rows=0 width=0)

(They are still executed separately, so the total cost is the sum of the two)
2023-02-26 14:41:50 +01:00
ilja
a7ec6e039c prune_objects can prune orphaned activities
We add an option to also prune remote activities who don't have existing objects any more they reference.
Rn, we only check for activities who only reference one object, not an array or embeded object.
2023-02-26 14:41:50 +01:00
7695010268 Prune Objects --keep-threads option (#350)
This adds an option to the prune_objects mix task.
The original way deleted all non-local public posts older than a certain time frame.
Here we add a different query which you can call using the option --keep-threads.

We query from the activities table all context id's where
    1. the newest activity with this context is still old
    2. none of the activities with this context is is local
    3. none of the activities with this context is bookmarked
and delete all objects with these contexts.

The idea is that posts with local activities (posts, replies, likes, repeats...) may be interesting to keep.
Besides that, a post lives in a certain context (the thread), so we keep the whole thread as well.

Caveats:
* ~~Quotes have a different context. Therefore, when someone quotes a post, it's possible the quoted post will still be deleted.~~ fixed in AkkomaGang/akkoma#379
* Although undocumented (in docs/docs/administration/CLI_tasks/database.md/#prune-old-remote-posts-from-the-database), the 'normal' delete action still kept old remote non-public posts. I added an option to keep this behaviour, but this also means that you now have to explicitly provide that option. **This could be considered a breaking change!**
* ~~Note that this removes from the objects table, but not from the activities.~~ See AkkomaGang/akkoma#427 for that.

Some statistics from explain analyse:
(cost=1402845.92..1933782.00 rows=3810907 width=62) (actual time=2562455.486..2562455.495 rows=0 loops=1)
 Planning Time: 505.327 ms
 Trigger for constraint chat_message_references_object_id_fkey: time=651939.797 calls=921740
 Trigger for constraint deliveries_object_id_fkey: time=52036.009 calls=921740
 Trigger for constraint hashtags_objects_object_id_fkey: time=20665.778 calls=921740
 Execution Time: 3287933.902 ms

***
**TODO**
1. [x] **Question:** Is it OK to keep it like this in regard to quote posts? If not (ie post quoted by local users should also be kept), should we give quotes the same context as the post they are quoting? (If we don't want to give them the same context, I'll have to see how/if I can do it without being too costly)
    * See AkkomaGang/akkoma#379
2. [x] **Question:** the "original" query only deletes public posts (this is undocumented, but you can check the code). This new one doesn't care for scope. From the docs I get that the idea is that posts can be refetched when needed. But I have from a trusted source that Pleroma can't refetch non-public posts. I assume that's the reason why they are kept here. I see different options to deal with this
    1. ~~We keep it as currently implemented and just don't care about scope with this option~~
    2. ~~We add logic to not delete non-public posts either (I'll have to see how costly that becomes)~~
    3. We add an extra --keep-non-public parameter. This is technically speaking breakage (you didn't have to provide a param before for this, now you do), but I'm inclined to not care much because it wasn't documented nor tested in the first place.
3. [x] See if we can do the query using Elixir
4. [x] Test on a bigger DB to see that we don't run into a timeout
5. [x] Add docs

Co-authored-by: ilja <git@ilja.space>
Reviewed-on: AkkomaGang/akkoma#350
Co-authored-by: ilja <akkoma.dev@ilja.space>
Co-committed-by: ilja <akkoma.dev@ilja.space>
2023-01-09 22:15:41 +00:00
f752126427 Remove quack, ensure adapter is finch 2022-12-11 23:22:35 +00:00
8e4de118c1 Don't persist local undone follow (#194)
same deal but backwards this time

Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: AkkomaGang/akkoma#194
2022-08-31 18:00:36 +00:00
0f132b802d purge chat and shout endpoints 2022-07-21 11:29:28 +01:00
Tusooa Zhu
4ed0a780b7 Fix digest test 2022-06-29 20:45:05 +01:00
Alex Gleason
54af527759
Upgrade Ecto to v3.6.2, remove deprecated ecto_explain 2021-06-23 13:22:32 -05:00
fe40f6f291 Switch from the deprecated "use Mix.config" to "import Config" 2021-05-20 14:02:58 -05:00
Alexander Strizhakov
03843a5386
migrating config to tmp folder 2021-03-23 14:29:02 +03:00
2217b35bd4 Merge branch 'ecto-rollback-in-test-env' into 'develop'
Possibility to run rollback in test env

See merge request pleroma/pleroma!3212
2021-02-26 16:47:53 +00:00
e854c35e65 Convert tests to all use clear_config instead of Pleroma.Config.put 2021-01-26 11:58:43 -06:00
f868dcf3ac Ensure consistent behavior between pleroma.user activate/deactivate mix tasks 2021-01-25 10:48:28 -06:00
5e8da27e14 Provide pleroma.user mix task for both activate and deactivate 2021-01-25 10:36:51 -06:00
28581e03ad Merge branch 'develop' into refactor/deactivated_user_field 2021-01-18 14:58:21 -06:00
Mark Felder
2c0fe2ea9e Remove toggle_confirmation; require explicit state change
Also cosmetic changes to make the code clearer
2021-01-15 13:11:51 -06:00
Mark Felder
d36182c088 Change user.confirmation_pending field to user.is_confirmed 2021-01-15 12:44:41 -06:00
Mark Felder
7516660753 Remove toggle_activation 2021-01-15 11:25:32 -06:00
Mark Felder
860b5c7804 Change user.deactivated field to user.is_active 2021-01-15 11:24:46 -06:00
Haelwenn (lanodan) Monnier
c4439c630f
Bump Copyright to 2021
grep -rl '# Copyright © .* Pleroma' * | xargs sed -i 's;Copyright © .* Pleroma .*;Copyright © 2017-2021 Pleroma Authors <https://pleroma.social/>;'
2021-01-13 07:49:50 +01:00
lain
e1e7e4d379 Object: Rework how Object.normalize works
Now it defaults to not fetching, and the option is named.
2021-01-04 13:38:31 +01:00
lain
e4f1d8f48c Merge branch 'cachex-test' into 'develop'
Test framework overhaul (speed, reliability)

See merge request pleroma/pleroma!3209
2020-12-26 10:26:35 +00:00
Alexander Strizhakov
538af14d52
possibility to run rollback in test env 2020-12-22 17:55:30 +03:00
lain
b17c36c45a Tests: More test fixes. 2020-12-21 12:59:11 +01:00
lain
9ba60f70d2 Tests: Make as many tests as possible async.
In general, tests that match these criteria can be made async:

- Doesn't use real Cachex.
- Doesn't write to the Config / Application Environment.
- Uses Mock. Using Mox is fine.
- Uses the streamer.
2020-12-21 12:21:40 +01:00
lain
b4b68b71fc Tests: Remove unneeded explicit cachex use.
Only use cachex when we're actually testing it.
2020-12-18 13:18:17 +01:00
Alex Gleason
80891e83d8
Merge remote-tracking branch 'upstream/develop' into registration-workflow 2020-12-17 09:05:36 -06:00
Alexander Strizhakov
cebe3c7def Fix for dropping posts/notifs in WS when mix task is executed
- start oban in mix tasks with empty queues, plugins and crontab
- fix for update_users_following_followers_counts
- fix for removed logo.png
- typo in resend confirmation emails mix task docs
- fix for uploads mix task (start Majic.Pool)
- fix for creating user mix task (start :fast_html app)
2020-12-14 11:02:32 -06:00
lain
1436a2fa2d Merge branch 'stream-follow-updates' into 'develop'
Stream follow updates

Closes #2299

See merge request pleroma/pleroma!3183
2020-12-09 15:52:31 +00:00
lain
ed76323776 Merge branch 'feature/configdb-mix-tasks' into 'develop'
Add mix tasks to give additional ConfigDB recovery and debugging options

See merge request pleroma/pleroma!3174
2020-12-07 18:06:06 +00:00
Mark Felder
e3dd0d45b7 Slip in a test to ensure we can use the atom syntax in mix task arguments 2020-12-07 11:21:06 -06:00
Ivan Tashkinov
e00c667145 [#3174] Refactoring: ConfigDB fetching functions, ConfigDB tests.
Minor fixes.
2020-12-06 18:02:30 +03:00
Mark Felder
657002e738 Answer new prompt interactively 2020-12-04 12:07:23 -06:00
Mark Felder
e379ab8277 Add --force flag for delete and reset commands
Bunch of reorganization and consolidation
2020-12-02 16:24:32 -06:00
Mark Felder
13947999ad Use a callback strategy to short circuit the functions and print a nice error 2020-12-02 12:33:34 -06:00
Mark Felder
3e6d9187a7 Add tests for config dumping 2020-12-02 10:43:22 -06:00
Egor Kislitsyn
35ba48494f
Stream follow updates 2020-12-02 00:18:58 +04:00
Alex Gleason
9546c1444c
Merge remote-tracking branch 'upstream/develop' into registration-workflow 2020-11-14 19:48:47 -06:00
Haelwenn (lanodan) Monnier
e2f573d68b
pleroma.instance: Fix Exiftool module name 2020-11-14 22:31:01 +01:00
rinpatch
cc45c69fff Remove release_env
While taking a final look at instance.gen before releasing I noticed
that the release_env task outputs messages in broken english. Upon
further inspection it seems to have even more severe issues which, in
my opinion, warrant it's at least temporary removal:
- We do not explain what it actually does, anywhere. Neither the task
 docs nor instance.gen, nor installation instructions.
- It does not respect FHS on OTP releases (uses /opt/pleroma/config even
 though we store the config in /etc/pleroma/config.exs).
- It doesn't work on OTP releases, which is the main reason it exists.
Neither systemd nor openrc service files for OTP include it.
- It is not mentioned in install guides other than the ones for Debian
and OTP releases.
2020-11-08 11:56:09 +03:00
Maksim Pechnikov
2030ffd490 fix test 2020-10-15 22:31:52 +03:00
Maksim Pechnikov
20e68b30f0 added generated pleroma.env 2020-10-15 22:28:26 +03:00
Alex Gleason
dc38dc8472
Replace User.toggle_confirmation/1 with User.confirm/1, fixes #2235 2020-10-13 22:01:27 -05:00
Mark Felder
09be8cb336 Credo 2020-10-13 12:49:43 -05:00
Mark Felder
409f694e4f Merge branch 'develop' into refactor/locked_user_field 2020-10-13 09:54:29 -05:00
Alexander Strizhakov
b081080dd9
fixes after rebase 2020-10-13 16:44:02 +03:00
Alexander Strizhakov
6bf85440b3
mix tasks consistency 2020-10-13 16:33:24 +03:00
Sergey Suprunenko
75a5dd41ee Add more tests for Database tasks and DigestEmailWorker 2019-08-19 16:10:00 +00:00
Roman Chvanikov
f6036ce3b9 Fix tests 2019-06-04 03:38:53 +03:00
Roman Chvanikov
bd325132ca Fix tests 2019-06-04 03:07:49 +03:00