ilja
92d2f8b401
Add --keep-non-public option
...
The prune_objects task already did this by default, but is undocumented.
Now we require an explicit parameter for it.
The parameter also works in combination with --keep-threads
Docs still needs to happen
2023-01-04 19:17:03 +01:00
ilja
04cc1d41ce
Build prune_objects --keep-threads query with Ecto
...
The query is now done using Ecto.
I also ran it on a local DB.
It Went from 4000834 records to 1734648 in about an hour without timeout.
2023-01-04 19:17:03 +01:00
ilja
eb503f093c
Prune Objects --keep-threads
...
This adds an option to the prune_objects mix task.
The original way deleted all non-local public posts older than a certain time frame.
Here we add a different query which you can call using the option --keep-threads.
We query from the activities table all context id's where
1. the newest activity with this context is still old
2. none of the activities with this context is is local
3. none of the activities with this context is bookmarked
and delete all objects with these contexts.
The idea is that posts with local activities (posts, replies, likes, repeats...) may be intersesting to keep.
Besides that, a post lives in a certain context (the thread), so we keep the whole thread as well.
Caveats:
* Quotes have a different context. Therefore, when someone quotes a post, it's possible the quoted post will still be deleted.
* Although undocumented (in docs/docs/administration/CLI_tasks/database.md/#prune-old-remote-posts-from-the-database), the 'normal' delete action still keeps old remote non-public posts. With this option we don't care about scope.
* I ran this on my instance, but directly on the DB. I still need to test to be sure that we don't get a time-out error or something.
Some statistics from explain analyse:
(cost=1402845.92..1933782.00 rows=3810907 width=62) (actual time=2562455.486..2562455.495 rows=0 loops=1)
Planning Time: 505.327 ms
Trigger for constraint chat_message_references_object_id_fkey: time=651939.797 calls=921740
Trigger for constraint deliveries_object_id_fkey: time=52036.009 calls=921740
Trigger for constraint hashtags_objects_object_id_fkey: time=20665.778 calls=921740
Execution Time: 3287933.902 ms
2023-01-04 19:17:03 +01:00
floatingghost
07a48b9293
giant massive dep upgrade and dialyxir-found error emporium ( #371 )
...
ci/woodpecker/push/woodpecker Pipeline is pending
Details
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #371
2022-12-14 12:38:48 +00:00
floatingghost
db60640c5b
Fixing up deletes a bit ( #327 )
...
ci/woodpecker/push/woodpecker Pipeline failed
Details
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #327
2022-12-01 15:00:53 +00:00
floatingghost
d2a185c013
Documentation updates for stable release ( #73 )
...
ci/woodpecker/push/docs Pipeline is pending
Details
ci/woodpecker/push/release Pipeline is pending
Details
ci/woodpecker/push/test Pipeline is pending
Details
Reviewed-on: #73
2022-07-15 12:27:16 +00:00
someone
4b940e441a
mix pleroma.database set_text_search_config now runs concurrently and infinitely
2021-08-15 13:49:12 -04:00
feld
bc51dea425
Update lib/mix/tasks/pleroma/database.ex
2021-06-07 20:02:28 +00:00
faried nawaz
5be9d13981
a better query to delete from hashtags
...
old query:
Delete on hashtags (cost=5089.81..5521.63 rows=6160 width=18)
-> Hash Semi Join (cost=5089.81..5521.63 rows=6160 width=18)
Hash Cond: (hashtags.id = ht.id)
-> Seq Scan on hashtags (cost=0.00..317.28 rows=17528 width=14)
-> Hash (cost=5012.81..5012.81 rows=6160 width=20)
-> Merge Anti Join (cost=0.70..5012.81 rows=6160 width=20)
Merge Cond: (ht.id = hto.hashtag_id)
-> Index Scan using hashtags_pkey on hashtags ht (cost=0.29..610.53 rows=17528 width=14)
-> Index Scan using hashtags_objects_pkey on hashtags_objects hto (cost=0.42..3506.48 rows=68158 width=14)
new query:
Delete on hashtags ht (cost=0.70..5012.81 rows=6160 width=12)
-> Merge Anti Join (cost=0.70..5012.81 rows=6160 width=12)
Merge Cond: (ht.id = hto.hashtag_id)
-> Index Scan using hashtags_pkey on hashtags ht (cost=0.29..610.53 rows=17528 width=14)
-> Index Scan using hashtags_objects_pkey on hashtags_objects hto (cost=0.42..3506.48 rows=68158 width=14)
2021-05-08 02:00:43 +05:00
faried nawaz
a0c9a2b4cc
mix prune_objects: remove unused hashtags after pruning remote objects
2021-05-08 02:00:42 +05:00
Ivan Tashkinov
40d4362261
[ #3213 ] `mix pleroma.database rollback` tweaks.
2021-02-23 18:11:25 +03:00
Ivan Tashkinov
5992382cf8
Merge remote-tracking branch 'remotes/origin/develop' into feature/object-hashtags-rework
...
# Conflicts:
# CHANGELOG.md
# lib/mix/tasks/pleroma/database.ex
# lib/pleroma/web/templates/feed/feed/_activity.rss.eex
2021-02-11 19:31:57 +03:00
Ivan Tashkinov
d1c6dd97aa
[ #3213 ] Partially addressed code review points.
...
migration rollback task changes, hashtags-related config handling tweaks, `hashtags.data` deletion (unused).
2021-02-07 22:24:12 +03:00
hyperion
8d4e0342e1
Update priv/repo/migrations/20190501125843_add_fts_index_to_objects.exs, priv/repo/optional_migrations/rum_indexing/20190510135645_add_fts_index_to_objects_two.exs files
2021-02-06 09:42:17 +00:00
Ivan Tashkinov
108e90b18e
[ #3213 ] Explicitly defined PKs in hashtags_objects and data_migration_failed_ids. Added "pleroma.database rollback" task to revert a single migration.
2021-01-31 22:03:59 +03:00
Ivan Tashkinov
e350898828
Merge remote-tracking branch 'remotes/origin/develop' into feature/object-hashtags-rework
2021-01-13 22:11:16 +03:00
Ivan Tashkinov
3e4d84729a
[ #3213 ] Prototype of data migrations functionality / HashtagsTableMigrator.
2021-01-13 22:07:38 +03:00
Haelwenn (lanodan) Monnier
c4439c630f
Bump Copyright to 2021
...
grep -rl '# Copyright © .* Pleroma' * | xargs sed -i 's;Copyright © .* Pleroma .*;Copyright © 2017-2021 Pleroma Authors <https://pleroma.social/ >;'
2021-01-13 07:49:50 +01:00
Ivan Tashkinov
8c972de045
[ #3213 ] transfer_hashtags mix task refactoring.
2021-01-10 11:44:39 +03:00
Ivan Tashkinov
0d521022fe
[ #3213 ] Removed PK from hashtags_objects table. Improved hashtags_transfer mix task (logging of failed ids).
2021-01-07 12:20:29 +03:00
Ivan Tashkinov
367f0c31c3
[ #3213 ] Added query options support for Repo.chunk_stream/4.
...
Used infinite timeout in transfer_hashtags select query.
2020-12-31 09:36:26 +03:00
Ivan Tashkinov
a25c1e8ec0
[ #3213 ] Improved `database.transfer_hashtags` mix task: proper rollback, speedup.
2020-12-30 14:35:19 +03:00
Haelwenn
3966add048
Revert "Merge branch 'features/hashtag-column' into 'develop'"
...
This reverts merge request !2824
2020-12-28 12:02:16 +00:00
Haelwenn (lanodan) Monnier
d0c2479710
pleroma.database fill_old_hashtags: Add month_limit argument
2020-12-28 11:05:25 +01:00
Ivan Tashkinov
cbb19d0e18
[ #3213 ] Hashtag-filtering functions in ActivityPub. Mix task for migrating hashtags to `hashtags` table.
2020-12-26 22:20:55 +03:00
Haelwenn (lanodan) Monnier
acb03d591b
Insert text representation of hashtags into object["hashtags"]
...
Includes a new mix task: pleroma.database fill_old_hashtags
2020-12-22 05:15:34 +01:00
Alexander Strizhakov
cebe3c7def
Fix for dropping posts/notifs in WS when mix task is executed
...
- start oban in mix tasks with empty queues, plugins and crontab
- fix for update_users_following_followers_counts
- fix for removed logo.png
- typo in resend confirmation emails mix task docs
- fix for uploads mix task (start Majic.Pool)
- fix for creating user mix task (start :fast_html app)
2020-12-14 11:02:32 -06:00
Maksim Pechnikov
599f8bb152
RepoStreamer.chunk_stream -> Repo.chunk_stream
2020-09-16 09:47:18 +03:00
Alexander Strizhakov
15aece7238
remove validate_expires_at from enqueue method
2020-09-10 21:52:31 +03:00
Alexander Strizhakov
9bf1065a06
schedule activity expiration in Oban
2020-09-10 21:50:40 +03:00
Alexander Strizhakov
29a7bcd5bb
reverting pinned posts in filtering
2020-08-12 20:01:21 +03:00
Alexander Strizhakov
3ab83f837e
don't load pinned activities in due_expirations
2020-08-12 19:46:47 +03:00
Alexander Strizhakov
eec1ba232c
don't expire pinned posts
2020-08-12 15:15:17 +03:00
Mark Felder
724ed354f2
Ensure only Note objects are set to expire
2020-08-11 11:28:22 -05:00
Mark Felder
cf4c97242b
Ensure we only expire Create activities with the Mix task
2020-08-08 12:40:52 -05:00
Egor Kislitsyn
e5557bf8ba
Add mix task to add expiration to all local statuses
2020-08-08 16:29:40 +04:00
Mark Felder
92fba24c74
Alpha sort
2020-05-27 17:17:06 -05:00
Mark Felder
30f96b19c1
Abstract out the database maintenance. I'd like to use this from AdminFE too.
2020-05-27 16:40:51 -05:00
Mark Felder
0d57e06626
Make clearer that this is time and resource consuming
2020-05-27 16:31:37 -05:00
Mark Felder
73ca57e4f1
Make it obvious a full vacuum can take a while
2020-05-27 16:27:29 -05:00
Mark Felder
a2f57bd82b
Permit easy access to vaccum full and analyze via a mix task
2020-05-27 16:27:07 -05:00
Mark Felder
05da5f5cca
Update Copyrights
2020-03-03 16:44:49 -06:00
Egor Kislitsyn
4ea1a61b00
Merge branch 'develop' into refactor/following-relationships
2019-10-21 14:19:15 +07:00
rinpatch
39e996528c
Fix a migration wiping user info of users that don't have any mutes
...
And introduce safe_jsonb_set
2019-10-18 15:22:07 +03:00
Egor Kislitsyn
059005ff82
Replace `user.following` with Pleroma.FollowingRelationship
2019-10-11 02:35:32 +07:00
rinpatch
e00403af23
Mix tasks: derive moduledoc from doc files
2019-10-03 14:00:23 +03:00
rinpatch
6baa037903
Move database maintenance tasks docs to a separate file
2019-10-03 11:17:29 +03:00
Haelwenn (lanodan) Monnier
447514dfa2
Bump copyright years of files changed in 2019
...
Done via the following command:
git diff 1e6c102bfc
--stat --name-only | cat - | xargs sed -i 's/2017-2018 Pleroma Authors/2017-2019 Pleroma Authors/'
2019-09-18 23:21:11 +02:00
Sergey Suprunenko
af4cf35e20
Strip internal fields including likes from incoming and outgoing activities
2019-08-10 18:47:40 +00:00
Ariadne Conill
b93498eb52
constants: add as_public constant and use it everywhere
2019-07-29 02:43:19 +00:00