678 commits

Author	SHA1	Message	Date
ilja	04cc1d41ce	Build prune_objects --keep-threads query with Ecto ... The query is now done using Ecto. I also ran it on a local DB. It Went from 4000834 records to 1734648 in about an hour without timeout.	2023-01-04 19:17:03 +01:00
ilja	eb503f093c	Prune Objects --keep-threads ... This adds an option to the prune_objects mix task. The original way deleted all non-local public posts older than a certain time frame. Here we add a different query which you can call using the option --keep-threads. We query from the activities table all context id's where 1. the newest activity with this context is still old 2. none of the activities with this context is is local 3. none of the activities with this context is bookmarked and delete all objects with these contexts. The idea is that posts with local activities (posts, replies, likes, repeats...) may be intersesting to keep. Besides that, a post lives in a certain context (the thread), so we keep the whole thread as well. Caveats: * Quotes have a different context. Therefore, when someone quotes a post, it's possible the quoted post will still be deleted. * Although undocumented (in docs/docs/administration/CLI_tasks/database.md/#prune-old-remote-posts-from-the-database), the 'normal' delete action still keeps old remote non-public posts. With this option we don't care about scope. * I ran this on my instance, but directly on the DB. I still need to test to be sure that we don't get a time-out error or something. Some statistics from explain analyse: (cost=1402845.92..1933782.00 rows=3810907 width=62) (actual time=2562455.486..2562455.495 rows=0 loops=1) Planning Time: 505.327 ms Trigger for constraint chat_message_references_object_id_fkey: time=651939.797 calls=921740 Trigger for constraint deliveries_object_id_fkey: time=52036.009 calls=921740 Trigger for constraint hashtags_objects_object_id_fkey: time=20665.778 calls=921740 Execution Time: 3287933.902 ms	2023-01-04 19:17:03 +01:00
floatingghost	9be6caf125	argon2 password hashing (#406) ... Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk> Reviewed-on: #406	2022-12-30 02:46:58 +00:00
FloatingGhost	5a405bdadf	document dump_to_file and load_from_file	2022-12-29 20:00:04 +00:00
FloatingGhost	d1bf8aa9ed	Add dump_to_file and load_from_file tasks	2022-12-29 19:56:35 +00:00
floatingghost	07a48b9293	giant massive dep upgrade and dialyxir-found error emporium (#371) ... Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk> Reviewed-on: #371	2022-12-14 12:38:48 +00:00
FloatingGhost	e6da301296	Add diagnostics http	2022-12-11 22:57:18 +00:00
floatingghost	09326ffa56	Diagnostics tasks (#348) ... a bunch of ways to get query plans to help with debugging Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk> Reviewed-on: #348	2022-12-07 11:12:34 +00:00
floatingghost	d55de5debf	Remerge of hashtag following (#341) ... this time with less idiot Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk> Reviewed-on: #341	2022-12-05 12:58:48 +00:00
floatingghost	ec6bf8c3f7	revert 4a94c9a31e ... revert Add ability to follow hashtags (#336) Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk> Reviewed-on: #336	2022-12-04 20:04:09 +00:00
floatingghost	4a94c9a31e	Add ability to follow hashtags (#336) ... Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk> Reviewed-on: #336	2022-12-04 17:36:59 +00:00
floatingghost	6b882a2c0b	Purge Rejected Follow requests in daily task (#334) ... Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk> Reviewed-on: #334	2022-12-03 23:17:43 +00:00
floatingghost	db60640c5b	Fixing up deletes a bit (#327) ... Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk> Reviewed-on: #327	2022-12-01 15:00:53 +00:00
floatingghost	e3085c495c	fix tests broken by relay defaults changing (#314) ... Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk> Reviewed-on: #314	2022-11-26 20:45:47 +00:00
FloatingGhost	856c57208b	Ensure deletes are handled after everything else	2022-10-11 14:30:08 +01:00
floatingghost	92ba2802fb	generate-keys-at-registration-time (#181) ... Reviewed-on: #181	2022-08-24 14:36:33 +00:00
floatingghost	61641957cb	fix compatibility with meilisearch (#164) ... Reviewed-on: #164	2022-08-16 22:56:49 +00:00
FloatingGhost	2033d7d4fc	ensure extra info in fix_follow_state prints	2022-07-29 19:50:26 +01:00
floatingghost	4c47992686	bugfix/follow-state (#104) ... Reviewed-on: #104	2022-07-23 18:58:45 +00:00
floatingghost	d2a185c013	Documentation updates for stable release (#73) ... Reviewed-on: #73	2022-07-15 12:27:16 +00:00
Norm	7dfc3f3d0e	Change default Postgres user/DB to akkoma	2022-07-12 12:41:30 -04:00
floatingghost	a9c82b62f2	Fixes for elasticsearch 8 (#52) ... Reviewed-on: #52	2022-07-06 18:57:00 +00:00
FloatingGhost	a036a01a1e	mix format	2022-07-04 17:38:16 +01:00
floatingghost	364b6969eb	Use finch everywhere (#33) ... Reviewed-on: #33	2022-07-04 16:30:38 +00:00
floatingghost	2937495712	fix ES import from live (#29) ... Reviewed-on: #29	2022-06-30 18:44:31 +00:00
FloatingGhost	bc9e76cce7	Add documentation for ES search	2022-06-30 17:36:57 +01:00
FloatingGhost	1ecdb19de5	Refactor ES on top of search behaviour	2022-06-30 16:28:31 +01:00
Ekaterina Vaartis	563b964690	Change updateId to uid because apparently that's the new name	2022-06-29 20:49:45 +01:00
Ekaterina Vaartis	b7462040cc	Change the meilisearch key auth to conform to 0.25.0	2022-06-29 20:49:45 +01:00
Ekaterina Vaartis	a4914add8c	Don't support meilisearch < 0.24.0, since it breaks things	2022-06-29 20:49:45 +01:00
Ekaterina Vaartis	cc3319ac1d	Make chunk size configurable	2022-06-29 20:49:45 +01:00
Ekaterina Vaartis	bac70a2bc1	Implement suggestions from the Meilisearch MR ... - Index unlisted posts - Move version check outside of the streaming and only do it once - Use a PUT request instead of checking manually if there is need to insert - Add error handling, sort of	2022-06-29 20:49:45 +01:00
Ekaterina Vaartis	0769f06bd1	Style fixes	2022-06-29 20:49:00 +01:00
Ekaterina Vaartis	86971fceaa	Support reindexing meilisearch >=0.24.0 ... It has has a different error code key	2022-06-29 20:48:44 +01:00
Ekaterina Vaartis	9e7d7ebd48	Add a reindex option ... Signed-off-by: Ekaterina Vaartis <vaartis@kotobank.ch>	2022-06-29 20:48:44 +01:00
Ekaterina Vaartis	5ed1759091	Reorder ranking rules for (maybe) better results	2022-06-29 20:48:44 +01:00
Ekaterina Vaartis	d1079f1aa3	Add the meilisearch.stats command	2022-06-29 20:48:29 +01:00
Ekaterina Vaartis	d5cc272a91	Add a message with a count of posts to index	2022-06-29 20:48:29 +01:00
Ekaterina Vaartis	dbf556cdcf	Implement meilisearch auth	2022-06-29 20:48:29 +01:00
Ekaterina Vaartis	5360cc1097	Make indexing logs rewrite themselves	2022-06-29 20:48:29 +01:00
Ekaterina Vaartis	0cf3654907	Rework task indexing to share code with the main module ... The code in the main module now scrubs new posts too	2022-06-29 20:48:29 +01:00
Ekaterina Vaartis	117f525fd6	Adjust content indexing to skip more unneeded stuff	2022-06-29 20:48:29 +01:00
Ekaterina Vaartis	14ef6ce80f	Mark only content as searchable for meilisearch	2022-06-29 20:48:29 +01:00
Ekaterina Vaartis	52a872432d	Make the chunk size smaller	2022-06-29 20:48:29 +01:00
Ekaterina Vaartis	a586ce0ddd	Use content instead of source and scrub it	2022-06-29 20:48:29 +01:00
Ekaterina Vaartis	c3a04166a0	Tweak search ordering to hopefully return newer results	2022-06-29 20:48:29 +01:00
Ekaterina Vaartis	7b3701e6b9	Make meilisearch sort on publish date converted to unix time	2022-06-29 20:48:29 +01:00
Ekaterina Vaartis	abf82a63ec	Make the indexing batch differently and more, show number indexed	2022-06-29 20:48:29 +01:00
Ekaterina Vaartis	e5ac2ffa07	Ensure only indexing public posts and implement clearing and delete	2022-06-29 20:48:29 +01:00
Ekaterina Vaartis	41db5c8653	Add logging to milisiearch index and make it use desc(id)	2022-06-29 20:48:29 +01:00