138 commits

Author	SHA1	Message	Date
Mark Felder	5da9cbd8a5	RichMedia refactor ... Rich Media parsing was previously handled on-demand with a 2 second HTTP request timeout and retained only in Cachex. Every time a Pleroma instance is restarted it will have to request and parse the data for each status with a URL detected. When fetching a batch of statuses they were processed in parallel to attempt to keep the maximum latency at 2 seconds, but often resulted in a timeline appearing to hang during loading due to a URL that could not be successfully reached. URLs which had images links that expire (Amazon AWS) were parsed and inserted with a TTL to ensure the image link would not break. Rich Media data is now cached in the database and fetched asynchronously. Cachex is used as a read-through cache. When the data becomes available we stream an update to the clients. If the result is returned quickly the experience is almost seamless. Activities were already processed for their Rich Media data during ingestion to warm the cache, so users should not normally encounter the asynchronous loading of the Rich Media data. Implementation notes: - The async worker is a Task with a globally unique process name to prevent duplicate processing of the same URL - The Task will attempt to fetch the data 3 times with increasing sleep time between attempts - The HTTP request obeys the default HTTP request timeout value instead of 2 seconds - URLs that cannot be successfully parsed due to an unexpected error receives a negative cache entry for 15 minutes - URLs that fail with an expected error will receive a negative cache with no TTL - Activities that have no detected URLs insert a nil value in the Cachex :scrubber_cache so we do not repeat parsing the object content with Floki every time the activity is rendered - Expiring image URLs are handled with an Oban job - There is no automatic cleanup of the Rich Media data in the database, but it is safe to delete at any time - The post draft/preview feature makes the URL processing synchronous so the rendered post preview will have an accurate rendering Overall performance of timelines and creating new posts which contain URLs is greatly improved.	2024-06-09 17:33:48 +01:00
Haelwenn (lanodan) Monnier	0c2f200b4d	ReceiverWorker: Make sure non-{:ok, _} is returned as {:error, …} ... Otherwise an error like `{:signature, {:error, {:error, :not_found}}}` ends up considered a success. Cherry-picked-from: a299ddb10e	2024-04-21 20:58:06 +02:00
Floatingghost	370576474c	only consider :op and :id args in duplicate checks	2024-04-19 11:39:27 +01:00
Floatingghost	d2cee15c15	mix format says no	2024-04-16 03:07:28 +01:00
Floatingghost	d70fa16383	oban options should be a keyword list	2024-04-16 02:58:50 +01:00
Floatingghost	5043571084	Enable oban job uniqueness ... by default just prevent job floods with a 1-seconds uniqueness check, but override in RemoteFetcherWorker for 5 minute uniqueness check over all states :infinity is an option we can go for maybe at some point, but that would prevent any refetches so maybe not idk.	2024-04-16 02:53:24 +01:00
Floatingghost	b7dd739de1	Make sure we return the right format for oban	2024-04-16 02:35:21 +01:00
Floatingghost	2fc25980d1	fix pattern matching in fetch errors	2024-04-13 23:55:26 +01:00
Mark Felder	2e369aef71	Allow the Remote Fetcher to attempt fetching an unreachable instance	2024-04-12 20:33:21 +01:00
Mark Felder	fed7a78c77	Oban jobs should be discarded on permanent errors	2024-04-12 20:33:17 +01:00
Mark Felder	ff515c05c3	Prevent requeuing Remote Fetcher jobs that exceed thread depth	2024-04-12 20:32:31 +01:00
Mark Felder	7e5004b3e2	Leverage existing atoms as return errors for the object fetcher	2024-04-12 20:32:13 +01:00
Mark Felder	e2b04fac5a	Skip remote fetch jobs for unreachable instances	2024-04-12 20:28:36 +01:00
Mark Felder	6d368808d3	Remove mistaken duplicate fetch	2024-04-12 20:28:31 +01:00
Mark Felder	132036f951	Cancel remote fetch jobs for deleted objects	2024-04-12 20:28:21 +01:00
Mark Felder	4c29366fe5	Mark instances as unreachable when returning a 403 from an object fetch ... This is a definite sign the instance is blocked and they are enforcing authorized_fetch	2024-04-12 20:27:33 +01:00
Oneric	1a7839eaf2	Prune old Update activities ... Once processed they serve no purpose anymore afaict. Therefor, lets prune them like other transient activities to not unnecessarily bloat the table.	2024-02-17 16:57:40 +01:00
floatingghost	6b882a2c0b	Purge Rejected Follow requests in daily task (#334) ... Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk> Reviewed-on: #334	2022-12-03 23:17:43 +00:00
floatingghost	db60640c5b	Fixing up deletes a bit (#327) ... Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk> Reviewed-on: #327	2022-12-01 15:00:53 +00:00
FloatingGhost	ee7059c9cf	Spin off imports into n oban jobs	2022-11-27 21:45:41 +00:00
floatingghost	2a1f17e3ed	and i yoink (#275) ... Co-authored-by: Mark Felder <feld@feld.me> Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk> Reviewed-on: #275	2022-11-14 15:07:26 +00:00
floatingghost	c1127e321b	Add configurable timeline per oban job (#273) ... Heavily inspired by https://git.pleroma.social/pleroma/pleroma/-/merge_requests/3777 Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk> Reviewed-on: #273	2022-11-13 23:55:51 +00:00
floatingghost	b7e8ce2350	Scrape instance nodeinfo (#251) ... Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk> Reviewed-on: #251	2022-11-06 22:49:39 +00:00
FloatingGhost	d3b9cfb03f	use :discard instead of cancel	2022-08-11 19:17:50 +01:00
floatingghost	1245141779	treat rejections in MRF as a reject in federator (#155) ... Reviewed-on: #155	2022-08-08 15:47:57 +00:00
Tusooa Zhu	f08241c8ab	Allow users to create backups without providing email address ... Ref: backup-without-email	2022-08-02 22:16:54 -04:00
Ekaterina Vaartis	7aebff799b	Fix meilisearch tests and jobs for oban	2022-06-29 20:49:45 +01:00
Ekaterina Vaartis	5def4a7d49	Use oban for search indexing	2022-06-29 20:49:45 +01:00
FloatingGhost	f2ff05637f	Allow for embedding object data in publish	2022-06-25 18:41:26 +01:00
lain	6e3df11693	Merge branch 'instance-deletion' into 'develop' ... AdminAPI: endpoint to delete all content from a remote instance See merge request pleroma/pleroma!3483	2021-11-15 14:24:13 +00:00
Alex Gleason	6a6e42c9bf	PollWorker defensive checks	2021-07-18 11:10:22 -05:00
Alex Gleason	0114754db2	MastodonAPI: Support poll notification	2021-07-17 22:19:38 -05:00
Alex Gleason	f67d00d12b	Add Instance.delete_users_and_activities/1 to delete all content from a remote instance	2021-07-17 14:06:35 -05:00
feld	008499f65a	Merge branch 'develop' into 'fix/2412-filters' ... # Conflicts: # CHANGELOG.md	2021-02-03 14:23:23 +00:00
feld	c3dd860a02	Merge branch 'fix/2449-scheduled-poll-bug' into 'develop' ... Fix for scheduled post with poll Closes #2449 See merge request pleroma/pleroma!3294	2021-02-03 14:22:23 +00:00
feld	6a2d3fb9a3	Merge branch 'fix/2473-purge-expired-activities-duplicates' into 'develop' ... fix and delete purge activities duplicates Closes #2473 See merge request pleroma/pleroma!3285	2021-02-02 17:22:23 +00:00
Alexander Strizhakov	0dc68c157f	fix for scheduled post with poll	2021-02-01 18:22:26 +03:00
Mark Felder	13d79c281f	Make attachment cleanup jobs a noop if the setting is disabled.	2021-01-28 14:42:20 -06:00
Alexander Strizhakov	6c987c7670	fix and delete purge activities duplicates	2021-01-28 16:57:41 +03:00
Alexander Strizhakov	39335d4251	fix for unique oban worker option	2021-01-28 16:57:03 +03:00
Alexander Strizhakov	875fbaae35	support for expires_in/expires_at in filters	2021-01-26 08:27:45 +03:00
Mark Felder	ebd7d1365b	Make setting user activation status explicit	2021-01-15 11:37:17 -06:00
Haelwenn (lanodan) Monnier	c4439c630f	Bump Copyright to 2021 ... grep -rl '# Copyright © .* Pleroma' * | xargs sed -i 's;Copyright © .* Pleroma .*;Copyright © 2017-2021 Pleroma Authors <https://pleroma.social/>;'	2021-01-13 07:49:50 +01:00
Mark Felder	86dcfb4eb9	More places we should be using Upload.base_url	2021-01-08 17:32:42 -06:00
Alexander Strizhakov	8d218ebaf5	Moving some background jobs into simple tasks ... - fetching activity data - attachment prefetching - using limiter to prevent overload	2020-11-11 13:39:49 +03:00
lain	dd2b3a8da9	Merge branch 'develop' of git.pleroma.social:pleroma/pleroma into feature/expire-mutes	2020-11-04 16:51:42 +01:00
Egor Kislitsyn	ad605e3e16	Rename Pleroma.Backup to Pleroma.User.Backup	2020-10-20 17:16:58 +04:00
Egor Kislitsyn	7c22c9afb4	Allow admins request user backups	2020-10-07 18:34:31 +04:00
Egor Kislitsyn	7fdd81d000	Add "Your backup is ready" email	2020-10-07 18:34:30 +04:00
Egor Kislitsyn	739cb1463b	Add backups deletion	2020-10-07 18:34:29 +04:00