Commit graph

643 commits

Author SHA1 Message Date
ccf1007883 Fix about a million tests 2024-10-26 05:05:48 +01:00
fc99c694e6 Add signing key modules 2024-10-26 05:05:28 +01:00
1a4238bf98 cosmetic: fix concurrent index creation warnings
All checks were successful
ci/woodpecker/pr/lint Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/build-amd64 Pipeline was successful
ci/woodpecker/pr/build-arm64 Pipeline was successful
ci/woodpecker/pr/docs Pipeline was successful
Since those old migrations will now most likely only run during db init,
there’s not much point in running them in the background concurrently
anyway, so just drop the cncurrent setting rather than disabling
migration locks.
2024-06-19 02:25:23 +02:00
5da9cbd8a5 RichMedia refactor
Rich Media parsing was previously handled on-demand with a 2 second HTTP request timeout and retained only in Cachex. Every time a Pleroma instance is restarted it will have to request and parse the data for each status with a URL detected. When fetching a batch of statuses they were processed in parallel to attempt to keep the maximum latency at 2 seconds, but often resulted in a timeline appearing to hang during loading due to a URL that could not be successfully reached. URLs which had images links that expire (Amazon AWS) were parsed and inserted with a TTL to ensure the image link would not break.

Rich Media data is now cached in the database and fetched asynchronously. Cachex is used as a read-through cache. When the data becomes available we stream an update to the clients. If the result is returned quickly the experience is almost seamless. Activities were already processed for their Rich Media data during ingestion to warm the cache, so users should not normally encounter the asynchronous loading of the Rich Media data.

Implementation notes:

- The async worker is a Task with a globally unique process name to prevent duplicate processing of the same URL
- The Task will attempt to fetch the data 3 times with increasing sleep time between attempts
- The HTTP request obeys the default HTTP request timeout value instead of 2 seconds
- URLs that cannot be successfully parsed due to an unexpected error receives a negative cache entry for 15 minutes
- URLs that fail with an expected error will receive a negative cache with no TTL
- Activities that have no detected URLs insert a nil value in the Cachex :scrubber_cache so we do not repeat parsing the object content with Floki every time the activity is rendered
- Expiring image URLs are handled with an Oban job
- There is no automatic cleanup of the Rich Media data in the database, but it is safe to delete at any time
- The post draft/preview feature makes the URL processing synchronous so the rendered post preview will have an accurate rendering

Overall performance of timelines and creating new posts which contain URLs is greatly improved.
2024-06-09 17:33:48 +01:00
72871d4514 Merge pull request 'Drop unused indices' (#767) from Oneric/akkoma:purge-unused-indices into develop
Some checks are pending
ci/woodpecker/push/build-amd64 Pipeline is pending
ci/woodpecker/push/build-arm64 Pipeline is pending
ci/woodpecker/push/docs Pipeline is pending
ci/woodpecker/push/lint Pipeline is pending
ci/woodpecker/push/test Pipeline is pending
Reviewed-on: #767
2024-05-28 01:35:18 +00:00
5256678901 Fix Exiftool migration id
Some checks are pending
ci/woodpecker/pr/build-amd64 Pipeline is pending
ci/woodpecker/pr/build-arm64 Pipeline is pending
ci/woodpecker/pr/docs Pipeline is pending
ci/woodpecker/pr/lint Pipeline is pending
ci/woodpecker/pr/test Pipeline is pending
Applying works fine with a 20220220135625 version, but it won’t be
rolled back in the right order. Fortunately this action is idempotent
so we can just rename and reapply it with a new id.

To also not break large-scale rollbacks past 2022 for anyone
who already applied it with the old id, keep a stub migration.
2024-05-07 00:16:21 +02:00
b7e3d44756 Drop unused indices
Some checks failed
ci/woodpecker/pr/lint Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/build-arm64 unknown status
ci/woodpecker/pr/docs unknown status
ci/woodpecker/pr/build-amd64 unknown status
This promotes and expands our existing optional migration.
Based on usage statistics from several instances, see:
#764

activities_hosts is now retained after all since it’s essential
for the "instance" query parameter of *oma’s public timeline to
reliably work in a reasonable amount of time. (Although akkoma-fe has
no support for this feature and apparently barely anyone uses it.)

activities_actor_index was already dropped before in
20221211234352_remove_unused_indices; no need to drop it again.

Birthday indices were introduced in pleroma starting with
20220116183110_add_birthday_to_users which is past the
last common migration 20210416051708.
2024-05-02 00:08:33 +02:00
timorl
cd7af81896
Rename StripLocation to StripMetadata for temporal-proofing reasons
Some checks are pending
ci/woodpecker/pr/build-amd64 Pipeline is pending
ci/woodpecker/pr/build-arm64 Pipeline is pending
ci/woodpecker/pr/docs Pipeline is pending
ci/woodpecker/pr/lint Pipeline is pending
ci/woodpecker/pr/test Pipeline is pending
2024-04-16 20:37:00 +02:00
timorl
b144218dce
Merge branch 'develop' into elseinspe
Some checks failed
ci/woodpecker/pr/lint Pipeline failed
ci/woodpecker/pr/test unknown status
ci/woodpecker/pr/build-arm64 unknown status
ci/woodpecker/pr/build-amd64 unknown status
ci/woodpecker/pr/docs unknown status
2024-04-14 20:31:33 +02:00
376f6b15ca Add ability to auto-approve followbacks
Some checks are pending
ci/woodpecker/pr/build-amd64 Pipeline is pending
ci/woodpecker/pr/build-arm64 Pipeline is pending
ci/woodpecker/pr/docs Pipeline is pending
ci/woodpecker/pr/lint Pipeline is pending
ci/woodpecker/pr/test Pipeline is pending
Resolves: #148
2024-02-13 15:42:37 +01:00
8cf183cb42 Drop Chat tables
All checks were successful
ci/woodpecker/pr/lint Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/build-amd64 Pipeline was successful
ci/woodpecker/pr/build-arm64 Pipeline was successful
ci/woodpecker/pr/docs Pipeline was successful
Chats were removed in 0f132b802d
2024-02-11 05:15:08 +01:00
5c164028cf ensure ap_enabled true if coming back pleroma
Some checks are pending
ci/woodpecker/push/build-amd64 Pipeline is pending
ci/woodpecker/push/build-arm64 Pipeline is pending
ci/woodpecker/push/docs Pipeline is pending
ci/woodpecker/push/lint Pipeline is pending
ci/woodpecker/push/test Pipeline is pending
2023-08-16 23:11:36 +01:00
6139c3346d Add extra rollbacks to pleroma develop
Some checks are pending
ci/woodpecker/push/build-amd64 Pipeline is pending
ci/woodpecker/push/build-arm64 Pipeline is pending
ci/woodpecker/push/docs Pipeline is pending
ci/woodpecker/push/lint Pipeline is pending
ci/woodpecker/push/test Pipeline is pending
2023-08-16 22:49:23 +01:00
98cb255d12 Support elixir1.15
Some checks failed
ci/woodpecker/push/build-amd64 Pipeline is pending
ci/woodpecker/push/build-arm64 Pipeline is pending
ci/woodpecker/push/docs Pipeline is pending
ci/woodpecker/push/test Pipeline is pending
ci/woodpecker/pr/test Pipeline failed
ci/woodpecker/pr/build-amd64 unknown status
ci/woodpecker/pr/build-arm64 unknown status
ci/woodpecker/pr/docs unknown status
OTP builds to 1.15

Changelog entry

Ensure policies are fully loaded

Fix :warn

use main branch for linkify

Fix warn in tests

Migrations for phoenix 1.17

Revert "Migrations for phoenix 1.17"

This reverts commit 6a3b2f15b7.

Oban upgrade

Add default empty whitelist

mix format

limit test to amd64

OTP 26 tests for 1.15

use OTP_VERSION tag

baka

just 1.15

Massive deps update

Update locale, deps

Mix format

shell????

multiline???

?

max cases 1

use assert_recieve

don't put_env in async tests

don't async conn/fs tests

mix format

FIx some uploader issues

Fix tests
2023-08-03 17:44:09 +01:00
Weblate
eba3cce77b Update translation files
Some checks failed
ci/woodpecker/push/test Pipeline was successful
ci/woodpecker/push/build-arm64 Pipeline failed
ci/woodpecker/push/docs unknown status
ci/woodpecker/push/build-amd64 Pipeline failed
Updated by "Squash Git commits" hook in Weblate.

Translation: Pleroma fe/Akkoma Backend (Config Descriptions)
Translate-URL: http://translate.akkoma.dev/projects/akkoma/akkoma-backend-config-descriptions/
2023-07-27 13:14:05 +00:00
ilja
3a13f91fff Adapt some migrations so they can be rolled back
Some checks are pending
ci/woodpecker/pr/woodpecker Pipeline is pending
This is useful for people who want to migrate back to Pleroma.
It's also added in the docs, but also noted that this is barely tested and to be used at their own risk.
2023-06-09 22:02:26 +02:00
ab34680554 switch to using an enum system for DM acceptance 2023-05-23 10:29:08 +01:00
d310f99d6a Add MRFs for direct message manipulation 2023-05-22 23:53:44 +01:00
ilja
8b2adc4fb4 Rename users_ap_id_COALESCE_follower_address_index for faster db restoration
By default Postgresql first restores the data and then the indexes when dumping and restoring the database.
Restoring index activities_visibility_index took a very long time.
users_ap_id_COALESCE_follower_address_index was later added because having this could speed up the restoration tremendously.
The problem now is that restoration apparently happens in alphabetical order, so this new index wasn't created yet
by the time activities_visibility_index needed it.
There were several work-arounds which included more complex steps during backup/restore.
By renaming this index, it should be restored first and thus activities_visibility_index can make use of it.
This speeds up restoration significantly without requiring more complex or unexpected steps from people.
2023-02-26 14:33:17 +01:00
ilja
b71db2f82d create_service_actor is now type Application
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
This is used for internal fetch and for relay. Both represent the instance and therefore are an aplication.
2023-02-04 21:00:21 +00:00
9c71782861 Test removed HTTP adapter
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
2022-12-11 23:50:31 +00:00
d55de5debf Remerge of hashtag following (#341)
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
this time with less idiot

Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #341
2022-12-05 12:58:48 +00:00
ec6bf8c3f7 revert 4a94c9a31e
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
revert Add ability to follow hashtags (#336)

Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #336
2022-12-04 20:04:09 +00:00
4a94c9a31e Add ability to follow hashtags (#336)
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #336
2022-12-04 17:36:59 +00:00
db60640c5b Fixing up deletes a bit (#327)
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #327
2022-12-01 15:00:53 +00:00
0cfd5b4e89 Add ability to set a default post expiry (#321)
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #321
2022-11-28 13:34:54 +00:00
a90c45b7e9 Add Signed Fetch Statistics (#312)
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
Close #304.

Notes:
 - This patch was made on top of Pleroma develop, so I created a separate cachex worker for request signature actions, instead of Akkoma's instance cache. If that is a merge blocker, I can attempt to move logic around for that.
 - Regarding the `has_request_signatures: true -> false` state transition: I think that is a higher level thing (resetting instance state on new instance actor key) which is separate from the changes relevant to this one.

Co-authored-by: Luna <git@l4.pm>
Reviewed-on: #312
Co-authored-by: @luna@f.l4.pm <akkoma@l4.pm>
Co-committed-by: @luna@f.l4.pm <akkoma@l4.pm>
2022-11-26 19:22:56 +00:00
e0032e4799 Add rollbacks for associated_object_id
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
2022-11-07 00:08:20 +00:00
b7e8ce2350 Scrape instance nodeinfo (#251)
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #251
2022-11-06 22:49:39 +00:00
Ilja
338612d72b Use EXIF data of image to prefill image description
During attachment upload Pleroma returns a "description" field.

* This MR allows Pleroma to read the EXIF data during upload and return the description to the FE using this field.
    * If a description is already present (e.g. because a previous module added it), it will use that
    * Otherwise it will read from the EXIF data. First it will check -ImageDescription, if that's empty, it will check -iptc:Caption-Abstract
    * If no description is found, it will simply return nil, which is the default value
* When people set up a new instance, they will be asked if they want to read metadata and this module will be activated if so

There was an Exiftool module, which has now been renamed to Exiftool.StripLocation
2022-10-23 14:46:16 +02:00
c6e63aaf6b Backend settings sync (#226)
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #226
2022-10-06 16:22:15 +00:00
ee2eb7752d Ensure rollback succeeds
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
2022-09-16 13:00:40 +01:00
911f8feb0a Ensure migrations succeed (#216)
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #216
2022-09-16 11:53:11 +00:00
b6891fe190 Migrations: generate unset user keys
User keys are now generated on user creation instead of "when needed",
to prevent race conditions in federation and a few other issues. This
migration will generate keys missing for local users.
2022-09-11 19:53:31 +01:00
2641dcdd15 Post editing (#202)
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Rebased from #103

Co-authored-by: Tusooa Zhu <tusooa@kazv.moe>
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #202
2022-09-06 19:24:02 +00:00
8e4de118c1 Don't persist local undone follow (#194)
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
same deal but backwards this time

Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #194
2022-08-31 18:00:36 +00:00
62e179f446 make conversation-id deterministic (#154)
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Reviewed-on: #154
2022-08-06 20:59:15 +00:00
0ec3a11895 don't persist undo of follows (#149)
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Reviewed-on: #149
2022-08-05 13:28:56 +00:00
c9600dbbbf local-only-fixed (#138)
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Reviewed-on: #138
2022-08-02 14:46:46 +00:00
54ed8760ff Merge branch 'from/upstream-develop/tusooa/server-announcements' into 'develop' (#85)
Some checks are pending
ci/woodpecker/push/docs Pipeline is pending
ci/woodpecker/push/release Pipeline is pending
ci/woodpecker/push/test Pipeline is pending
Reviewed-on: #85
2022-07-18 13:08:36 +00:00
3cbc401fe0 upgrade oban to v11 (#84)
All checks were successful
ci/woodpecker/push/test Pipeline was successful
ci/woodpecker/push/docs Pipeline was successful
ci/woodpecker/push/release Pipeline was successful
Reviewed-on: #84
2022-07-18 10:48:49 +00:00
Pete
56203aae30 Add index hotspots
Some checks failed
ci/woodpecker/push/release Pipeline was successful
ci/woodpecker/push/lint Pipeline was successful
ci/woodpecker/pr/lint Pipeline was successful
ci/woodpecker/pr/release Pipeline was successful
ci/woodpecker/pr/test Pipeline failed
ci/woodpecker/push/test Pipeline failed
2022-07-01 13:59:26 +01:00
Tusooa Zhu
7726148472 Send emails i18n'd using backend-stored user language 2022-06-29 20:45:19 +01:00
d884087e7c re-add mastofe_settings column 2022-01-08 21:47:52 +00:00
0d012ebea1 Revert "Merge branch 'remove/mastofe' into 'develop'"
This reverts commit 6b3842cf50, reversing
changes made to 6b1282a829.
2022-01-08 21:44:37 +00:00
Ivan Tashkinov
a7bdefc208 mix format 2021-12-29 11:44:33 +03:00
Ivan Tashkinov
0c7fb520bf Added index on [:target_id, :relationship_type] to :user_relationships (speeds up Notification.exclude_blockers/_). 2021-12-29 11:41:21 +03:00
Alex Gleason
fa35e24a5e
Apps: add user_id index 2021-12-27 18:05:35 -06:00
Alex Gleason
f5c3d45120
Merge remote-tracking branch 'origin/develop' into apps-api-endpoint 2021-12-27 18:01:25 -06:00
marcin mikołajczak
de006443f0 MastoAPI: Profile directory 2021-12-26 02:35:17 +00:00