Commit graph

5584 commits

Author SHA1 Message Date
ilja
b71db2f82d create_service_actor is now type Application
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
This is used for internal fetch and for relay. Both represent the instance and therefore are an aplication.
2023-02-04 21:00:21 +00:00
aeb68a0ad1 paginate follow requests (#460)
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
matches https://docs.joinmastodon.org/methods/follow_requests/#get mostly

Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #460
2023-02-04 20:51:17 +00:00
d394ab0a8a Merge branch 'develop' of akkoma.dev:AkkomaGang/akkoma into develop
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
2023-01-15 18:58:26 +00:00
90088cce11 Support TLD wildcards in MRF matches
Fixes #431
2023-01-15 18:57:49 +00:00
sfr
20cd8a0fc4 URL encode remote emoji pack names (#362)
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
fix #246

Co-authored-by: Sol Fisher Romanoff <sol@solfisher.com>
Reviewed-on: #362
Co-authored-by: sfr <sol@solfisher.com>
Co-committed-by: sfr <sol@solfisher.com>
2023-01-15 18:14:04 +00:00
975bc6d7e8 Merge pull request 'fix: Give error message to users when address has already been validated' (#435) from cheerfulstoic/akkoma:develop into develop
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
Reviewed-on: #435
2023-01-15 18:06:12 +00:00
f3c118ca23 Mix format
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
2023-01-15 18:00:03 +00:00
7ca9ce9d67 fix: Give error message to users when address has already been validated
Some checks failed
ci/woodpecker/pr/woodpecker Pipeline failed
Plus other errors.
2023-01-12 22:08:10 +01:00
ff5793198f add inbound language test
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
2023-01-11 15:42:13 +00:00
cc63a89b5d Fix tests 2023-01-10 10:29:17 +00:00
f86bf16430 Add language support on /api/v1/statuses 2023-01-10 10:29:17 +00:00
7695010268 Prune Objects --keep-threads option (#350)
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
This adds an option to the prune_objects mix task.
The original way deleted all non-local public posts older than a certain time frame.
Here we add a different query which you can call using the option --keep-threads.

We query from the activities table all context id's where
    1. the newest activity with this context is still old
    2. none of the activities with this context is is local
    3. none of the activities with this context is bookmarked
and delete all objects with these contexts.

The idea is that posts with local activities (posts, replies, likes, repeats...) may be interesting to keep.
Besides that, a post lives in a certain context (the thread), so we keep the whole thread as well.

Caveats:
* ~~Quotes have a different context. Therefore, when someone quotes a post, it's possible the quoted post will still be deleted.~~ fixed in #379
* Although undocumented (in docs/docs/administration/CLI_tasks/database.md/#prune-old-remote-posts-from-the-database), the 'normal' delete action still kept old remote non-public posts. I added an option to keep this behaviour, but this also means that you now have to explicitly provide that option. **This could be considered a breaking change!**
* ~~Note that this removes from the objects table, but not from the activities.~~ See #427 for that.

Some statistics from explain analyse:
(cost=1402845.92..1933782.00 rows=3810907 width=62) (actual time=2562455.486..2562455.495 rows=0 loops=1)
 Planning Time: 505.327 ms
 Trigger for constraint chat_message_references_object_id_fkey: time=651939.797 calls=921740
 Trigger for constraint deliveries_object_id_fkey: time=52036.009 calls=921740
 Trigger for constraint hashtags_objects_object_id_fkey: time=20665.778 calls=921740
 Execution Time: 3287933.902 ms

***
**TODO**
1. [x] **Question:** Is it OK to keep it like this in regard to quote posts? If not (ie post quoted by local users should also be kept), should we give quotes the same context as the post they are quoting? (If we don't want to give them the same context, I'll have to see how/if I can do it without being too costly)
    * See #379
2. [x] **Question:** the "original" query only deletes public posts (this is undocumented, but you can check the code). This new one doesn't care for scope. From the docs I get that the idea is that posts can be refetched when needed. But I have from a trusted source that Pleroma can't refetch non-public posts. I assume that's the reason why they are kept here. I see different options to deal with this
    1. ~~We keep it as currently implemented and just don't care about scope with this option~~
    2. ~~We add logic to not delete non-public posts either (I'll have to see how costly that becomes)~~
    3. We add an extra --keep-non-public parameter. This is technically speaking breakage (you didn't have to provide a param before for this, now you do), but I'm inclined to not care much because it wasn't documented nor tested in the first place.
3. [x] See if we can do the query using Elixir
4. [x] Test on a bigger DB to see that we don't run into a timeout
5. [x] Add docs

Co-authored-by: ilja <git@ilja.space>
Reviewed-on: #350
Co-authored-by: ilja <akkoma.dev@ilja.space>
Co-committed-by: ilja <akkoma.dev@ilja.space>
2023-01-09 22:15:41 +00:00
a8cd859ef9 Use actual ISO8601 timestamps for masto API (#425)
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
Some users post posts with spoofed timestamp, and some clients will have issues with certain dates. Tusky for example crashes if the date is any sooner than 1 BCE (“year zero” in the representation).

I limited the range of what is considered a valid date to be somewhere between the years 1583 and 9999 (inclusive).

The numbers have been chosen because:

- ISO 8601 only allows years before 1583 with “mutual agreement”
- Years after 9999 could cause issues with certain clients as well

Co-authored-by: Charlotte 🦝 Delenk <lotte@chir.rs>
Reviewed-on: #425
Co-authored-by: darkkirb <lotte@chir.rs>
Co-committed-by: darkkirb <lotte@chir.rs>
2023-01-09 22:12:28 +00:00
336d06b2a8 Significantly tighten HTTP CSP
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
2023-01-02 15:21:19 +00:00
6e646c4cbc Use a genserver to periodically fetch metrics
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
ci/woodpecker/pr/woodpecker Pipeline was successful
Ref https://github.com/beam-telemetry/telemetry_metrics_prometheus_core/issues/52
2023-01-01 18:32:14 +00:00
c4b46ca460 Add /api/v1/followed_tags
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
ci/woodpecker/pr/woodpecker Pipeline was successful
2022-12-31 18:09:34 +00:00
745e15468e Use same context for quote posts as the post that's being quoted (#379)
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
See #350 (comment)

When making quotes through Mast-API, they will now have the same context as the quoted post. This also results in them being showed when fetching the thread. I checked Misskey to see how it's there, and they show the quotes there as well, see e.g. <https://mk.toast.cafe/notes/98u1g0tulg>.

An example from Akkoma:

Co-authored-by: ilja <git@ilja.space>
Reviewed-on: #379
Reviewed-by: floatingghost <hannah@coffee-and-dreams.uk>
Co-authored-by: ilja <akkoma.dev@ilja.space>
Co-committed-by: ilja <akkoma.dev@ilja.space>
2022-12-31 18:09:27 +00:00
bf7ff6a337 Put rich media processing in a Task
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
2022-12-30 20:11:53 +00:00
9be6caf125 argon2 password hashing (#406)
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #406
2022-12-30 02:46:58 +00:00
a5e98083f2 Add link verification in profile fields (#405)
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #405
2022-12-29 20:56:06 +00:00
af7c3fab98 Do not crash on invalid atom in configDB
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
2022-12-21 00:16:39 +00:00
Atsuko Karagi
4a78c431cf Simplified HTTP signature processing
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
2022-12-19 20:41:48 +00:00
Atsuko Karagi
e17c71a389 Respect restrict_unauthenticated in /api/v1/accounts/lookup 2022-12-19 20:32:16 +00:00
c092fc9fd6 Add translation module for Argos Translate (#351)
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
Argos Translate is a Python module for translation and can be used as a command line tool.

This is also the engine for LibreTranslate, for which we already have a module.
Here we can use the engine directly from our server without doing requests to a third party or having to install our own LibreTranslate webservice (obviously you do have to install Argos Translate).

One thing that's currently still missing from Argos Translate is auto-detection of languages (see <https://github.com/argosopentech/argos-translate/issues/9>). For now, when no source language is provided, we just return the text unchanged, supposedly translated from the target language. That way you get a near immediate response in pleroma-fe when clicking Translate, after which you can select the source language from a dropdown.

Argos Translate also doesn't seem to handle html very well. Therefore we give admins the option to strip the html before translating. I made this an option because I'm unsure if/how this will change in the future.

Co-authored-by: ilja <git@ilja.space>
Reviewed-on: #351
Co-authored-by: ilja <akkoma.dev@ilja.space>
Co-committed-by: ilja <akkoma.dev@ilja.space>
2022-12-19 13:06:39 +00:00
3d546409b2 remove now-unused test
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
2022-12-17 23:21:24 +00:00
52d8183787 drop admin scopes on create app instead of rejecting 2022-12-17 23:14:49 +00:00
dcac8adb3d Add option to modify HTTP pool size
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
2022-12-16 18:33:00 +00:00
584f99b69d fix markdown link
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
2022-12-16 13:24:18 +00:00
372eea4e7c add changelog entry for custom emoji
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
2022-12-16 13:20:48 +00:00
20e3cb2b25 fix csp-induced HTML match error
Some checks failed
ci/woodpecker/push/woodpecker Pipeline was successful
ci/woodpecker/pr/woodpecker Pipeline failed
2022-12-16 12:19:24 +00:00
ca70d42541 mix format
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
2022-12-16 11:18:14 +00:00
48d302a60f allow disabling prometheus entirely
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
2022-12-16 11:17:04 +00:00
6d8e4d5e05 add test for metrics controller
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
2022-12-16 10:56:17 +00:00
c2054f82ab allow users with admin:metrics to read app metrics
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
2022-12-16 03:32:51 +00:00
b8be8192fb do not allow non-admins to register tokens with admin scopes
this didn't actually _do_ anything in the past,
the users would be prevented from accessing the resource,
but they shouldn't be able to even create them
2022-12-16 03:25:14 +00:00
07a48b9293 giant massive dep upgrade and dialyxir-found error emporium (#371)
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #371
2022-12-14 12:38:48 +00:00
duponin
3e9c0b380a
Return 413 when an actor's banner or background exceeds the size limit 2022-12-12 17:28:14 -05:00
duponin
c9304962c3
Uploading an avatar media exceeding max size returns a 413
Until now it was returning a 500 because the upload plug were going
through the changeset and ending in the JSON encoder, which raised
because struct has to @derive the encoder.
2022-12-12 17:28:09 -05:00
77e9a52450 allow http AS profile in ld+json header
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
2022-12-12 19:06:04 +00:00
9c71782861 Test removed HTTP adapter
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
2022-12-11 23:50:31 +00:00
f752126427 Remove quack, ensure adapter is finch
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
2022-12-11 23:22:35 +00:00
affc910372 Remove hackney/gun in favour of finch 2022-12-11 19:19:31 +00:00
68894089e8 Do not fetch anything from blocked instances
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
2022-12-10 00:09:45 +00:00
739ed14f54 Revert "mandate published on notes"
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
ci/woodpecker/pr/woodpecker Pipeline was successful
This reverts commit e49b583147.
2022-12-09 20:59:26 +00:00
e49b583147 mandate published on notes
Some checks failed
ci/woodpecker/push/woodpecker Pipeline is pending
ci/woodpecker/pr/woodpecker Pipeline failed
fixes #356
2022-12-09 20:27:54 +00:00
f5a315f04c Add URL and code to :not_found errors
Ref #355
2022-12-09 20:13:31 +00:00
9db4c2429f Remove FollowBotPolicy 2022-12-09 19:59:27 +00:00
6f83ae27aa extend reject MRF to check if originating instance is blocked 2022-12-09 19:57:29 +00:00
ilja
1f863f0a36 Fix MRF policies to also work with Update
Some checks failed
ci/woodpecker/pr/woodpecker Pipeline failed
Objects who got updated would just pass through several of the MRF policies, undoing moderation in some situations.
In the relevant cases we now check not only for Create activities, but also Update activities.

I checked which ones checked explicitly on type Create using `grep '"type" => "Create"' lib/pleroma/web/activity_pub/mrf/*`.

The following from that list have not been changed:
* lib/pleroma/web/activity_pub/mrf/follow_bot_policy.ex
    * Not relevant for moderation
* lib/pleroma/web/activity_pub/mrf/keyword_policy.ex
    * Already had a test for Update
* lib/pleroma/web/activity_pub/mrf/object_age_policy.ex
    * In practice only relevant when fetching old objects (e.g. through Like or Announce). These are always wrapped in a Create.
* lib/pleroma/web/activity_pub/mrf/reject_non_public.ex
    * We don't allow changing scope with Update, so not relevant here
2022-12-08 23:22:05 +01:00
ilja
ce517ff4e5 Fix tagpolicy to also work with Update
Some checks are pending
ci/woodpecker/pr/woodpecker Pipeline is pending
Objects who got updated would just pass the TagPolicy, undoing the moderation that was set in place for the Actor.
Now we check not only for Create activities, but also Update activities.
2022-12-08 21:53:42 +01:00