Commit graph

9033 commits

Author SHA1 Message Date
b88e6560e0 Reblog content should be ""
Fixes 
2023-03-02 11:04:27 +00:00
ilja
57eef6d764 prune_objects can prune orphaned activities who reference an array of objects
E.g. Flag activities have an array of objects

We prune the activity when NONE of the objects can be found

Note that the cost of finding and deleting these is ~4x higher than finding and deleting the non-array ones

Only string:
Delete on activities  (cost=506573.48..506580.38 rows=0 width=0)

Only Array:
Delete on activities  (cost=3570359.68..4276365.34 rows=0 width=0)

(They are still executed separately, so the total cost is the sum of the two)
2023-02-26 14:41:50 +01:00
ilja
a7ec6e039c prune_objects can prune orphaned activities
We add an option to also prune remote activities who don't have existing objects any more they reference.
Rn, we only check for activities who only reference one object, not an array or embeded object.
2023-02-26 14:41:50 +01:00
d3089ec399 Ensure we can update contentMap on update 2023-02-23 11:00:55 +00:00
ilja
b4952a81fe Interpret \n as newline for MFM
Markdown doesn't generally consider `\n` a newline,
but Misskey does for MFM.

Now we do to for MFM (and not for Markdown) :)
2023-02-18 19:56:11 +01:00
ilja
b71db2f82d create_service_actor is now type Application
This is used for internal fetch and for relay. Both represent the instance and therefore are an aplication.
2023-02-04 21:00:21 +00:00
aeb68a0ad1 paginate follow requests ()
matches https://docs.joinmastodon.org/methods/follow_requests/#get mostly

Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: 
2023-02-04 20:51:17 +00:00
d394ab0a8a Merge branch 'develop' of akkoma.dev:AkkomaGang/akkoma into develop 2023-01-15 18:58:26 +00:00
90088cce11 Support TLD wildcards in MRF matches
Fixes 
2023-01-15 18:57:49 +00:00
63ce25f32c Merge pull request 'Correct og:description tag in static-fe' () from sfr/akkoma:fix/og-description into develop
Reviewed-on: 
2023-01-15 18:15:20 +00:00
sfr
20cd8a0fc4 URL encode remote emoji pack names ()
fix 

Co-authored-by: Sol Fisher Romanoff <sol@solfisher.com>
Reviewed-on: 
Co-authored-by: sfr <sol@solfisher.com>
Co-committed-by: sfr <sol@solfisher.com>
2023-01-15 18:14:04 +00:00
7ca9ce9d67 fix: Give error message to users when address has already been validated
Plus other errors.
2023-01-12 22:08:10 +01:00
ff5793198f add inbound language test 2023-01-11 15:42:13 +00:00
78c44f31ca fix no-language-specified federation 2023-01-11 15:25:34 +00:00
22068f0853 fix unused variable warnings 2023-01-10 10:58:17 +00:00
cc63a89b5d Fix tests 2023-01-10 10:29:17 +00:00
f86bf16430 Add language support on /api/v1/statuses 2023-01-10 10:29:17 +00:00
7695010268 Prune Objects --keep-threads option ()
This adds an option to the prune_objects mix task.
The original way deleted all non-local public posts older than a certain time frame.
Here we add a different query which you can call using the option --keep-threads.

We query from the activities table all context id's where
    1. the newest activity with this context is still old
    2. none of the activities with this context is is local
    3. none of the activities with this context is bookmarked
and delete all objects with these contexts.

The idea is that posts with local activities (posts, replies, likes, repeats...) may be interesting to keep.
Besides that, a post lives in a certain context (the thread), so we keep the whole thread as well.

Caveats:
* ~~Quotes have a different context. Therefore, when someone quotes a post, it's possible the quoted post will still be deleted.~~ fixed in 
* Although undocumented (in docs/docs/administration/CLI_tasks/database.md/#prune-old-remote-posts-from-the-database), the 'normal' delete action still kept old remote non-public posts. I added an option to keep this behaviour, but this also means that you now have to explicitly provide that option. **This could be considered a breaking change!**
* ~~Note that this removes from the objects table, but not from the activities.~~ See  for that.

Some statistics from explain analyse:
(cost=1402845.92..1933782.00 rows=3810907 width=62) (actual time=2562455.486..2562455.495 rows=0 loops=1)
 Planning Time: 505.327 ms
 Trigger for constraint chat_message_references_object_id_fkey: time=651939.797 calls=921740
 Trigger for constraint deliveries_object_id_fkey: time=52036.009 calls=921740
 Trigger for constraint hashtags_objects_object_id_fkey: time=20665.778 calls=921740
 Execution Time: 3287933.902 ms

***
**TODO**
1. [x] **Question:** Is it OK to keep it like this in regard to quote posts? If not (ie post quoted by local users should also be kept), should we give quotes the same context as the post they are quoting? (If we don't want to give them the same context, I'll have to see how/if I can do it without being too costly)
    * See 
2. [x] **Question:** the "original" query only deletes public posts (this is undocumented, but you can check the code). This new one doesn't care for scope. From the docs I get that the idea is that posts can be refetched when needed. But I have from a trusted source that Pleroma can't refetch non-public posts. I assume that's the reason why they are kept here. I see different options to deal with this
    1. ~~We keep it as currently implemented and just don't care about scope with this option~~
    2. ~~We add logic to not delete non-public posts either (I'll have to see how costly that becomes)~~
    3. We add an extra --keep-non-public parameter. This is technically speaking breakage (you didn't have to provide a param before for this, now you do), but I'm inclined to not care much because it wasn't documented nor tested in the first place.
3. [x] See if we can do the query using Elixir
4. [x] Test on a bigger DB to see that we don't run into a timeout
5. [x] Add docs

Co-authored-by: ilja <git@ilja.space>
Reviewed-on: 
Co-authored-by: ilja <akkoma.dev@ilja.space>
Co-committed-by: ilja <akkoma.dev@ilja.space>
2023-01-09 22:15:41 +00:00
357f80a714 Merge pull request 'Changed references of "Pleroma" to "Akkoma" in email text' () from knova/akkoma:develop into develop
Reviewed-on: 
2023-01-09 22:13:45 +00:00
a8cd859ef9 Use actual ISO8601 timestamps for masto API ()
Some users post posts with spoofed timestamp, and some clients will have issues with certain dates. Tusky for example crashes if the date is any sooner than 1 BCE (“year zero” in the representation).

I limited the range of what is considered a valid date to be somewhere between the years 1583 and 9999 (inclusive).

The numbers have been chosen because:

- ISO 8601 only allows years before 1583 with “mutual agreement”
- Years after 9999 could cause issues with certain clients as well

Co-authored-by: Charlotte 🦝 Delenk <lotte@chir.rs>
Reviewed-on: 
Co-authored-by: darkkirb <lotte@chir.rs>
Co-committed-by: darkkirb <lotte@chir.rs>
2023-01-09 22:12:28 +00:00
13d943667e Changed references of "Pleroma" to "Akkoma" in email text
I know this is probably small peanuts in the grand scheme of things, but it bugged me when I was messing around with my own Akkoma instance.
2023-01-08 03:29:09 +00:00
f2b925f32c
exiftool doesn’t support JPEG XL either 2023-01-07 14:49:58 +01:00
b98fe4476c fix "exiftool not support svg files' ()
Faced with this issue today, Pleroma responds with status 400 (Bad request) if Exiftool.StripLocation is added to the list of filter modules for uploads. Here is logs:

```
13:27:25.201 [info] POST /api/v1/media

13:27:25.232 request_id=FzdspaAnrA6cyv0APgVR [error] Elixir.Pleroma.Upload.Filter: Filter Elixir.Pleroma.Upload.Filter.Exiftool.StripLocation failed: {:error, "Elixir.Pleroma.Upload.Filter.Exiftool.StripLocation: %ErlangError{original: :enoent}"}

13:27:25.232 request_id=FzdspaAnrA6cyv0APgVR [error] Elixir.Pleroma.Upload store (using Pleroma.Uploaders.Local) failed: "Elixir.Pleroma.Upload.Filter.Exiftool.StripLocation: %ErlangError{original: :enoent}"
```

# This fix solves this problem.

Reviewed-on: 
Co-authored-by: ihor <ikandreew@gmail.com>
Co-committed-by: ihor <ikandreew@gmail.com>
2023-01-05 15:22:48 +00:00
336d06b2a8 Significantly tighten HTTP CSP 2023-01-02 15:21:19 +00:00
57e51fe62c Migrate Pleroma.Web to phoenix 1.6 formats 2023-01-02 03:29:02 +00:00
6e646c4cbc Use a genserver to periodically fetch metrics
Ref https://github.com/beam-telemetry/telemetry_metrics_prometheus_core/issues/52
2023-01-01 18:32:14 +00:00
c4b46ca460 Add /api/v1/followed_tags 2022-12-31 18:09:34 +00:00
745e15468e Use same context for quote posts as the post that's being quoted ()
See 

When making quotes through Mast-API, they will now have the same context as the quoted post. This also results in them being showed when fetching the thread. I checked Misskey to see how it's there, and they show the quotes there as well, see e.g. <https://mk.toast.cafe/notes/98u1g0tulg>.

An example from Akkoma:

Co-authored-by: ilja <git@ilja.space>
Reviewed-on: 
Reviewed-by: floatingghost <hannah@coffee-and-dreams.uk>
Co-authored-by: ilja <akkoma.dev@ilja.space>
Co-committed-by: ilja <akkoma.dev@ilja.space>
2022-12-31 18:09:27 +00:00
b8f280b4b5 Rich media doesn't need to be a map 2022-12-31 03:53:52 +00:00
bf7ff6a337 Put rich media processing in a Task 2022-12-30 20:11:53 +00:00
Sol Fisher Romanoff
1d884fd914
Correct og:description tag in static-fe 2022-12-30 07:14:54 +02:00
5d4c291d52 update references to pleroma in docs 2022-12-30 03:43:35 +00:00
9be6caf125 argon2 password hashing ()
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: 
2022-12-30 02:46:58 +00:00
a5e98083f2 Add link verification in profile fields ()
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: 
2022-12-29 20:56:06 +00:00
5a405bdadf document dump_to_file and load_from_file 2022-12-29 20:00:04 +00:00
d1bf8aa9ed Add dump_to_file and load_from_file tasks 2022-12-29 19:56:35 +00:00
af7c3fab98 Do not crash on invalid atom in configDB 2022-12-21 00:16:39 +00:00
Atsuko Karagi
4a78c431cf Simplified HTTP signature processing 2022-12-19 20:41:48 +00:00
Atsuko Karagi
e17c71a389 Respect restrict_unauthenticated in /api/v1/accounts/lookup 2022-12-19 20:32:16 +00:00
c092fc9fd6 Add translation module for Argos Translate ()
Argos Translate is a Python module for translation and can be used as a command line tool.

This is also the engine for LibreTranslate, for which we already have a module.
Here we can use the engine directly from our server without doing requests to a third party or having to install our own LibreTranslate webservice (obviously you do have to install Argos Translate).

One thing that's currently still missing from Argos Translate is auto-detection of languages (see <https://github.com/argosopentech/argos-translate/issues/9>). For now, when no source language is provided, we just return the text unchanged, supposedly translated from the target language. That way you get a near immediate response in pleroma-fe when clicking Translate, after which you can select the source language from a dropdown.

Argos Translate also doesn't seem to handle html very well. Therefore we give admins the option to strip the html before translating. I made this an option because I'm unsure if/how this will change in the future.

Co-authored-by: ilja <git@ilja.space>
Reviewed-on: 
Co-authored-by: ilja <akkoma.dev@ilja.space>
Co-committed-by: ilja <akkoma.dev@ilja.space>
2022-12-19 13:06:39 +00:00
233c4bb3ba revert 28ab09d377
revert Remove unused dependencies
2022-12-19 02:34:46 +00:00
28ab09d377 Remove unused dependencies 2022-12-19 02:26:04 +00:00
3d546409b2 remove now-unused test 2022-12-17 23:21:24 +00:00
52d8183787 drop admin scopes on create app instead of rejecting 2022-12-17 23:14:49 +00:00
dcac8adb3d Add option to modify HTTP pool size 2022-12-16 18:33:00 +00:00
7b76fdeed3 update stats every 5 minutes 2022-12-16 17:22:56 +00:00
b91e671c0d add remote user count for the heck of it 2022-12-16 17:22:26 +00:00
1f5bc4d68a remove unused variable 2022-12-16 12:36:34 +00:00
9a320ba814 make 2fa UI less awful 2022-12-16 11:50:25 +00:00
48d302a60f allow disabling prometheus entirely 2022-12-16 11:17:04 +00:00