As discussed on IRC this is already buggy on large instnaces
and conflicts with future cleanup and optimisation work.
The only remaining users currently appear to be Pleroma-derived web
frontends, namely akkoma-fe, pleroma-fe, pl-fe and Mangane.
We will switch akkoma-fe over to conversations before the actual
removal.
This was originally added in a97b642289 as
part of https://git.pleroma.social/pleroma/pleroma/-/merge_requests/1818
with the hope/intent of using it to hide DMs by default from timelines.
This never came to be with thisd parameter still being unused today in
al of akkoma-fe, pleroma-fe, Husky, pl-fe, Mangange, fedibird-fe and
masto-fe. Given thypically only a small percentage of posts are DMs,
as well as issues with the SQL function and index used to implement this
a removal likely won't bother anyone while allowing us to clean up
bugged code and improve actually used functions.
While unlikely, we don't have a way to ascertain for sure whether there
are still users so for now only drop it from API Spec which leads to
the parameter being stripped out before processing and keep the actual
logic. If any users exist and complain we can easily revert this, if not
we will follow up with a proper cleanup.
This reduces both the size the indexes take on the disk
and the overhead of maintaining it since it now only
needs to be updated for some entries.
For the email index restriction queries needed to be updated,
the last_active_at index is used only by one query which already
also restricted by "local" and for the remaining restrictions
no query adaptations are needed.
This was replaced by the always-on exclusion from the inspect protocol
implemented in the previous commit which is far more robus.
This partially reverts commit 2b4b68eba7;
the more detailed response for generic HTTP errors is retained.
Duped code just means double the chance to mess up. This would have
prevented the leak of confidential info more minimally fixed in
6a8b8a14999f3ed82fdaedf6a53f9a391280df2f and now furthermore
fixes the representation of Update activites which _need_ to have their
object inlined, as well as better interop for follow Accept and Reject
activities and all other special cases already handled in Transmogrifier.
It also means we get more thorough tests for free.
This also already adds JSON-LD context and does not add bogus Note-only
fields as happened before due to this views misuse of prepare_object
for activities. The doc of prepare_object clearly states it is only
intended for creatable objects, i.e. (for us) Notes and Questions.
When the object associated with the activity was preloaded
(which happens automatically with Activity.normalize used in the
controller) Object.normalize’s "id_only" option did not actually work.
This option and it’s usage were introduced to fix display of Undo
activities in e88f36f72b.
For "Undo"s (and "Delete"s) there is no object preloaded
(since it is already gone from the database) thus this appeared
to work and for the particular case considered there in fact did.
Create activities use different rendering logic and thus remained
unaffected too.
However, for all other types of Activities (yes, including Update
which really _should_ include a properly sanitised, full object)
this new attempt at including "just the id", lead to it instead
including the full, unsanitised data of the referenced object.
This is obviously bad and can get worse due to access restrictions
on the activity being solely performed based on the addressing
of the activity itself, not of the (unintentionally) embedded
object.
Starting with the obvious, this leaks all "internal" fields
but as already mentioned in 8243fc0ef4
all current "internal" fields from Constants.object_internal_fields
are already publicised via MastoAPI etc anyway. Assuming matching
addressing of the referenced object and activity this isn't problematic
with regard to confidentiality.
Except, the internal "voters" field recording who voted for a poll
is currently just omitted from Constants.object_internal_fields
and indeed confidential information (fix in subsequent commit).
Fortunately this list is for the poll as a whole and there are no
inlined lists for individual choices. While this thus leaks _who_
voted for a poll, it at least doesn't directly expose _what_ each voter
chose if there are multiple voters.
As alluded to before, the access restriction not being aware
of the misplaced object data into account makes the issue worse.
If the activity addressing is not a subset of the referenced object’s
addressing, this will leak private objects to unauthorised users.
This begs the question whether such mismatched addressing can occur.
For remote activities the answer is ofc a resounding YES,
but we only serve local ActivityPub objects and for the latter
it currently(!) seems like a "no".
For all intended interactions, the user interacting must already have
access to the object of interest and our ActivityPub Builder
already uses a subset of the original posts addressing for
posts not publicly accessible. This addressing creation logic
was last touched six years ago predating the introduction of this
exposure blunder.
The rather big caveat her being, until it was fixed just yesterday in
dff532ac72 it was indeed possible to
interact with posts one is not allowed to actually see. Combined, this
allowed unauthorised access to private posts. (The API ID of such
private posts can be obtained e.g. from replies one _is_ allowed to see)
During the time when ActivityPub C2S was supported there might have been
more ways to create activities with mismatched addressing and sneak a
peek on private posts. (The AP id can be obtained in an analogous way)
Replaces and fixes e88f36f72b.
Since there never were any users of the
bugged "id_only" option it is removed.
This was reported by silverpill <silverpill@firemail.cc> as an
ActivityPub interop issue, since this blunder of course also
leads to invalid AP documents by adding an additional layer
in form of the "data" key and directly exposing the internal
Pleroma representation which is not always identical to valid AP.
Fixes: #1017
In our CI we sometimes fail assert_receive statmenets even in retry
making CI results unreliabl'ish. The default is 100ms, now it is 1s.
See e.g: https://ci.akkoma.dev/repos/1/pipeline/504/5
It was mistakenly removed as part of the bundled frontend in
82fa766ed7 but actually it
is used as the default value for user profile banners.
Mastodon API typespecs define the relevant field as non-nullable in
585545d0d5/app/javascript/mastodon/api_types/accounts.ts (L30)
and clients do in fact rely on it being non-null, meaning we can't just
omit the value if a user didn't setup a bespoke banner.
Signed-off-by: nicole mikołajczyk <git@mkljczk.pl>
For future use. Mark all current frontends as first-party.
The key is named 'blind_trust' instead of 'first_party' to (hopefully)
increase the chances someone notices if some random, malicous frontend
adds the key into their install instructions
It doesn't make sense to like, react, reply, etc to something you cannot
see and is unexpected for the author of the interacted with post and
might make them believe the reacting user actually _can_ see the post.
Wrt to fav, reblog, reaction indexes the missing visibility check was
also leaking some (presumably/hopefully) low-severity data.
Add full-API test for all modes of interactions with private posts.
If "thread containment" isn’t skipped the API will only return posts
whose entire ancestor chain is also visible to the current user.
For example if this contianment is active and the current user
follows A but not B and A replies to a followers-only post of B
the current user will not be able to see A’s post eventhough
per ActivityPub-semantics (an behaviour of other implementations)
the current user was addressed and has read permissions to A’s post.
(Though this semantics frequently surprise some users.)
There is a user-level option to control whether or not to perform
this kind of filtering and a server-wide config option under :instance.
If this containment is _disabled_ (i.e. skip_thread_containment: true)
server-wide, user-level options are ignored an filtering never takes
place.
This is cheked via the database function "thread_visibility" which
recursively calls itself on the ancestor of the currently inspected post
and for each post performs additional queries to check the follow
relationship between author and current user.
While this implementation introduced in
https://git.pleroma.social/pleroma/pleroma/-/merge_requests/971
performs better than the previous elixir-side iteration due to
less elixir-database roundtrips the perf impact is still ridiculously
large and when fetching and entire conversation / context at once there
are many redundnat checks and queries.
Due to this an option to dis/enable the "containment" was added and
promptly defaulted to disabling "containment" in
593b8b1e6a six years ago.
This default remained unchanged since and the implementation wasn’t
overhauled for improved performance either. Essentially this means
the feature has already been entirely disabled oout-of-the box
without too much discoverability for the last six years. It is thus
not too unlikely that no actual users of it exist today.
The user-level setting also didn’t made its way into any known clients.
Surveying current versions of akkoma-fe, husky, pleroma-fe, pl-fe,
Mangane and just to be sure also admin-fe, fedibird-fe and masto-fe
none of them appears to expose a control for the user-level setting.
pl-fe’s pl-api acknowledges the existance of the setting in the API
definition but the parameter appears unused in any actual logic.
Similarly Mangane and pl-fe have a few matches for
"skip_thread_visibility" in test samples of user setting responses
but again no actual uses in active code.
While the idea behind the feature is sensible (though care must be taken
to not mislead users into believing _all_ software would apply the same
restrictions!), the current implementation is very much not sensible.
With the added code complexity and apparent lack of users it is unclear
whether keeping the feature around and attempting to overhaul the
implementation is even worth it.
Thus start pritning a big fat warning for any potentially existing users
prompting for feedback. If there are no responses two releases from now
on it will presumably be safe to just entirely drop it.
It only served for a niche, admin nice-to-have informational stat
without too much value but was unreasonably costly to maintain
adding overhead with multiple queries added to all modifications
to the fairly busy activities table.
The database schema of the counter table and the activity_visibility function
used for counter updates also did not know about "local" visibility (nor the
recently removed "list" visibility) and misattributed them to the "direct" counter.
On my small instance this nearly halved the average
insert time for activiteis from 0.926 ms to 0.465 ms.
No known client ever used this. Currently among akkoma-fe, pleroma-fe,
Husky, Mangane and pl-fe only the latter acknowledes the existence of
the in_reply_to_conversation_id paramter in its API definitions,
but even pl-fe does never actually use the parameter anywhere.
Since the API parameter already was converted to DMs internally,
we do not need to make special considerations for already existing
old conversation-addressed posts. Since they share the context they
should also continue to show up in the intended thread anyway.
The pleroma.participants subkey mentioned in docs did already not exist
prior to this commit. Instead the accounts key doesn’t automatically update
and this affects conversations retrieved from the Mastodon API endpoint too
(which may be considered a pre-existing bug).
If desired clients can already avoid unintended participant additions
by using the explicit-addressing feature originally introduced in
https://git.pleroma.social/pleroma/pleroma/-/merge_requests/1239.
With the above-mentioned feature/bug of conversation participants
not updating automatically it can replace almost everything
conversation addressing was able to do. The sole exception being
creating new non-reply posts in the same context.
Neither conversation addressing nor explicit addressing
achieves robust, federated group chats though.
Resolves: #812
This feature was both conceptually broken and through bitrotting
the implementation was also buggy with the handling of certain
list-post interactions just crashing.
Remote servers had no way to know who belongs to a list and thus
posts basically showed just up as weird DM threads with different
participants on each instance. And while on the local instance
addition and removal from a listed grated and revoked post
access retroactively, it never acted retroactively on remotes.
Notably our "activity_visibility" database function also didn’t
know about "list visibility" instead treating them as direct messages.
Furthermore no known client actualy allows creating such messages
and the lack of complaints about the accumulutaed bugs supports
the absence of any users.
Given this there seems no point in fixing the implementation.
To reduce complexity of visibility handling it will be dropped instead.
Note, a similar effect with less federation weirdness can already be achieved
client-side using the explicit-addressing feature originally introduced in
https://git.pleroma.social/pleroma/pleroma/-/merge_requests/1239.
Ref: #812
While *oma, *key, GtS and even Mastodon federate a full media type for attachments,
posts from Bridgy only contain a generic type and the URLs also appear to never end
with a file exstension. This lead to our old type detection always classifying them
as "unknown" and it showing up like a generic document attachment in frontends.
We can (for vanilla Masto API clients) avoid this by falling back to the
federated generic type.
(Note: all other software mentioned at the start appears to always use "Document"
for the generic type of attachments regardless of the also federated actual full type)
For clients relying on the full mime type provided by an *oma extension,
like currently akkoma-fe, this in itself does not fix the display
but it is a necessary prerequisite to handling this more gracefully.
A MIME type MUST always contain both the type and subtype part.
Also, we already add this binary type for new incoming attachments
without a pre-existing MIME type entry anyway.
This caches query results from our own database, but does not respect
transaction boundaries or syncs to transaction success/failure. Long
lifetimes increase the chance of such desync occuring and being written
back to the database, see: #956
Until 1d02a9a35d recently fixed it
this used a 3 second lifetime anyway, so this won’t result in
performance degradations but hopefully prevents a rise in desyncs.
This avoids the importing the heavy Gettext backend in some places
and makes it clearer what’s actually used in the project and what’s
used by the Gettext library.
With the to-be-pulled-in Gettext API change this split
will be even more helpful for code clarity.
As a bonus documentation is improved and
the unused locale_or_default function removed.
Only the default parameters changed from 3.x to 4.x.
It now matches the proposed defaults suggested in the RFC
for constrained environments.
The prior defaults have worse latency, but probably slightly better
security while spawning fewer threads, so let’s stick with them.
By setting the defaults in the config instance owners
can (continue to) tweak these for the specific setup.
Or at least they were supposed to be compatible.
Due to an ecto optimisation change it is now illegal to use
replace_all_except if the union of conflict_targets and the fields
exempted from updating are equal to _all_ fields of this table.
See: https://github.com/elixir-ecto/ecto/issues/4633
Mastodon 3.3 added support for temproary mutes but uses "duration"
instead of our older "expires_in". Even Husky only sets "duration"
nowadays.
Signed-off-by: marcin mikołajczak <git@mkljczk.pl>
Cherry-picked-from: 5d3d6a58f7
Of course the aprent post might still be deleted after the reply was
already created, but in this case the reply will still show up as a
reply and be federated as a reply with a reference to the parent post.
If the parent was already deleted before the reply gets created however
it used to be indistinguishable from a root post both in Masto API and
ActivityPub.
From a UX perspective, users likely will like to know if the post
they’re replying to no longer exists by the time they finished writing.
The natural language error will show up in akkoma-fe without clearing
the post form, meaning users can decide to discard the reply or copy it
to post as a new root post. It seems sensibly to for other clients to
behave like this too, but so far no more clients were actually tested.
Furthermore, this used to allow replying to all sorts of activities not
just posts which was rather non-sensical (and after all processsing
steps turned into a reply to the object referenced by the activity).
In particular this allowed replying to an user object by specifying the
db ID of a follow request activity (if the latter was somehow obtained).
Note: empty-string in_reply_to parameters are explicitly ignored since
45ebc8dd9a to workaround one buggy client;
see: https://git.pleroma.social/pleroma/pleroma/-/issues/355.
It’s not clear if this workaround is still necessary,
but it is preserved by this commit.
Resolves: #522
While the function signature allows returning many errors at once,
OpenApiSpex.cast_and_validate currently only ever returns the first
invalid field it encounters. Thus we need to retry multiple times to
clean up all offenders.
Fixes: #992 (comment)
Markdown requires an indentation of 4 for a following paragraph to
continue a list item. Previously, the continuing paragraphs were only
indented by 2 spaces, leading to the list being interrupted and
numbering restarted each time.
No caller of `reload` actually uses the result in any way
so there’s no need to wait for a response and risk running
into a timeout (by default 5 seconds).
Discovered-by: sn0w <me@sn0w.re>
Based-on: 1fb54d5c2c
Reloading the entire emoji set from disk, reparsing all pack JSON files,
etc is unnecessarily costly for large emoji sets. We already know which
single or few changes we want to apply, so do just that instead.
No caller cares about the order
(and although, rare with concurrent reads at same time like a write
the table might return unordered results anyway).
Unordered sets have a constant read time,
ordered sets logarithmic times, but there’s no benfit for us
Display will fail for all but Create and Announce anyway since
0c9bb0594a. We exclude Announce
activities from redirects here since they are not identical
with the announced post and akkoma-fe stripping the repeat header
on he /notice/ page might lead to confusion about which is which.
In particular those redirects exiting breaks the assumptions from
the above commit’s commit message and made it possible to obtain
database IDs for activities other than one’s own likes allowing
slightly more mischief with the rendering bug it fixed.
Note: while 0c9bb0594a speculated about
public likes also leaking IDs to other users, the public like endpoint
is actually paginated by post id/date not like id/date like the private
endpoint. Thus it does not allow getting database IDs of others’ likes.
Happens commonly for e.g. replies to follower-only posts
if no one one your instance follows the replied-to account
or replies/quotes of deleted posts.
Before this change Masto API response would treat those
replies as root posts, making it hard to automatically or
mentally filter them out.
With this change replies already show up sensibly as
recognisable replies in akkoma-fe.
Quotes of unavailable posts however still show up as if they
weren’t quotes at all, but this can only be improved client-side.
Fixes: #715
They do nothing. As documented[1] only three specific
options regarding timeouts are parsed for individual request
and none of them is set by AdapterHelper, only pool-specific options.
In particular this means we always relied on Mint’s default CA cert
verification based on queries to the CAStore package (which we include).
[1]: https://hexdocs.pm/finch/Finch.html#request/3-options
It required a bunch of and even call-specific boilerplate
and is not necessary since we can just capture the real logger
as laready done in other tests.
While its data was included in healthcheck responses,
it was not used to determine the healthy status
and for informational purposes Prometheus metrics,
ObanWeb dashboard or the Phoenix live dashboard are all better fits.
In particular, the data shown in healtcheck responses had no temporal
information, but there’s quite a difference between X failures scattered
across many days of uptime and X failures within a couple minutes.
If the id of another activity type was used
it would show the post referenced by the activity
but wrongly attributing it to the activity actor
instead of the actual author.
E.g. ids of like activities can be obtained from
pagiantion info of the favourites endpoint.
For all other activity types the id would need to be guessed
which is considered practically infeasible for Flake UUIDs.
This should’ve been mostly harmless in practice, since:
- since the activity has the same context as the original post,
both the original and misattributed duplicate will show up in the
same thread
- only posts liked by a user can be misattributed to them,
presumably making it hard/impossible to associate someone with
content they disagree with
- by default only the liking user themself can view their like history
and therefore obtain IDs for their like activities.
Notably though, there is a user seting to allow anyone to browse
ones like history and therefore obtain like IDs. However, since
akkoma-fe has no support for actually displaying those, there might
be no actual users of this features.
This is more intuitive for users.
On the flip side, this makes the API more confusing
since now min_id/max_id no longer correspond to ids returned in the
response and only link headers can be used to traverse all response
pages. However, Mastodon already does this according to its
documentation, so clients should already handle this well.
The only other usage of get_follow_requests_query is only interested
in the total count of requests (from active users), thus changing its
select part is safe.
Also gets rid of (outside tests unused) User.get_follow_requests.
Fixes: #380
The old approach required adding a special virtual field
to any table potentially needing such foreign-id pagination and
also still required manually sorting according to pagiantion settings
since the pagination helper does not know whether
this virtual field was set or not.
Using lists with each entry containing the pagination id and the actual
entry insterad allows any table to use this mechanism unchanged and
does not require manually sorting.
Since it was unused, this also drops the pagination mode paramter from
fetch_favourited_with_fav_id.
Furthermore, as a side effect of this change a bug in the favourite
benchmark is fixed. It used to incorrectly attempt to use IDs of
the liked objects for pagination instead of the like IDs as advertised
in Link headers.
Split into scheduled (intentionally delayed until a later trigger date)
and available (eligible for immediate processing but did not yet start).
This will help in diagnosing overloaded instances or too-low queue
limits as well as expose configuration mishaps like
#924.
(The latter by violently crashing the telemetry poller process while
attempting put_in for a non-configured queue creating well visible logs)
Instancce stats are cached only renewed every 5 minutes anyway
and IO stats are cumulative over the entire runtime so no info
is lost.
Polling those every 10s is wasteful and the next commit will add a
periodic measurement which is (comparetively) more costly to compute.
Overall fairly minor changes, but most importantly the Host header passed to the backend is set to `$host` instead of `$http_host` as HTTP/3 no longer uses the Host header, leaving it unset. `$host` will fallback to the `server_name` value in that situation while `$http_host` doesn't.
Ref: https://community.nginx.org/t/proxy-params-by-default-blocks-reverse-proxy-for-http-3/6792
It’s only used in one place and there not even all of
its functionality is needed. It’s not only simpler and shorter,
but easier to understand if Tesla’s keyword list is just inlined.
The only useful bit which is now migrated to Pleroma.HTTP is
addition of the user-agent header (except, sometimes, in tests)
The web_push_encryption lib assumes HTTPoison semantics
which is why we also need to convert the header format.
Inspecting the libraries source shows that Tesla won’t
understand the options anyway and its only used to enable TLS/SSL.
When this was ported from Pleroma in
5da9cbd8a5
we did not take into acount that Akkoma’s and Pleroma’s
HTTP backend take different options.
There’s no need for the :pool option
and enforcing a body limit on download
is currently not possible with Finch
Ideally we’d use a single common HTTP request error format handling
for _all_ HTTP requests (including non-ActivityPub requests, e.g. NodeInfo).
But for the purpose of this commit this would create too much noise
and it is significant effort to go through all error pattern matches etc
too ensure it is still all correct or update as needed.
The Signature module now handles interaction with the HTTPSignature library
and the plug everything related to HTTP itself. It now also no longer needs to be public.
To achieve this signatures are now generated by a custom
Tesla Middleware placed after the FollowRedirects Middleware.
Any requests which should be signed needs
to pass the signing key via opts.
This also unifies the associated header logic between fetching and
publishing, notably resolving a divergence wrt the "host" header.
Relevant spec demands the host header shall include a port
identification if not using the protocols standard port.
Fixes: #731
While the previous version was already plenty fast for smaller instances
there seems to be a more-than-linear slowdown for large instances.
These prefilters should (hopefully) not exclude anything in need of
fixing, but seem to already cut down significantly on the overall
processing time.
The remote part is included in federated emoji names by e.g.
Iceshrimp.NET ever since remote emoji support was added in
4d21aa1670
and as of writing it still continues to do so.
It adds no value for us though; we add the remote part automatically
based on the URL and it makes it more difficult to correctly coalesce
the original reaction (from a user for whom the moji was local)
and the subsequent reactions with the identical emoji from users of
other instances. Additionally the remote part can cause issues when
later used with our REST API.
For non-reactions this is unproblematic and thus
there’s no need to change anything there.
Use a migration to fix up existing activities.
This will cause some (further) desync from the inlined reactions
array, but will be fixable with the resync mix task and avoids
issues when running the resync without first fixing existing activities.
This was already removed from mix.exs in
ea5a2a9f21
but as it turns out it was also re-set
during runtime.
Since we never set it outside of CI in
the first place there’s no need to
force-disable it here.
Even if --keep-threads is used, replies of
pinned posts might still be pruned as documented
for this option.
Thus keep-threads is no reason to skip reply counter recalculation.
ActivityPub spec demands each actor has at least an inbox and outbox.
Furthermore, the current representation wouldn’t even be accepted by
ourselves, since our processing requires objects to be flagged with a
sensible type else we don't know what to do with it.
Including the nickname is just a peemptive measure.
There were no reports of this causing problems in real-world deployments
and at least for federation with other Akkoma instances we should have
never run into this, since we _always_ expose the full representation of
the instance actor and atm also always use the latter for fetching
remote content (which prevents us from fetching followers-only content).
Nonetheless, serving something which violates spec and we wouldn’t even
accept ourselves seems obviously bad, so fix it and add tests to prevent
this from reoccuring.
Fixes bug introduced in 8f322456a0
Presumably those inlined copies were added to avoid the need for queries
each time the info is needed. However, they tend to desync from the actual
activities for not yet fully understood reasons; see:
#956
As a workaround until the root cause is identified and fixed and/or
we no longer rely on the inliend copies add a mix task to regenerate
the inlined "cache" from the authorative activity data.
Does not yet deal with inlined emoji reactions
since its format is a pain to deal with.
This emulates the previous effective behaviour of only
running tests on pull requests and skipping straight
to building artefacts and docs after merge.
Pull requests which do not pass tests shouldn’t be merged in the first place.
Most HTTPS requests actually fall into the single-digit millisecond
range or below on average. Even the more costly endpoints almost always
average around the lower third of the millisecond magnitude.
Only endpoints doing synchronous remote HTTP fetches (e.g. for signing
keys) occasionally spike into the order of seconds.
As is, the bucket resolution is completely unfit to reason about
anything and even just averages are better indications.
Most database queries take less than a millisecond and even in total
almost all take less than 50ms for me. Decode time is but a tiny
fraction of that and queue time usually only takes a small part of total
time too (but may spike on high load).
Shift the buckets down to be able to
give insight into all relevant cases.
In particular this allows to determine whether high averages
are the result of generally high processing times or just a few
outliers lifting the whole average up (e.g. slow network fetches).
Exact numbers are biased towards my setup for lack of other comparison
data, but at least the order of magnitude should be ok everywhere.
The old code was unnecessarily complicated, full of unused and/or
duplicated functions making it hard to understand what will actually
happen and for whom at runtime.
Since we only support a single HTTP backend this can be greatly simplified.
Now everything gets default options from a single place and only
functions to modify parts actually difffering across calls are exposed.
No HTTP3/QUIC support yet.
Note, allowing both here means we don't actually profit from HTTP2 multiplexing
due to Finch(? or maybe a dependency of Finch?) limitations. But it means we can
now interact with HTTP2-only instances (if such exist) and still may get minor
gains from header compression etc
Adventurous admins can change the config to allow only HTTP2,
thus profiting from multiplexing (but breaking federation with
HTTP1-only instances which are in fact observed to exist).
The next Tusky release is going to remove support for the v1 filters API,
see: https://codeberg.org/tusky/Tusky/pulls/5215.
Since Akkoma doesn't support the v2 API this
could cause significant issues for Akkoma users.
And improve monitoring documentation in particular
more detailed instructions for setting up Prometheus
metric scraping and the reference Grafana dashboard.
Currently translated at 0.1% (2 of 1004 strings)
Translated using Weblate (Turkish)
Currently translated at 100.0% (0 of 0 strings)
Added translation using Weblate (Turkish)
Co-authored-by: Anonymous <noreply@weblate.org>
Co-authored-by: Hasan Yıldız <hasanyildiz0@yaani.com>
Co-authored-by: Weblate <noreply@weblate.org>
Translate-URL: http://translate.akkoma.dev/projects/akkoma/akkoma-backend-config-descriptions/tr/
Translation: Pleroma fe/Akkoma Backend (Config Descriptions)
the :do_not_federate checks were omitted from the undo pipeline,
which could lead to them federating.
this commit enforces :outgoing_blocks on undos as well - and
any existing value of :do_not_federate is preserved, if one exists
Fixes#957
Resolves interop issue with a (reverted but possibly returning) bridgy change
as was reported in the comments of
#831.
This won't change anything for the problem originally reported there.
Notably we now always fetch the full collection (up to the configured
item count limit) instead of only using the first page if its link was
inlined.
If only steps are conditional the whole CI workflow
will be held up waiting until a slot is available to start them
just to then not do anything at all.
This allows us to drop "when" condition from individual steps
whenever it is now redundant with the top-level condition.
The lint pipeline spent ~7 minutes downloading and compiling
and only a few seconds actually checking the style.
The former is fully redundandt with what’s done during test anyway.
- This adds extra tests to be sure that scrubbing still happens.
- When doing this I notices that the htmlMfm key wasn't stored in the database when comming through the federator. This has been now been fixed too.
- We also test that values true, false or no attribute all work for incomming messages.
Previously all such requests led to '401 Unauthorized'
whih might have triggered retries.
Now, to not leak any MRF info, we just indicate an
accept for POST requests without actually processing the object
and indiscriminately return "not found" for GET requests.
Notably this change also now causes all signed fetch requests from
blocked domains to be rejected even if authorized_fetch isn’t enabled.
Fixes: #929
To make it usable in scenarios without a draft.
The next commit adds a user for the new function.
This does technically change behaviour a bit, since
"private" relies to "direct" messages no longer implicitly
address the parent post’s actor, but this seems like a contrived
scenario and was likely never intended to actually occur anyway
as cocorroborated by the absence of tests for it.
A pool timeout shorter than the receive timeout
makes race conditions leading to active connections
being killed more likely and laso just doesn’t make
much sense in general.
See: https://github.com/sneako/finch/pull/292
This was added in a924e117fd
with its name mirroring Finch’s own config option, but with
pool_timeout such a setting already existed since
2fe1484ed3 and the new one
was never actually used.
It may still crash due to a race condition between checking for file
existence and opening/streaming, but File.stream! has no safe version
we can use to avoid this completely.
Just not deleting such files during a reload is easy enough.
Until now only a limited number of self-replies were inlined as an
anonymous, unordered ActivityPub collection. Notably the advertised
replies might be private posts.
However, providing all (non-private) replies allows for better thread
consistency across instances if the remote server cooperates.
The collection existing as a stndalone object has two advantages
for this. For one, if it was still anonymous, _all_ replies would need
to be inlined, which might be too bloated in pathological cases.
Secondly, it allows remote servers to update the thread by traversing
the reply collection independent of the original post. (If the remote
part knows about chronological ordering, it can in theory
even efficiently resume from where it previously stopped)
An OrderedCollection uses orderedItems instead of the items key.
So far inlined orderedCollections thus failed to be processed.
Ordered replies collections are used e.g. by IceShrimp.NET and Sharkey,
while Mastodon uses a partially inlined plain Collection.
Not all endpoints use OpenAPISpex’ string-to-atom mapping
and they’ll end up with path params being promoted to
query params in pagination next/prev links.
Fix this by never including path params in the first place
Ecto.cast is will convert valid string keys to atoms, but can
only deal with inputs which use either string keys everywhere
or atom keys everywhere.
Since :id_type is used before the case it must be an atom,
thus it was impossible to use it with string paramteres before.
Up until now queries were always forced into descending ID order
(reverse chronological order with our ID schemes).
Now it’s possible to request the reverse by passing `oder_asc: true`.
The initial info message listing all found packs ought to be sufficient
and with many packs installed thiscan create multiple pages of log
messages on each emoji reload or server start.
Any errors or non-indexed packs are still logged to higher levels.
This requirement was originally added together with splicing the
inbox owner into the non b* addressing fields to make bcc transports
work in https://git.pleroma.social/pleroma/pleroma/-/merge_requests/390.
Later on this was relaxed to always allow deliveries devoid of any
addressing at all in f6cb963df2
and always allow deliveries from actors the owner is following in
750b369d04 to fix interop issues with
Mastodon and Honk respectively.
The justification for both the filtering and splicing comes from
one sentence in AP spec’s inbox section:
> In general, the owner of an inbox is likely
> to be able to access all of their inbox contents.
While this may provide plausible justification for splicing the owner
into cc, it is less clear how this requires or justifies the set of
filtering rules employed here.
Surveying a few other implementations no similar
filtering or splicing appears to be employed.
Furthermore, spec-compliant servers will strip bto/bcc _before_
delivery to remote servers, meaning any compliant bcc transport
out there will NOT contain any explicit addressing of the inbox owner.
Thus the addressing requirement directly opposes
the goal of the original patch.
Currently the requirement for the owner to be addressed once again
is causing interop issues. It turns out to be the root cause of
a long-standing (2+ years) bug preventing meaningful federation.
Bridgy sends e.g. Follow activities and Accepts for Follows directly
to the affected user’s personal inbox while solely addressing
the public scope in the to field. Notably follow relations never
getting established prevented the "accept if followed" allow rule
to ever come into effect.
To make matters worse non-addressed messages simply lead to a
vague "internal server error" response being sent back
which likely slowed down locating the issue.
Furthermore additional issues wrt to signatures cropped up after
the 500-response issues wa first reported, but they seem to have
already been fixed in the meantime, possibly with the signature
handling overhaul in Akkoma.
Given it repeatedly caused issues, does not appear to align with common
practice in the wider fedi ecosystem and apparently contradicts its
original intention, simply remove the requirement.
This is confirmed to fix bridgy interop.
The addressing splicing actually should also add the inbox owner to bto
or bcc instead of cc, but for now this is not changed and in practice
bto/bcc delivery appears to be basically unused anyway.
Most headers are automatically checked by the library after this
upgrade. But since digest is only required for requests with a body
and body processing is handled outside the lib atm, we need to
explicity pass the presence or absence along or not get feedback
about creating broken signatures.
This makes bugs in our signatures more apparent
allowing faster discovery and fixing
This property was introduced as a way to gauge whether and
how much enabling authfetch might break passive federation in
#312.
However, with the db field defaulting to false, there’s no distinction
between instances without valid signatures and those which just never
attempted to fetch anything from the local instance.
Furthermore, this was never exposed anywhere and required manually
checking the database or cachex state via a remote shell.
Given the above it appears this doesn't actually
provide anything useful, thus drop it.
The most common permanent receiver error arises for likes/boosts
when we don’t yet know the rlevant object and can't fetch it
due to the remote being overwhelmed or otherwise down.
Before this changes all retries were rather rapid
thus not giving the remote enough time to recover
and usually all failing. Now the remote has about 20
minutes to recover before we give up.
Transient errors from race conditions and (presumably)
weird database-cache interactions also occur regularly.
However, they resolve within the first one or two retries
and those intial retries still happen relatively quickly.
Only scrubbing "content" leads to differences between
"content" and "contentMap" eventhough the latter should
ideally match the former exactly for the primary language’s entry.
While ideally, for locally generated posts there should be no difference
between applying the scrubber or not, as it turns out automatically
generated attachment links didn't match the form expected by our default
scrubber.
Currently Akkoma never uses nor exposes the value of contentMap entries,
thus this oversight was harmless wrt to safety and at most pertubed
the language detection for our posts perfomed by remote servers.
Fixes: #928
Despite its name this property is not supposed to be a full URI,
but just a bare domain witout protocol. Furthermore, it’s supposed
to be the WebFinger domain used in userhandles and NOT the domain used
for API and ActivityPub objects (which every caller will already know
anyway).
Not following this caused issues for Pachli and Tusky.
Reported-by: nikclayton
Added in Mastodon 2.9.2 (June 2019) this is plain-text-only and supposed
to be shorter compared to the older description field.
Some clients were reported to require this field to properly function.
Reported-by: https://akkoma.dev/paulyd
The queue was not actually registered leading to
jobs getting scheduled each week but never processed
forever lingering in 'available' state.
Fixes omission in: a4f834a687
And use the project’s preferred spelling when
not referring to specific distro package.
Spelling of RedHat packages was left as is
as I can’t access RedHat’s package repository.
This allows to retain posts and boosts of remote actors with local
follows regardless of age.
With the "full" setting this can be taken further treating such
followed actors just like local users even keeping all posts they
liked or reacated to.
Pinned objects and their threads will be refetched
on user refresh which by default happens after a day
once a user is encountered again in any form including a mention.
We observed pruning pinned objects usually results in heavy load for
hours after a database prune due to a clogged up remote fetch queue as
pinned posts and their threads of many (most?) users get refetched.
Thus do not prune pinned posts by default.
Keeping closer to earlier behaviour this will still prune threads of
pinned posts regardless of --keep-threads if nothing else prevenets it.
Statmenets for keeping and breaking threads vastly differ
and the whole if block doesn't even fit on one screen.
Thus move each version out into its own function to
improve readability
Since bcfbfbcff5 (part of
<#789>), the overhead for
cleaning up orphaned attachments has been drattically reduced.
Most admins are unaware of this option even existing, but may notice an
increase in the size of the uploads directory (or S3 bucket size if used
instead) even if auto-expiring posts are used. This should hopefully
make this problem more manageable.
For those that still encounter performance issues, the setting can still
be disabled if needed.
In theory a pedantic reading of the spec indeed suggests
DMs must only be delivered to personal inboxes. However,
in practice the normative force of real-world implementations
disagrees. Mastodon, Iceshrimp.NET and GtS (the latter notably has a
config option to never use sharedInboxes) all unconditionally prefer
sharedInbox for everything without ill effect. This saves on duplicate
deliveries on the sending and processing on the receiving end.
(Typically the receiving side ends up rejecting
all but the first copy as duplicates)
Furthermore current determine_inbox logic also actually needs up
forcing personal inboxes for follower-only posts, unless they
additionally explicitly address at least one specific actor.
This is even much wasteful and directly contradicts
the explicit intent of the spec.
There’s one part where the use of sharedInbox falls apart,
namely spec-compliant bcc and bto addressing. AP spec requires
bcc/bto fields to be stripped before delivery and then implicitly
reconstructed by the receiver based on the addressed personal inbox.
In practice however, this addressing mode is almost unused. Neither of
the three implementations brought up above supports it and while *oma
does use bcc for list addressing, it does not use it in a spec-compliant
way and even copies same-host recipients into cc before delivery.
Messages with bcc addressing are handled in another function clause,
always force personal inboxes for every recipient and not affected by
this commit.
In theory it would be beneficial to use sharedInbox there too for all
but bcc recipients. But in practice list addressing has been broken for
quite some time already and is not actually exposed in any frontend,
as discussed in #812.
Therefore any changes here have virtually no effect anyway
and all code concerning it may just be outright removed.
Currently FlakeId.flake_id crashes if receiving non-UTF-8 binaries,
but we use it e.g. in the /:nick_or_id path used in akkoma-fe user
profiles.
With the upgrade such invalid binaries simply fail the id check.
Reported-in: https://meta.akkoma.dev/t/frontend-unicodeconversionerror/847
This allows discovering a page represents an ActivityPub object
and also where to find the underlying representation.
Other servers already implement this and some tools
came to rely or profit from it.
The alternate link is provided both with the "application/activity+json"
format as used by Mastodon and the standard-compliant media type.
Just like the feed provider, ActivityPub links are always enabled
unless access to local posts is restricted and not configurable.
The commit is based on earlier work by Charlotte 🦝 Deleńkec
but with fixes and some tweaks.
Co-authored-by: Charlotte 🦝 Deleńkec <lotte@chir.rs>
:discard marks jobs as "discarded", i.e. jobs which permanently failed
due to e.g. exhausting all retries or explicitly being discared due to a
fatal error.
:cancel marks jobs as "cancelled" which does not imply failure.
While neither method counts as a job "exception" in the set of
telemetries we currently export via Prometheus, the different state
is visible in the (not-exported) metadata of oban job telemetry.
We can use handlers of those events to build bespoke statistics.
Ideally we'd like to distinguish in the receiver worker between
"invalid" and "already present or delete of unknown" documents,
but this is cumbersome to get get right with a list of
free-form, human-readable descriptions oof the violated constraints.
For now, just count both as an fatal error.
# but that is cumbersome to get right with a list of string error descriptions
We now add the htmlMfm key when relevant, store this in the database, and we see it when fetching using e.g.
curl -L -H 'Accept: application/activity+json' "$ap_id"
The `@context` of the Activity Pub message now also contains `htmlMfm: https://w3id.org/fep/c16b#htmlMfm`.
When an incomming post has `htmlMfm: true`, we will not re-parse the content.
FEDERATION.md is adapted to show the `htmlMfm` term is used.
CUrrently internal actors are supposed to be identified in the database
by either a NULL nickname or a nickname prefixed by "internal.". For old
installations this is true, but only if they were created over five
years ago before 70410dfafd.
Newer installations will use "relay" as the nickname of the realy actor
causing ii to be treated as a regular user.
In particular this means all installations in the last five years never
made use of the reduced endpoint case, thus it is dropped.
Simplify this distinction by properly marking internal actors asa an
Application type in the database. This was already implemented before by
ilja in #457 but accidentally
reverted during a translation update in
eba3cce77b. This commit effectively
restores this patch together with further changes.
Also service actors unconditionally expose follow* collections atm,
eventhough the internal fetch actor doesn't actually implement them.
Since they are optional per spec and with Mastodon omitting them too
for its instance actor proving the practical viability, we should just
omit them. The relay actor however should continue to expose such
collections and they are properly implemented here.
Here too we now just use the values or their absence in the database.
We do not have any other internal.* actors besides fetch atm.
Fixes: #855
Co-authored-by: ilja space <git@ilja.space>
E.g. \*oma federates (most) follower-only posts multiple times
to each personal inbox. This commonly leads to race conditions
with jobs of several copies running at the same time and getting
past the initial "already known" check but then later all but
one will crash with an exception from the unique db index.
Since the only special thing we do with copies anyway is to discard them,
just don't create such duplicate jobs in the first place.
For the same reason and since failed jobs don't count towards
duplicates, this should have virtually no effect on federation.
Since we later only consider the Create activity for
access permission checks, but the semantically more
sensible set of fields are the object’s.
Changing the check itself to use the object may have unintended
consequences on already existing legacy posts as the old code
which processed it when it arrived may have never considered
effects on the objects addressing fields.
While the object itself has the expected adressing for an
"unlisted" post, we always use the Create activity’s
adressing fields for permission checks.
To avoid unintended effects on legacy objects
we will continue to use the activity for access perm checks,
but fix its addressing fields based on its object data.
Ref: https://git.pleroma.social/pleroma/pleroma/-/issues/3323
With the current strategy the individual
and cumulative backoff looks like this
(the + part denotes max extra random delay):
attempt backoff_single cumulative
1 16+30 16+30
2 47+60 63+90
3 243+90 ≈ 4min 321+180
4 1024+120 ≈17min 1360+300 ≈23+5min
5 3125+150 ≈20min 4500+450 ≈75+8min
6 7776+180 ≈ 2.1h 12291+630 ≈3.4h
7 16807+210 ≈ 4.6h 29113+840 ≈8h
8 32768+240 ≈ 9.1h 61896+1080 ≈17h
9 59049+270 ≈16.4h 120960+1350 ≈33h
10 100000+300 ≈27.7h 220975+1650 ≈61h
We default to 5 retries meaning the least backoff runs with attempt=4.
Therefore outgoing activiities might already be permanently dropped by a
downtime of only 23 minutes which doesn't seem too implausible to occur.
Furthermore it seems excessive to retry this quickly this often at the
beginning.
At the same time, we’d like to have at least one quick'ish retry to deal
with transient issues and maintain reasonable federation responsiveness.
If an admin wants to tolerate one -day downtime of remotes,
retries need to be almost doubled.
The new backoff strategy implemented in this commit instead
switches to an exponetial after a few initial attempts:
attempt backoff_single cumulative
1 16+30 16+30
2 143+60 159+90
3 2202+90 ≈37min 2361+180 ≈40min
4 8160+120 ≈ 2.3h 10521+300 ≈ 3h
5 77393+150 ≈21.5h 87914+450 ≈24h
Initial retries are still fast, but the same amount of retries
now allows a remote downtime of at least 40 minutes. Customising
the retry count to 5 allows for whole-day downtimes.
This was accidentally broken in c8e0f7848b
due to a one-letter mistake in the plug option name and an absence of
tests. Therefore it was once again possible to serve e.g. Javascript or
CSS payloads via uploads and emoji.
However due to other protections it was still NOT possible for anyone to
serve any payload with an ActivityPub Content-Type. With the CSP policy
hardening from previous JS payload exloits predating the Content-Type
sanitisation, there is currently no known way of abusing this weakened
Content-Type sanitisation, but should be fixed regardless.
This commit fixes the option name and adds tests to ensure
such a regression doesn't occur again in the future.
Reported-by: Lain Soykaf <lain@lain.com>
When note editing support was added, it was omitted to strip internal
fields from edited notes and their history.
This was uncovered due to Mastodon inlining the like count as a "likes"
collection conflicting with our internal "likes" list causing validation
failures. In a spot check with likes/like_count it was not possible to
inject those internal fields into the local db via Update, but this
was not extensively tested for all fields and avenues.
Similarly address normalisation did not normalise addressing in the
object history, although this was never at risk of being exploitable.
The revision history of the Pleroma MR adding edit support reveals
recusrive stripping was intentionally avoided, since it will end up
removing e.g. emoji from outgoing activities. This appears to still
be true. However, all current internal fields ("pleroma_interal"
appears to be unused) contain data already publicised otherwise anyway.
In the interest of fixing a federation bug (and at worst potential data
injection) quickly outgoing stripping is left non-recursive for now.
Of course the ultimate fix here is to not mix remote and internal data
into the same map in the first place, but unfortunately having a single
map of all truth is a core assumption of *oma's AP doc processing.
Changing this is a masive undertaking and not suitable for providing
a short-term fix.
We expect most requests to be made for the actual canonical ID,
so check this one first (starting without query headers matching the
predominant albeit spec-breaking version).
Also avoid unnecessary rerewrites of the digest header on each route
alias by just setting it once before iterating through aliases.
This matches behaviour prioir to the SigningKey migration
and the expected semantics of the http_signatures lib.
Additionally add a min interval paramter, to avoid
refetch floods on bugs causing incompatible signatures
(like e.g. currently with Bridgy)
User updates broke with the migration to separate signing keys
since user data carries signing keys but we didn't allow the
association data to be updated.
Previously there were mainly two attack vectors:
- for raw keys the owner <-> key mapping wasn't verified at all
- keys were retrieved with refetching allowed
and only the top-level ID was sanitised while
usually keys are but a subobject
This reintroduces public key checks in the user actor,
previously removed in 9728e2f8f7
but now adapted to account for the new mapping mechanism.
Notably at least two instances were not properly guarded from path
traversal attack before and are only now fixed by using SafeZip:
- frontend installation did never check for malicious paths.
But given a malicious froontend could already, e.g. steal
all user tokens even without this, in the real world
admins should only use frontends from trusted sources
and the practical implications are minimal
- the emoji pack update/upload API taking a ZIP file
did not protect against path traversal. While atm
only admins can use these emoji endpoints, emoji
packs are typically considered "harmless" and used
without prior verification from various sources.
Thus this appears more concerning.
This will replace all the slightly different safety workarounds at
different ZIP handling sites and ensure safety is actually consistently
enforced everywhere while also making code cleaner and easiert to
follow.
The first collection page is (sometimes?) inlined
which caused crashes when attempting to log the fetch failure.
But there’s no need to fetch and we can treat it like the other inline collection
And for now treat partial fetches as a success, since for all
current users partial collection data is better than no data at all.
If an error occurred while fetching a page, this previously
returned a bogus {:ok, {:error, _}} success, causing the error
to be attached to the object as an reply list subsequently
leading to the whole post getting rejected during validation.
Also the pinned collection caller did not actually handle
the preexisting error case resulting in process crashes.
Oban cataches crashes to handle job failure and retry,
thus it never bubbles up all the way and nothing is logged by default.
For better debugging, catch and log any crashes.
The object lookup is later repeated in the validator, but due to
caching shouldn't incur any noticeable performance impact.
It’s actually preferable to check here, since it avoids the otherwise
occuring user lookup and overhead from starting and aborting a
transaction
Most of them actually only accept either activities or a
non-activity object later; querying both is then a waste
of resources and may create false positives.
The only thing this does is changing the updated_at field of the user.
Afaict this came to be because prior to pins federating this was split
into two functions, one of which created a changeset, the other applying
a given changeset. When this was merged the bits were just copied into
place.
User.get_or_fetch_by_(apid|nickname) are the only external users of fetch_and_prepare_user_from_ap_id,
thus there’s no point in duplicating logging, expecially not at error level.
Currently (duplicated) _not_found errors for users make up the bulk of my logs
and are created almost every second. Deleted users are a common occurence and not
worth logging outside of debug
To facilitate this ObjectValidator.fetch_actor_and_object is adapted to
return an informative error. Otherwise we’d be unable to make an
informed decision on retrying or not later. There’s no point in
retrying to fetch MRF-blocked stuff or private posts for example.
This is the only user of fetch_actor_and_object which previously just
always preteneded to be successful. For all the activity types handled
here, we absolutely need the referenced object to be able to process it
(other than Announce whether or not processing those activity types for
unknown remote objects is desirable in the first place is up for debate)
All other users of the similar fetch_actor already properly check success.
Note, this currently lumps all reolv failure reasons together,
so even e.g. boosts of MRF rejected posts will still exhaust all
retries. The following commit improves on this.
It makes decisions based on error sources harder since all possible
nesting levels need to be checked for. As shown by the return values
handled in the receiver worker something else still nests those,
but this is a first start.
This value is currently only used by Prometheus metrics
but (after optimisng the peer query inthe preceeding commit)
the most costly part of instance stats.
This query is one of the top cost offenders during an instances
lifetime. For small instances it was shown to take up 30-50% percent of
the total database query time, while for bigger isntaces it still held
a spot in the top 3 — alost as or even more expensive overall than
timeline queries!
The good news is, there’s a cheaper way using the instance table:
no need to process each entry, no need to filter NULLs
and no need to dedupe. EXPLAIN estimates the cost of the
old query as 13272.39 and the cost of the new query as 395.74
for me; i.e. a 33-fold reduction.
Results can slightly differ. E.g. we might have an old user
predating the instance tables existence and no interaction with since
or no instance table entry due to failure to query nodeinfo.
Conversely, we might have an instance entry but all known users got
deleted since.
However, this seems unproblematic in practice
and well worth the perf improvment.
Given the previous query didn’t exclude unreachable instances
neither does the new query.
Ideally we’d like to split this up more and count most invalid documents
as an error, but silently drop e.g. Deletes for unknown objects.
However, this is hard to extract from the changeset and jobs canceled
with :discard don’t count as exceptions and I’m not aware of a idiomatic
way to cancel further retries while retaining the exception status.
Thus at least keep a log, but since superfluous "Delete"s
seem kinda frequent, don't log at error, only info level.
Retrying seems unlikely to be helpful:
- if it timed out, chances are the short delay before reattempting
won't give the remote enough time to recover from its outage and
a longer delay makes the job pointless as users likely scrolled
further already. (Likely this is already be the case after the
first 20s timeout)
- if the remote data is so borked we couldn't even parse it far enough
for an "invalid metadata" error, chances are it will remain borked
upon reattempt
There were two issues leading to needles effort:
Most importnatly, the use of AP IDs as "source_url" meant multiple
simultaneous jobs got scheduled for the same instance even with the
default unique settings.
Also jobs were scheduled uncontionally for each processed AP object
meaning we incured oberhead from managing Oban jobs even if we knew it
wasn't necessary. By comparison the single query to check if an update
is needed should be cheaper overall.
The return value is never used here; later stages which actually need it
fetch the user themselves and it doesn't matter wheter we wait for the
fech here or later (if needed at all).
Even more, this early fetch always fails if the user was already deleted
or never known to begin with, but we get something referencing it; e.g.
the very Delete action carrying out the user deletion.
This prevents processing of the Delete, but before that it will be
reattempted several times, each time attempring to fetch the
non-existing profile, wasting resources.
It was used to migrate OStatus connections to ActivityPub if possible,
but support for OStatus was long since dropped, all new actors always AP
and if anything wasn't migrated before, their instance is already marked
as unreachable anyway.
The associated logic was also buggy in several ways and deleted users
got set to ap_enabled=false also causing some issues.
This patch is a pretty direct port of the original Pleroma MR;
follow-up commits will further fix and clean up remaining issues.
Changes made (other than trivial merge conflict resolutions):
- converted CHANGELOG format
- adapted migration id for Akkoma’s timeline
- removed ap_enabled from additional tests
Ported-from: https://git.pleroma.social/pleroma/pleroma/-/merge_requests/3880
Otherwise attachments have a high chance to disappear with akkoma-fe’s
“delete & redraft” feature when cleanup is enabled in the backend. Since
we don't know whether a deletion was intended to be part of a redraft
process or even if whether the redraft was abandoned we still have to
delete attachments eventually.
A thirty minute delay should provide sufficient time for redrafting.
Fixes: #775
The cleanup attachment worker was run for every deleted post,
even if it’s a remote post whose attachments we don't even store.
This was especially bad due to attachment cleanup involving a
particularly heavy query wasting a bunch of database perf for nil.
This was uncovered by comparing statistics from
#784 and
#765 (comment)
Current logic unconditionally adds public adressing to "cc"
and follower adressing to "to" after attempting to strip it
from the other one. This creates serious problems:
First the bug prompting this investigation and fix,
unconditional addition creates duplicates when adressing
URIs already were in their intended final field; e.g.
this is prominently the case for all "unlisted" posts.
Since List.delete only removes the first occurence,
this then broke follower-adress stripping later on
making the policy ineffective.
It’s also just not safe in general wrt to non-public adressing:
e.g. pre-existing duplicates didn’t get fully stripped,
bespoke adressing modes with only one of public addressing
or follower addressing are mangled — and most importantly:
any belatedly received DM or follower-only post
also got public adressing added!
Shockingly this last point was actually asserted as "correct" in tests;
it appears to be a mistake from mindless match adjustments
while fixing crashes on nil adressing in
10c792110e.
Clean up this sloppy logic up, making sure no more duplicates are
added by us, all instances of relevant adresses are purged and only
readded when they actually existed to begin with.
Current AP spec demands anonymous objects to have an id value,
but explicitly set it to JSON null. Howeveras it turns out this is
incompatible with JSON-LD requiring `@id` to be a string and thus AP
spec is incompatible iwth the Ativity Streams spec it is based on.
This is an issue for (the few) AP implementers actually performing
JSON-LD processing, like IceShrimp.NET.
This was uncovered by IceShrimp.NET’s zotan due to our adoption of
anonymous objects for emoj in f101886709.
The issues is being discussed by W3C, and will most likely be resolved
via an errata redefining anonymous objects to completely omit the id
field just like transient objects already do. See:
https://github.com/w3c/activitypub/issues/476
Fixes: #848
Currently pruning hashtags with the prune_objects task only accounts
for whether that hashtag is associated with an object, but this may
lead to a foreign key constraint violation if that hashtag has no
objects but is followed by a local user.
This adds an additional check to see if that hashtag has any followers
before proceeding to delete it.
Instead of trying to update the version of asdf being used, just point
users to the guide on their website.
Ideally we'd do this for Elixir and Erlang as well, but new versions of
those packages may sometimes have compatibility issues with Akkoma. For
now, update those to the latest OTP and Elixir versions known to be
comaptible with Akkoma.
Turns out this is also used to set the default values in adminfe.
However, this URL may break with newer Akkoma-FE versions.
Instead, set this to blank so that it falls back to the default
NSFW cover image set at build time on Akkoma-FE.
Since we now remember the final location redirects lead to
and use it for all further checks since
3e134b07fa, these redirects
can no longer be exploited to serve counterfeit objects.
This fixes:
- display URLs from independent webapp clients
redirecting to the canonical domain
- Peertube display URLs for remote content
(acting like the above)
As hinted at in the commit message when strict checking
was added in 8684964c5d,
refetching is more robust than display URL comparison
but in exchange is harder to implement correctly.
A similar refetch approach is also employed by
e.g. Mastodon, IceShrimp and FireFish.
To make sure no checks can be bypassed by forcing
a refetch, id checking is placed at the very end.
This will fix:
- Peertube display URL arrays our transmogrifier fails to normalise
- non-canonical display URLs from alternative frontends
(theoretical; we didnt’t get any actual reports about this)
It will also be helpful in the planned key handling overhaul.
The modified user collision test was introduced in
https://git.pleroma.social/pleroma/pleroma/-/merge_requests/461
and unfortunately the issues this fixes aren’t public.
Afaict it was just meant to guard against someone serving
faked data belonging to an unrelated domain. Since we now
refetch and the id actually is mocked, lookup now succeeds
but will use the real data from the authorative server
making it unproblematic. Instead modify the fake data further
and make sure we don’t end up using the spoofed version.
- pass env vars the proper™ way
- write log to file
- drop superfluous command_background
- make settings easily overwritable via conf.d
to avoid needing to edit the service file directly
if e.g. Akkoma was installed to another location
There was one test who used MFM and now failed due to the new representation. This is now adapted so it doesn't fail any more.
There was another test failing, but I don't see how this could have been affected by the MFM changes...
But I did draw in newer dependencies, so I thought maybe a newer EARMARK dependency was now failing, and indeed.
By explicitly asking for 1.4.46 (according to mix.lock the version it was before), it now works again.
This is what was failing. It seems that earmark 1.4.47 removed everything before the comment, which it should not do.
1) test format_input/3 with markdown raw HTML (Pleroma.Web.CommonAPI.UtilsTest)
test/pleroma/web/common_api/utils_test.exs:213
Assertion with == failed
code: assert result == ~s"<a href=\"http://example.org/\">OwO</a>"
left: ""
right: "<a href=\"http://example.org/\">OwO</a>"
stacktrace:
test/pleroma/web/common_api/utils_test.exs:216: (test)
See <#381>
We can't use the HTML content as-is.
[FEP-c16b](https://codeberg.org/fediverse/fep/src/branch/main/fep/c16b/fep-c16b.md) was
written to have a "scrubber friendly" way of representing MFM functions in HTML. Here
we add support in the backend for the functions Foundkey supports. The front-ends then
needs to add support to make sure the HTML representation is turned into a correct view.
(I.e. by help of CSS and Javascript)
FEP-c16b also has a discovery mechanism to indicate to recieving servers that they can
use the `content` directly. This is not implemented in this commit
Ever since the browser frontend switcher was introduced in
de64c6c54a /akkoma counts as
an API prefix and thus gets skipped by frontend plugs
breaking the old swagger ui path of /akkoma/swagger-ui.
Do the simple thing and change the frontend path to
/pleroma/swaggerui which isn't an API path and can't collide
with frontend user paths given pleroma is areserved nickname.
Reported in
https://meta.akkoma.dev/t/view-all-endpoints/269/7https://meta.akkoma.dev/t/swagger-ui-not-loading/728
Multiple profiles can be specified as a space-separated list
and the possibility of additional profiles is explicitly brought up
in ActivityStream spec
Not _yet_ supported as of exiftool 12.87, though
at first glance it seems like standard BMP files
can't store any metadata besides colour profiles
Fixes the specific case from
AkkomaGang/akkoma-fe#396
although the frontend shouldn’t get bricked regardless.
The debug logs are very noisy and can be enabled during analysis
of a specific error believed to be SQL-related
--
Before log capturing those debug messages were still hidden,
but with log capturing they show up in the output of failed
tests unless disabled.
Cherry-picked-from: e628d00a81
Currently `mix test` prints a slew of logs in the terminal
with messages from different tests intermsparsed. Globally
enabling capture log hides log messages unless a test fails
reducing noise and making it easier to anylse the important
(from failed tests) messages.
Compiler warnings and a few messages not printed via Logger
still show up but its much more readable than before.
Ported from: 3aed111a42
We have a bunch of mysterious sporadic failures which usually disappear
when rerunning failed jobs only. Ideally we should locate and fix the
cause of those psoradic failures, but until we figure this out retrying
once makes CI status less useless.
Fragments are already always stripped anyway
so listing one specific fragment here is
unnecessary and potentially confusing.
This effectively reverts
4457928e32
but keeps the added bridgy testcase.
Usually an id should point to another AP object
and the image file isn’t an AP object. We currently
do not provide standalone AP objects for emoji and
don't keep track of remote emoji at all.
Thus just federate them as anonymous objects,
i.e. objects only existing within a parent context
and using an explicit null id.
IceShrimp.NET previously adopted anonymous objects
for remote emoji without any apparent issues. See:
333611f65e
Fixes: #694
We’ve received reports of some specific instances slowly accumulating
more and more binary data over time up to OOMs and globally setting
ERL_FULLSWEEP_AFTER=0 has proven to be an effective countermeasure.
However, this incurs increased cpu perf costs everywhere and is
thus not suitable to apply out of the box.
Apparently long-lived Phoenix websocket processes are known to
often cause exactly this by getting into a state unfavourable
for the garbage collector.
Therefore it seems likely affected instances are using timeline
streaming and do so in just the right way to trigger this. We
can tune the garbage collector just for websocket processes
and use a more lenient value of 20 to keep the added perf cost
in check.
Testing on one affected instance appears to confirm this theory
Ref.:
https://www.erlang.org/doc/man/erlang#ghlink-process_flag-2-idp226https://blog.guzman.codes/using-phoenix-channels-high-memory-usage-save-money-with-erlfullsweepafterhttps://git.pleroma.social/pleroma/pleroma/-/merge_requests/4060
Tested-by: bjo
Since those old migrations will now most likely only run during db init,
there’s not much point in running them in the background concurrently
anyway, so just drop the cncurrent setting rather than disabling
migration locks.
Currently Akkoma doesn't have any proper mitigations against BREACH,
which exploits the use of HTTP compression to exfiltrate sensitive data.
(see: #721 (comment))
To err on the side of caution, disable gzip compression for now until we
can confirm that there's some sort of mitigation in place (whether that
would be Heal-The-Breach on the Caddy side or any Akkoma-side
mitigations).
Ever since 364b6969eb
this setting wasn't used by the backend and a noop.
The stated usecase is better served by setting the base_url
to a local subdomain and using proxying in nginx/Caddy/...
Websites are increasingly getting more bloated with tricks like inlining content (e.g., CNN.com) which puts pages at or above 5MB. This value may still be too low.
Rich Media parsing was previously handled on-demand with a 2 second HTTP request timeout and retained only in Cachex. Every time a Pleroma instance is restarted it will have to request and parse the data for each status with a URL detected. When fetching a batch of statuses they were processed in parallel to attempt to keep the maximum latency at 2 seconds, but often resulted in a timeline appearing to hang during loading due to a URL that could not be successfully reached. URLs which had images links that expire (Amazon AWS) were parsed and inserted with a TTL to ensure the image link would not break.
Rich Media data is now cached in the database and fetched asynchronously. Cachex is used as a read-through cache. When the data becomes available we stream an update to the clients. If the result is returned quickly the experience is almost seamless. Activities were already processed for their Rich Media data during ingestion to warm the cache, so users should not normally encounter the asynchronous loading of the Rich Media data.
Implementation notes:
- The async worker is a Task with a globally unique process name to prevent duplicate processing of the same URL
- The Task will attempt to fetch the data 3 times with increasing sleep time between attempts
- The HTTP request obeys the default HTTP request timeout value instead of 2 seconds
- URLs that cannot be successfully parsed due to an unexpected error receives a negative cache entry for 15 minutes
- URLs that fail with an expected error will receive a negative cache with no TTL
- Activities that have no detected URLs insert a nil value in the Cachex :scrubber_cache so we do not repeat parsing the object content with Floki every time the activity is rendered
- Expiring image URLs are handled with an Oban job
- There is no automatic cleanup of the Rich Media data in the database, but it is safe to delete at any time
- The post draft/preview feature makes the URL processing synchronous so the rendered post preview will have an accurate rendering
Overall performance of timelines and creating new posts which contain URLs is greatly improved.
This lets us:
- avoid issues with broken hash indices for PostgreSQL <10
- drop runtime checks and legacy codepaths for <11 in db search
- always enable custom query plans for performance optimisation
PostgreSQL 11 is already EOL since 2023-11-09, so
in theory everyone should already have moved on to 12 anyway.
From experience, setting DB type to "Online transaction processing
system" seems to give the most optimal configuration in terms of
performance.
I also increased the recomended max connections to 25-30 as that leaves
some room for maintenance tasks to run without running out of
connections.
Finally, I removed the example configs since they're probably out of
date and I think it's better to direct people to use PGTune instead.
Logger output being visible depends on user configuration, but most of
the prints in mix tasks should always be shown. When running inside a
mix shell, it’s probably preferable to send output directly to it rather
than using raw IO.puts and we already have shell_* functions for this,
let’s use them everywhere.
Pruning can go on for a long time; give admins some insight into that
something is happening to make it less frustrating and to make it easier
which part of the process is stalled should this happen.
Again most of the changes are merely reindents;
review with whitespace changes hidden recommended.
May sometimes be helpful to get more predictable runtime
than just with an age-based limit.
The subquery for the non-keep-threads path is required
since delte_all does not directly accept limit().
Again most of the diff is just adjusting indentation, best
hide whitespace-only changes with git diff -w or similar.
This gives feedback when to stop rerunning limited batches.
Most of the diff is just adjusting indentation; best reviewed
with whitespace-only changes hidden, e.g. `git diff -w`.
This part of pruning can be very expensive and bog down the whole
instance to an unusable sate for a long time. It can thus be desireable
to split it from prune_objects and run it on its own in smaller limited batches.
If the batches are smaller enough and spaced out a bit, it may even be possible
to avoid any downtime. If not, the limit can still help to at least make the
downtime duration somewhat more predictable.
Using only the admin key works as well currently
and Akkoma needs to know the admin key to be able
to add new entries etc. However the Meilisearch
key descriptions suggest the admin key is not
supposed to be used for searches, so let’s not.
For compatibility with existings configs, search_key remains optional.
This makes show-key’s output match our documentation as of Meilisearch
1.8.0-8-g4d5971f343c00d45c11ef0cfb6f61e83a8508208. Since I’m not sure
if older versions maybe only provided description, it will fallback to
the latter if no name parameter exists.
Meilisearch is already configured to return results sorted by a
particular ranking configured in the meilisearch CLI task.
Resorting the returned top results by date partially negates this and
runs counter to what someone with tweaked settings expects.
Issue and fix identified by AdamK2003 in
#579
But instead of using a O(n^2) resorting, this commit directly
retrieves results in the correct order from the database.
Closes: #579
And while add it point to this via a top-level
FEDERATION.md document as standardised by FEP-67ff.
Also add a few missing descriptions to the config cheatsheet
and move the recently removed C2S extension into an appropiate
subsection.
Trying to display non-media as media crashed the renderer,
but when posting a status with a valid, non-media object id
the post was still created, but then crashed e.g. timeline rendering.
It also crashed C2S inbox reads, so this could not be used to leak
private posts.
Afaict this was never used, but keeping this (in theory) possible
hinders detecting which objects are actually media uploads and
which proper ActivityPub objects.
It was originally added as part of upload support itself in
02d3dc6869 without being used
and `git log -S:activity_type` and `git log -Sactivity_type:`
don't find any other commits using this.
In Mastodon media can only be used by owners and only be associated with
a single post. We currently allow media to be associated with several
posts and until now did not limit their usage in posts to media owners.
However, media update and GET lookup was already limited to owners.
(In accordance with allowing media reuse, we also still allow GET
lookups of media already used in a post unlike Mastodon)
Allowing reuse isn’t problematic per se, but allowing use by non-owners
can be problematic if media ids of private-scoped posts can be guessed
since creating a new post with this media id will reveal the uploaded
file content and alt text.
Given media ids are currently just part of a sequentieal series shared
with some other objects, guessing media ids is with some persistence
indeed feasible.
E.g. sampline some public media ids from a real-world
instance with 112 total and 61 monthly-active users:
17.465.096 at t0
17.472.673 at t1 = t0 + 4h
17.473.248 at t2 = t1 + 20min
This gives about 30 new ids per minute of which most won't be
local media but remote and local posts, poll answers etc.
Assuming the default ratelimit of 15 post actions per 10s, scraping all
media for the 4h interval takes about 84 minutes and scraping the 20min
range mere 6.3 minutes. (Until the preceding commit, post updates were
not rate limited at all, allowing even faster scraping.)
If an attacker can infer (e.g. via reply to a follower-only post not
accessbile to the attacker) some sensitive information was uploaded
during a specific time interval and has some pointers regarding the
nature of the information, identifying the specific upload out of all
scraped media for this timerange is not impossible.
Thus restrict media usage to owners.
Checking ownership just in ActivitDraft would already be sufficient,
since when a scheduled status actually gets posted it goes through
ActivityDraft again, but would erroneously return a success status
when scheduling an illegal post.
Independently discovered and fixed by mint in Pleroma
1afde067b1
In MastoAPI media descriptions are updated via the
media update API not upon post creation or post update.
This functionality was originally added about 6 years ago in
ba93396649 which was part of
https://git.pleroma.social/pleroma/pleroma/-/merge_requests/626 and
https://git.pleroma.social/pleroma/pleroma-fe/-/merge_requests/450.
They introduced image descriptions to the front- and backend,
but predate adoption of Mastodon API.
For a while adding an `descriptions` array on post creation might have
continued to work as an undocumented Pleroma extension to Masto API, but
at latest when OpenAPI specs were added for those endpoints four years
ago in 7803a85d2c, these codepaths ceased
to be used. The API specs don’t list a `descriptions` parameter and
any unknown parameters are stripped out.
The attachments_from_ids function is only called from
ScheduledActivity and ActivityDraft.create with the latter
only being called by CommonAPI.{post,update} whihc in turn
are only called from ScheduledActivity again, MastoAPI controller
and without any attachment or description parameter WelcomeMessage.
Therefore no codepath can contain a descriptions parameter.
Direct users to add in the appropriate headers and update the listening
port instead of copy/pasting a config that's already outdated and
probably would otherwise have to be synced with the main example nginx
config.
Since the configuration options on the nginx side already exist in the
sample config, there's no need to tell users to copy-paste those
settings in again.
The /var/tmp directory is not mounted as tmpfs unlike /tmp which is
mounted as such on some distros like Fedora or Arch. Since there isn't
really a benefit to having the cache on tmpfs, this change should allow
for a larger cache if needed without worrying about running out of RAM.
Applying works fine with a 20220220135625 version, but it won’t be
rolled back in the right order. Fortunately this action is idempotent
so we can just rename and reapply it with a new id.
To also not break large-scale rollbacks past 2022 for anyone
who already applied it with the old id, keep a stub migration.
This promotes and expands our existing optional migration.
Based on usage statistics from several instances, see:
#764
activities_hosts is now retained after all since it’s essential
for the "instance" query parameter of *oma’s public timeline to
reliably work in a reasonable amount of time. (Although akkoma-fe has
no support for this feature and apparently barely anyone uses it.)
activities_actor_index was already dropped before in
20221211234352_remove_unused_indices; no need to drop it again.
Birthday indices were introduced in pleroma starting with
20220116183110_add_birthday_to_users which is past the
last common migration 20210416051708.
The current 10 GiB cache size is too large to fit into tmpfs for VMs and
other machines with smaller RAM sizes. Most non-Debian distros mount
/tmp on tmpfs.
Documentation was already clear on this only stripping GPS tags.
But there are more potentially sensitive metadata tags (e.g. author
and possibly description) and the name alone suggests a broader effect.
Thus change the filter to strip all metadata except for colourspace info
and orientation (technically it strips everything and then readds
selected tags).
Explicitly stripping CommonIFD0 is needed since -all does not modify
IFD0 due to TIFF storing some actual image data there. CommonIFD0 then
strips a bunch of commonly used actual metadata tags from IFD0, to my
understanding leaving TIFF image data and custom metadata tags intact.
As of exiftool 12.57 both formats are supported, but EXIF data is
optional for JXL and if exiftool doesn’t find a preexisting metadata
chunk it will create one and treat it as a minor error resulting in
a non-zero exit code.
Setting -ignoreMinorErrors avoids failing on such uploads.
Due to JSON-LD compaction the full address of public scope
may also occur in shorter forms and the spec requires us to treat them
all equivalently. To save us the pain of repeatedly checking for all
variants internally, normalise inbound data to just one form.
See note at: https://www.w3.org/TR/activitypub/#public-addressing
This needs to happen very early, even before the other addressing fixes
else an earlier validator will reject the object. This in turn required
to move the list-tpye normalisation earlier as well, but since I was
unsure about putting empty lists into the data when no such field
existed before, I excluded this case and thus the later fixing had to be
kept as well.
Fixes: #670
Alongside moving to certbot's nginx plugin, also use conf.d instead of
recreating the sites-{available,enabled} setup that Debian/Ubuntu uses.
Furthermore, also request a certificate for the media domain at the same
time since that's now required.
The API parameter is not a timestamp but an offset.
If a sufficient amount of time passes between the tests
expires_at calculation and the internal calculation during processing
of the request the strict equality assertion fails. (Either a direct
assertion or indirect via job lookup).
To avoid this lower comparison granularity.
literally nothing uses C2S AP, and it's another route into core
systems which requires analysis and maintenance. A second API
is just extra surface for potentially bad things so let's take
it out back and obliterate it
by default just prevent job floods with a 1-seconds
uniqueness check, but override in RemoteFetcherWorker
for 5 minute uniqueness check over all states
:infinity is an option we can go for maybe at some point,
but that would prevent any refetches so maybe not idk.
We were overzealous with matching on a raw error from the object fetch that should have never been relied on like this. If we can't fetch successfully we should assume that the collection is private.
Building a more expressive and universal error struct to match on may be something to consider.
These tests relied on the removed Fetcher.fetch_object_from_id!/2 function injecting the error tuple into a log message with the exact words "Object containment failed."
We will keep this behavior by generating a similar log message, but perhaps this should do a better job of matching on the error tuple returned by Transmogrifier.handle_incoming/1
"id" is used for the canonical link to the AS2 representation of an object.
"url" is typically used for the canonical link to the HTTP representation.
It is what we use, for example, when following the "external source" link
in the frontend. However, it's not the link we include in the post contents
for quote posts.
Using URL instead means we include a more user-friendly URL for Mastodon,
and a working (in the browser) URL for Threads
previously we would uncritically take data and format it into
tags for static-fe and the like - however, instances can be
configured to disallow unauthenticated access to these resources.
this means that OG tags as a vector for information leakage.
_technically_ this should only occur if you have both
restrict_unauthenticated *AND* you run static-fe, which makes no
sense since static-fe is for unauthenticated people in particular,
but hey ho.
This brings it in line with its documentation and akkoma-fe’s
expectations. For backwards compatibility URL parameters are still
accept with lower priority. Unfortunately this means duplicating
parameters and descriptions in the API spec.
Usually Plug already pre-merges parameters from different sources into
the plain 'params' parameter which then gets forwarded by Phoenix.
However, OpenApiSpex 3.x prevents this; 4.x is set to change this
https://github.com/open-api-spex/open_api_spex/issues/334https://github.com/open-api-spex/open_api_spex/issues/92
Fixes: #691
Fixes: #722
Per the XRD specification:
> 2.4. Element <Alias>
>
> The <Alias> element contains a URI value that is an additional
> identifier for the resource described by the XRD. This value
> MUST be an absolute URI. The <Alias> element does not identify
> additional resources the XRD is describing, **but rather provides
> additional identifiers for the same resource.**
(http://docs.oasis-open.org/xri/xrd/v1.0/os/xrd-1.0-os.html#element.alias, emphasis mine)
In other words, the alias list is expected to link to things which are
not just semantically the same, but exactly the same. Old user accounts
don't do that
This change should not pose a compatibility issue: Mastodon does not
list old accounts here (See e1fcb02867/app/serializers/webfinger_serializer.rb (L12))
The use of as:alsoKnownAs is also not quite semantically right here
(see https://www.w3.org/TR/did-core/#dfn-alsoknownas, which defines
it to be used to refer to identifiers which are interchangable) but
that's what DID get for reusing a property definition that Mastodon
already squatted long before they got to it
The default refresh interval of 1 day is woefully inadequate here;
users expect to be able to add the alias to their new account and
press the move button on their old account and have it work.
This allows callers to specify a maximum age before a refetch is
triggered. We set that to 5s for the move code, as a nice compromise
between Making Things Work and ensuring that this can't be used
to hammer a remote server
This will crop them to a square matching behaviour of Husky and *key
and allowing us to never worry about consistent alignment.
Note, akkoma-fe instead displays the full image with inserted spacing.
The docker-compose.yml file is likely to be edited quite extensively by
admins when setting up an instance. This would likely cause problems
when dealing with updating Akkoma as merge conflicts would likely occur.
Docker-compose already has the ability to use override files in addition
to the main `docker-compose.yml` file. Admins can instead put any
overrides (additional volumes, container for elasticsearch, etc.) into a
file that won't be tracked by git and thus won't run into merge
conflicts in the future. In particular, the
`docker-compose.override.yml` will be checked by docker compose in
addition to the main file if it exists and override definitions from the
latter with the former.
Prior to this change, anyone, authenticated or not, could submit a search
query for an activity by URL, and cause the fetcher to go fetch it. That
shouldn't happen if `limit_to_local_content` is set to `:all` or if it's
set to `:unauthenticated` and the query came from an unauthenticated
source.
There were two warnings, these are now fixed.
I moved the fonts folder into the css folder. Antother option was to change the relative path,
but it seems that after changing it in the css file, the path got changed back when rebuilding the site.
Maybe it needs to be changed somewhere else, idk, this worked.
CREATE DATABASE was running in a transaction block with CREATE USER. This isn't allowed (any more?).
This is now two separate commands.
I also did some other touch-ups including
* making it OTP-first,
* add backup of static directory because this contains e.g. custom emoji, and
* remove the suggestion for using the setup_db.psql file. The reason is because I fear it causes more confusion than what it's worth.
* Firstly, OTP installations won't have this file because it's created in /tmp.
* Secondly, the instance has been reinstalled and thus a new setup_db.psql with different password may have been created, causing only more confusion.
During attachment upload Pleroma returns a "description" field.
* This MR allows Pleroma to read the EXIF data during upload and return the description to the FE using this field.
* If a description is already present (e.g. because a previous module added it), it will use that
* Otherwise it will read from the EXIF data. First it will check -ImageDescription, if that's empty, it will check -iptc:Caption-Abstract
* If no description is found, it will simply return nil, which is the default value
* When people set up a new instance, they will be asked if they want to read metadata and this module will be activated if so
There was an Exiftool module, which has now been renamed to Exiftool.StripLocation
2022-10-23 14:46:16 +02:00
564 changed files with 62838 additions and 28616 deletions
It is due to be removed in one of the next releases if no strong arguments for keeping it are brought up.
It is already semi-broken for large threads and conflicts with pending optimisation and cleanup work.
- support for `exclude_visibilities` in timeline and notification endpoints has been dropped
- support for list visibility / list addressing has been dropped due to lack of usage, maintenance burden and redundancy with the still supported explicit-addressing feature
- support for conversations addressing has been dropped due to lack of usage, maintenance burden and being mostly redundant with explicit addressing
- per-visibility status counters have been dropped from `/api/v1/pleroma/admin/stats`
due to unreasonably perf costs added on most database operations.
For now, the response still contains the fields, but with stubbed-out values.
### Added
- status responses include two new fields for ActivityPub cross-referencing: `akkoma.quote_apid` and `akkoma.in_reply_to_apid`
- attempting to reply to an already deleted post will return an error
(in akkoma-fe the error will be shown and your draft message retained so you can decide
for yourself whether to discard it or copy and repost as a, now intentional, new thread)
- the notification endpoint now supports the `types` parameter for filtering added in vanilla Mastodon
- the mute endpoint now supports the `duration` parameter added in vanilla Mastodon
(fixes temporary mutes created via e.g. Husky)
### Fixed
- replies and quotes to unresolvable posts now fill out IDs for replied to
status, user or quoted status with a 404-ing ID to make them recognisable as
replies/quotes instead of pretending they’re root posts
- querying a status using the ID of a non-post AP activity no longer displays
a duplicate of the post referenced by said activity with mangled author information
- fix users being able to interact (like, emoji react, ...) with posts they cannot access
- fix AP fetches of local non-Create, non-Undo activities exposing the raw, unsanitised content of the referenced object
- the above two combined allowed local users to gain access to private posts
of user they do not follow, but follow a follower of the author.
(remote users and other scenarios were to our knowledge not able to achieve this due to other restrictions)
- fix RSS and Atom feeds of hashtag timelines potentially exposing more information than Mastodon API when restricting unauthenticated API access
- fix mentioning and sending DMs to users with non-ASCII-alphanumerical usernames
- correctly hide and show inlined fallback links for quotes from Mastodon instances
- API requests with multiple unsupported parameters now will ignore all of them up to a certain limit.
If there are too many unsupported parameters this is indicated in the returned error message.
- expose generic type of attachment via Masto API if remote did not send a full MIME type but indicated a generic one
(the \*oma-specific full mime type field in the API response remains generic however, since we don't have this info)
- add back the default banner image we advertise in Masto API
- correctly redirect `/users/:nickname.rss` to the RSS instead of Atom feed
### Changed
- depreacted the `included_types` parameter in the notification endpoint; replaced by `types`
- depreacted the `expires_in` parameter in the mute endpoint; replaced by `duration`
- optimised emoji addition and removal
- emoji reloading now happens asynchronously so you won't run into timeout issues with many emoji and/or a slow disk
- upgraded all of our dependencies; this should reduce issues when running akoma with OTP28
- prefer "summary" over "name" for the attachment alt text of incoming ActivityPub documents;
this fixes alt text federation from GtS and Honk
- slightly improve index overhead for the users table
## 2025.10
### REMOVED
- Dropped `accepts_chat_messages` column from users table in database;
it has been unused for almost 3 years
- Healthcheck responses no longer contain job queue data;
it was useless anyway due to lacking any temporal information about failures
and more complete data can be obtained from Prometheus metrics.
### Added
- We mark our MFM posts as FEP-c16b compliant, and retain remote HTML representations for incoming posts marked as FEP-c16b-compliant. (Safety scrubbers are still applied)
- Prometheus stats now exposes failed ActivityPub deliveries
which failed all attempts and the failure reason
- status and user HTML pages now provide ActivityPub alternate links
- the `prune_objects` mix task no longer deletes pinned posts by default
- added `--prune-pinned` and `--keep-followed {posts,full,none}` options to the `prune_objects` mix task
- timestamps of incoming HTTP signatures are now verified.
By default up to two hour old signatures and a maximal clock skew
of 40 min for future timestamps or explicit expiry deadlines are accepted
- Added `short_description` field to `api/v1/instance` for Mastodon compatibility; the corresponding
new setting `:pleroma, :instance, :short_description` is also preferred for nodeinfo use
- Note AP objects now expose full `replies` collections and those collections can be accessed on their own;
previously only self-replies were inlined as an anonymous collection into the Note object
- Added a reference Grafana dashboard and improved documentation for Prometheus metrics
- New mix task `clean_inlined_replies` to delete some unused data from objects
- New mix task `resync_inlined_caches` to retroactively fix various issues with e.g. boosts, emoji reacts and likes
- It is now possible to allow outgoing requests to use HTTP2 via config option,
but due to bugs in the relevant backend this is not the default nor recommended.
- Prometheus metrics now expose count of scheduled and pending jobs per queue
### Fixed
- Internal actors no longer pretend to have unresolvable follow(er|ing) collections
- fixed user-level default post expiry duration overriding `expires_in` values explicitly passed during post creation
- fix crashes on non-UTF8 usernames for the API paths taking both nicknames and IDs
- fixed divergences in fields used to determine visibility;
this lead e.g. to unlisted replies from Pleroma instances being partially treated as private posts
- fixed our fetch actor advertising bogus follower and following collection ActivityPub IDs
- fix network-path references not being handled by media proxy
- federation with bridgy now works
- remote signing keys are no longer refreshed multiple times per incoming request
- fix digest emails never being sent and clogging the job queue even if not enabled
- `api/v1/instance`’s `uri` field now correctly shows the bare WebFinger domain
- fixed bug leading to `content` and the `contentMap` entry of the primary language to sometimes diverge
- reloading emoji with a broken `pack.json` file being on disk no longer crashes the whole server
- fixed blocked servers being able to access local objects when authorized fetch isn’t enabled
even when the remote server identifies itselfs
- fixed handling of inlined "featured" collections
- fixed user endpoint serving invalid ActivityPub for minimal, authfetch-fallback responses
- remote emoji reacts from IceShrimp.NET instances are now handled consistently and always merged with identical other emoji reactions
- ActivityPub requests signatures are now renewed when following redirects making sure path and host actually match the final URL
- private replies no longer increase the publicly visible reply counter
- unblock activities are no longer federated when block federation is disabled (the default)
- fix like activity database IDs rendering as misattributed posts
### Changed
- Internal and relay actors are now again represented with type "Application"
- `cleanup_attachments` is now enabled by default
- shared inboxes are now generally preferred over personal inboxes, cutting down on duplicate publishing churn
- instance actors are now really of type `Service`
- ActivityPub delivery attempts are spaced out more and increased by one
now giving up after 24h instead of ~20min by default before
- inboxes now fake a succcess reply on incoming Delete documents whose signing key is unknown but gone;
this prevents older Mastodon from repeatedly trying to deliver Deletes of actors we never knew anyway
- The config option `config :pleroma, :http, :pool_max_idle_time` was removed; it never actually
did anything and was redundant with `config :pleroma, :http, :pool_timeout` which actually works.
- repeated attempt to process incoming ActivityPub objects are spaced out more, allowing unreachable remotes
more time to come back up when e.g. processing repeats of a post not yet locally known
- `/api/v1/statuses/:id/reblog` now honours all possible visibilities except `list` and `conversation`
instead of mapping them down to a boolean private/public
- we no longer repeatedly try to deliver to explicitly deleted inboxes
- Config option `Pleroma.Web.MediaProxy.Invalidation.Http, :options` and
the `:http` subkey of `:media_proxy, :proxy_opts` now only accept
adapter-related settings inside the `:adapter` subkey, no longer on the top-level
- follow requests are now ordered reverse chronologically
## 2025.03
### Added
- Oban (worker) dashboard at `/akkoma/oban`
### Fixed
- fixed some holes in SigningKey verification potentially allowing they key-user mapping to be poisoned
- frontend ZIP files can no longer traverse to paths outside their install dir
- fixed user updates trying but failing to renew signing key information
- fixed signing key refresh on key rotation
### Changed
- Dropped obsolete `ap_enabled` indicator from user table and associated buggy logic
- The remote user count in prometheus metrics is now an estimate instead of an exact number
since the latter proved unreasonably costly to obtain for a merely nice-to-have statistic
- Various other tweaks improving stat query performance and avoiding unecessary work on received AP documents
- The HTML content for new posts (both Client-to-Server as well as Server-to-Server communication) will now use a different formatting to represent MFM. See [FEP-c16b](https://codeberg.org/fediverse/fep/src/branch/main/fep/c16b/fep-c16b.md) for more details.
- HTTP signatures now test the most likely request-target alias first cutting down on overhead
## 2025.01.01
Hotfix: Federation could break if a null value found its way into `should_federate?\1`
## 2025.01
### Added
- New config option `:instance, :cleanup_attachments_delay`
- It is now possible to display custom source URLs in akkoma-fe;
the settings are part of the frontend configuration
### Fixed
- Media proxy no longer attempts to proxy embedded images
- Fix significant uneccessary overhead of attachment cleanup;
it no longer attempts to cleanup attachments of deleted remote posts
- Fix “Delete & Redraft” often losing attachments if attachment cleanup was enabled
- ObjectAge policy no longer lets unlisted posts slip through
- ObjectAge policy no longer leaks belated DMs and follower-only posts
- the NodeINfo endpoint now uses the correct content type
### Changed
- Anonymous objects now federate completely without an id
adopting a proposed AP spec errata and restoring federation
with e.g. IceShrimp.NET and fedify-based implementations
## 3.13.3
### BREAKING
- Minimum PostgreSQL version is raised to 12
- Swagger UI moved from `/akkoma/swaggerui/` to `/pleroma/swaggerui/`
