[bug] 500 internal server error when federating Like activity from Bridgy Fed #438
Labels
No Label
approved, awaiting change
bug
configuration
documentation
duplicate
enhancement
extremely low priority
feature request
Fix it yourself
help wanted
invalid
mastodon_api
needs docs
needs tests
not a bug
planned
pleroma_api
privacy
question
static_fe
triage
wontfix
No Milestone
No project
No Assignees
4 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: AkkomaGang/akkoma#438
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Details
tested against akko.wtf, not my own instance
Version
3.5.0-12-g63f2d1cb
PostgreSQL version
No response
What were you trying to do?
Hi! I tried federating a
Like
from Bridgy Fed to this post on akko.wtf (running backend v3.5.0-12-g63f2d1cb), and it failed. Details below. Also tracking here. Not urgent, thanks in advance for looking!What did you expect to happen?
HTTP 200 or 202 on the inbox delivery request to
https://akko.wtf/users/rei/inbox
.What actually happened?
HTTP 500 error with body
{"errors":{"detail":"Internal server error"}}
.This is the same error we got from Pleroma, so it's probably activity handling code that hasn't changed since the fork. I'm guessing it choked on some part of the AS2 that's a composite object when it expects a string, maybe
actor
.Bridgy Fed log here. Full AS2 object we delivered is below.
Logs
Severity
I cannot use the software
Have you searched for this issue?
my guess would be that the usernames are failing validation since they contain restricted characters, and thus failing the http signature check
I'll have to check , but a rejection would probably be the correct behaviour
Hah, true! Afaik neither the AS2 core nor vocab specs define allowed or restricted characters for
preferredUsername
, but regardless, you're right, that Markdown value is clearly wrong. Good eyes, thanks for the catch, I'll fix and try again.Ah, my mistake, that was a copy paste artifact from the Bridgy Fed log where it got auto-linked. The actual
preferredUsername
value was justsnarfed.org
. I've updated the activity JSON in the issue here.I tried debugging, but got stuck :( Here's what I did in case someone else wants to debug further:
content-type: application/activity+json
curl -v 'https://fed.brid.gy/r/https://snarfed.org/2023-01-18_luna-nova' -H 'accept: application/activity+json' | jq .
Next thing I guess is to see if we can trigger a Like from a Birdy Fed instance and follow in Akkoma what happens with it, but at first glance, that seems more involved than simply making an account and pressing a like button (pls tell me if I'm wrong, or what would be the easiest way to try this).
[1] Click to expand
I add the following test to test/pleroma/web/activity_pub/activity_pub_controller_test.exs under `describe "/inbox"`. Note that I change the object and actor to ones who I added first in the test. The object shouldn't matter, and I assume the actor also isn't the problem since we can properly fetch that one.Note that I change the actor id, but keep it an object.
test/fixtures/fedi-birdyfed-like-activity.json has the following content I got from the OP (I also tried with the content I fetched with
curl 'https://fed.brid.gy/r/https://snarfed.org/2023-01-18_luna-nova' -H 'accept: application/activity+json'
). Note that I changed the id to use domainhttp://localhost:4001
. This is because we otherwise get a containment error.yeah I did some similar testing, to the same result
whatever is happening, it's happening during http sig verification
and bridgy is... not the easiest thing to get a test instance of
Thanks for all the sleuthing, sorry this is hard to test! I'm happy to federate another like from Bridgy Fed whenever you want.
sending the content of the
Like
with signature verification off ends up with it processing just fine, so yea it's 100% in sigsnow to try and isolate the part of bridgy that does that, this will be fun
af769de99e/common.py (L120-L151)
hm, after a rather... painful time trying to extract it, it seems to process fine :<<<
i'm going to have to debug this in prod aren't i
More examples of this 🤷🏻♀️
And, I don't know if it's the same but similar issues in Tootik.
Hi! Sorry this hasn't been easier to debug. Happy to help if I can!
New example activity below that https://idiomdrottning.org/users/Sandra/inbox (@snan which Akkoma version?) 500ed on just now. Often this is because there's a full object (which is still valid AS2/AP) in a field where the receiving server (Akkoma here) expects just an id, eg
attributedTo
below. Not sure if that's contributing here.The version I was running when these requests happened, and still am running as I am writing this, is git commit
ebfb617b26
probably better known as one commit after the v3.10.4 tag. (It was the head ofstable
when last I recompiled.)@snarfed, could it be some signature issue? That's another area where Akkoma can be pretty strict 🤷🏻♀️
Hmm! Other fediverse servers have been accepting Bridgy Fed's signatures for a long time, but sure, it's definitely possible.
BF generates HTTP Sigs based on the cavage 12 draft standard. It includes the Date, Host, Content-Type, Digest, (
SHA-256=...
), and special(request-target)
headers. Code: https://github.com/snarfed/bridgy-fed/blob/main/activitypub.py#L434-L487I didn't dive too far into your implementation, but one thing that struck me in the comment was the (request-target) header
this is actually a pseudo-header that will not make it through most http clients/reverse proxies
Oh! And I am on a rev proxy!
True!
(request-target)
isn't actually sent as an HTTP header, it's a special synthetic value that's only included in the HTTP Sig.https://datatracker.ietf.org/doc/html/draft-cavage-http-signatures-12#section-2.3
https://docs.joinmastodon.org/spec/security/#http
I started compacting
actor
,author
, andattributedTo
down to just string ids in outgoing activities, and that got Akkoma to accept aLike
! Replies are still 500ing though. Example delivered to https://akko.wtf/users/rei/inbox just now: