[bug] Spamming Fetch Requests #831
Labels
No labels
approved, awaiting change
broken setup
bug
cannot reproduce
configuration
documentation
duplicate
enhancement
extremely low priority
feature request
Fix it yourself
help wanted
invalid
mastodon_api
needs change/feedback
needs docs
needs tests
not a bug
not our bug
planned
pleroma_api
privacy
question
static_fe
triage
wontfix
No milestone
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
AkkomaGang/akkoma#831
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Your setup
Docker
Extra details
docker-compose
Version
3.13.2
PostgreSQL version
No response
What were you trying to do?
Just using akkoma. Nothing fancy.
I will mark it as I cannot use the software, because of the high request rate, might get me banned over all the network.
What did you expect to happen?
The server should not make so many requests to another.
Lower CPU usage
What actually happened?
My instance got blocked, according to the admin of the mastodon instance that blocked us:
So I went to check the logs, hence I recently restarted the container I didn't have logs of his instance bu I have some spamming requests, more than one per second, to the same instance.
Log: https://bin.disroot.org/?8f5065007aaec160#3nJr5kXdSc3pgaxNxHGkJgxGJC7C8ZBEru1gGXBj42tL
Video: https://i.imgur.com/mS1CYtR.mp4
So I wonder if that's what made I get banned, might happen to someone else.
Also some other errors after I temporally blocked the instance that I was spamming in the log above:
https://bin.disroot.org/?6fd0b7d6a3370c54#DidqCB6fY6e7vCJHeSDyP9k7Qsxw1LMeUubMcd3u3BnB
The errors are still spammy, but not so often, at least the log throws one per second.
Also, the CPU usage gets pretty high: https://i.imgur.com/tqH1QUl.png
I am running akkoma on an arm64 3GHz 4 CPUs. Should not have such a high load.
Logs
Severity
I cannot use the software
Have you searched for this issue?
For good measure I restarted the container and got all the starting log as well:
I think I saw this recently too. We (Bridgy Fed) shipped pinned post support via the
featuredcollection a couple days ago, and surprisingly our inbound HTTP request volume from Pleroma and Akkoma instances skyrocketed over three orders of magnitude (!), from <1qps total to 4-6kqps (!!!).When I rolled back our change and stopped advertising and serving the
featuredcollection, the request flood immediately stopped.The repeated
Could not decode featured collection...lines in the logs above make me suspect that this is the same problem.Here's what our
featuredcollection looked like, both inside actors and when served separately:cannot replicate, local instance says :unsupported and exits, which seems sane
@floatingghost barring reproducing/fixing the request flood, do you see what in the shape of the
featuredcollection above might be making Akkoma unhappy? I'm happy to change it to be more Akkoma/Pleroma-compatible if that prevents this.Here are the top servers and versions, based on incoming
User-Agents, over the ~5m period 2025-05-19 13:52:49-13:58:21 UTC. Looks like Akkoma was all pre 3.11. Happy to close if this is a known bug that was fixed since 3.10!Part of Pleroma's fix for this was evidently to relax their checks on
alsoKnownAs. Maybe more too, I don't fully know. Details in https://git.pleroma.social/pleroma/pleroma/-/issues/3379#note_112979 and https://git.pleroma.social/pleroma/pleroma/-/merge_requests/4367Sorry it took a while (though unfortunately it might take a while longer for a fix to land unless someone else picks this up)
While we only accept AP-like IDs here, this doesn’t cause any rejections, the entries are just dropped (otherwise this would have prevent any meaningful federation with bridgy at all), so this doesn’t have naything to do with fetch spams.
As you observed, only older Akkoma instances showed a significant increase is requests; this is likely because we improved failure handling and properly respect rate-limit responses which Pleroma presumably never added. However, this doesn’t mean there’s no issue, just that the symptoms aren’t as ridiculous.
The issue here is that the user data processing isn’t using our generic collection handling (since it was never migrated after the former was created) and is hardcoded to assume a simple ID here, not an inlined collection. The way this assumption was hardcoded ends up breaking the entire user data processing.
The proper fix is to migrate all remaining users of the minimal collection handling inside the ActivityPub module to our generic Collection fetcher.
@Oneric thanks for the explanation! Makes sense, I appreciate it.
@snarfed
developshould now be able to handle the inlined featured collectionI did not yet clean up all occurrences of old inline collection logic though; that’s tracked in #950
As for the still-available logs from OP (everything on the external paste service disappeared; please always copy logs into the issue itself using
<details></details>for long text):The first few errors are about too many profile fields, which was fixed a while ago. The
featuredrelated logs are few, about different users and show the collection resolved as a404a410, which already previously should’ve been handled well.Thanks for your work here @Oneric! Much appreciated.