[bug] loss of posts from Akkoma

mirabilos commented

2024-07-23 21:53:09 +00:00

Your setup

No response

Extra details

No response

Version

3.13.2

PostgreSQL version

No response

What were you trying to do?

We have a setup with multiple communication partners who occasionally send each other direct messages:

ME = me on GtS 0.16.0rc3
SO = sender on Akkoma 3.13.2
TP = third party on Mastodon 4.2.10

What I first noticed was that occasionally, SO said she had sent me DMs (to just ME or to both ME and TP) which I haven’t received. I thought this a federation issue in GotoSocial, as it was in its early stages, and opened https://github.com/superseriousbusiness/gotosocial/issues/2496 there, but it was eventually closed because we had no way to look deeper at it.

What now follows is a summary of the last post in the issue, so feel free to skip it if you read it there:

today, SO sent a DM to ME and TP
I got it, went to reply in Semaphore and saw only @SO in the textarea and added @TP manually
TP asked what post I was referring to, as he did not receive the thread starter from SO

At this point I begun debugging. Looking into the GtS DB showed me that while there are two <a class="u-url mention" …> links in the content, the mentions column has an array with just one account ID (namely mine).

(At this point, SO (on Akkoma) tried to edit the post, adding another @TP mention, but while the textual change showed up, neither did it show up in the mentions array nor did TP become able to see it.)

I did a couple database queries more. I looked in instances which instances all used the exact same Akkoma version (6 in total), then did a query across statuses for multiple <a class="u-url mention" but lower array size of mentions, but unlisted or public posts (so I could actually look at the AP with a shell script I have that can do signed GET requests).

A bit of fiddling later (turns out Akkoma requires the actor to have an inbox and outbox, unlike Mastodon, GtS and Pleroma), and I was able to look at a public Note from Akkoma… which had:

[…]
  "tag": [
    {
      "href": "https://stereophonic.space/users/newt",
      "name": "@newt@stereophonic.space",
      "type": "Mention"
    },
    {
      "href": "https://suya.place/users/a1ba",
      "name": "@a1ba@suya.place",
      "type": "Mention"
    }
  ],
[…]

So, two mentions. However, in the GtS database, there was only one mention! (The thing I can obviously not look at is the POST to my inbox which probably populated the GtS DB.)

Unfortunately, the operator of the Akkoma instance I found the public post on is not knowledgeable enough to debug it, and I’m not sure if the admin of SO’s instance is (they run multiple instances of multiple different instance softwares, so they’re probably too busy) plus debugging something with private messages is iffy.

I’ve no idea if there’s something we can do on the GtS side to debug this. If you could figure out if there’s something on the Akkoma side, a fix for this would be very welcome.

What did you expect to happen?

Posts that mention someone trigger a notification for that someone and are visible to that someone.

What actually happened?

Posts sometimes get lost (trigger no notification (even if I have notifications for the sender in general enabled)) and sometimes are even not visible to a mentioned communication partner (private messages).

Logs

I have no way to obtain any logs on the Akkoma side of this, as these are operated by others.

Severity

I cannot use it as easily as I'd like

Have you searched for this issue?

I have double-checked and have not found this issue mentioned anywhere.

### Your setup _No response_ ### Extra details _No response_ ### Version 3.13.2 ### PostgreSQL version _No response_ ### What were you trying to do? We have a setup with multiple communication partners who occasionally send each other direct messages: - ME = me on GtS 0.16.0rc3 - SO = sender on Akkoma 3.13.2 - TP = third party on Mastodon 4.2.10 What I first noticed was that occasionally, SO said she had sent me DMs (to just ME or to both ME and TP) which I haven’t received. I thought this a federation issue in GotoSocial, as it was in its early stages, and opened <https://github.com/superseriousbusiness/gotosocial/issues/2496> there, but it was eventually closed because we had no way to look deeper at it. What now follows is a summary of the last post in the issue, so feel free to skip it if you read it there: - today, SO sent a DM to ME and TP - I got it, went to reply in Semaphore and saw only `@SO` in the textarea and added `@TP` manually - TP asked what post I was referring to, as he did not receive the thread starter from SO At this point I begun debugging. Looking into the GtS DB showed me that while there are two `<a class="u-url mention" …>` links in the content, the `mentions` column has an array with just one account ID (namely mine). (At this point, SO (on Akkoma) tried to edit the post, adding *another* `@TP` mention, but while the textual change showed up, neither did it show up in the `mentions` array nor did TP become able to see it.) I did a couple database queries more. I looked in `instances` which instances all used the exact same Akkoma version (6 in total), then did a query across `statuses` for multiple `<a class="u-url mention"` but lower array size of `mentions`, but unlisted or public posts (so I could actually look at the AP with a shell script I have that can do signed GET requests). A bit of fiddling later (turns out Akkoma requires the actor to have an `inbox` and `outbox`, unlike Mastodon, GtS and Pleroma), and I was able to look at a public Note from Akkoma… which had: ``` […] "tag": [ { "href": "https://stereophonic.space/users/newt", "name": "@newt@stereophonic.space", "type": "Mention" }, { "href": "https://suya.place/users/a1ba", "name": "@a1ba@suya.place", "type": "Mention" } ], […] ``` So, two mentions. However, in the GtS database, there was only one mention! (The thing I can obviously *not* look at is the POST to my inbox which probably populated the GtS DB.) Unfortunately, the operator of the Akkoma instance I found the public post on is not knowledgeable enough to debug it, and I’m not sure if the admin of SO’s instance is (they run multiple instances of multiple different instance softwares, so they’re probably too busy) plus debugging something with private messages is iffy. I’ve no idea if there’s something we can do on the GtS side to debug this. If you could figure out if there’s something on the Akkoma side, a fix for this would be very welcome. ### What did you expect to happen? Posts that mention someone trigger a notification for that someone and are visible to that someone. ### What actually happened? Posts sometimes get lost (trigger no notification (even if I have notifications for the sender in general enabled)) and sometimes are even not visible to a mentioned communication partner (private messages). ### Logs ```shell I have no way to obtain any logs on the Akkoma side of this, as these are operated by others. ``` ### Severity I cannot use it as easily as I'd like ### Have you searched for this issue? - [x] I have double-checked and have not found this issue mentioned anywhere.

mirabilos added the

bug

label 2024-07-23 21:53:09 +00:00

mirabilos commented

2024-07-23 21:56:07 +00:00

Author

To clarify, both Akkoma → GotoSocial and Akkoma → Mastodon loses messages and access, which is why I thought here would be the next logical good place to try and debug this.

Oneric commented

2024-07-31 20:22:01 +00:00

Member

Unfortunately, the operator of the Akkoma instance I found the public post on is not knowledgeable enough to debug it, and I’m not sure if the admin of SO’s instance is (they run multiple instances of multiple different instance softwares, so they’re probably too busy) plus debugging something with private messages is iffy.

If none of the involved Akkoma instance operators can/wants to debug, i’m afraid it will be near impossible to get any insight onto what happpens at Akkoma’s end.

Still, some notes:

So, two mentions. However, in the GtS database, there was only one mention!

If it has two mentions in the federated AP response, but GtS only parsed one, this suggests something goes wrong after GtS received the post. In theory it’s also possible the post originally had only one mention and was later edited ig, but you can test this by deleting your locally cached version and then refetching the post from its id. If it still only shows one mention, start debugging GtS’ processing

Otherwise at least for the * → Akkoma direction, direct messages may end up intentionally dropped if the intended receiver set up their account to e.g. reject DMs from non-followers. Idk if Mastodon and GtS have similar settings. Similar if any of the involved servers blocks another server or user.

> Unfortunately, the operator of the Akkoma instance I found the public post on is not knowledgeable enough to debug it, and I’m not sure if the admin of SO’s instance is (they run multiple instances of multiple different instance softwares, so they’re probably too busy) plus debugging something with private messages is iffy. If none of the involved Akkoma instance operators can/wants to debug, i’m afraid it will be near impossible to get any insight onto what happpens at Akkoma’s end. Still, some notes: > So, two mentions. However, in the GtS database, there was only one mention! If it has two mentions in the federated AP response, but GtS only parsed one, this suggests something goes wrong _after_ GtS received the post. In theory it’s also possible the post originally had only one mention and was later edited ig, but you can test this by deleting your locally cached version and then refetching the post from its id. If it still only shows one mention, start debugging GtS’ processing Otherwise at least for the `* → Akkoma` direction, direct messages may end up intentionally dropped if the intended receiver set up their account to e.g. reject DMs from non-followers. Idk if Mastodon and GtS have similar settings. Similar if any of the involved servers blocks another server or user.

BlueRaccoon commented

2025-03-05 15:43:37 +00:00

Apologies for bumping this... 8 month old thread, but I've honestly been suffering from this bug for months now on a public Akkoma node I run, and I somehow missed that this bug report was opened or I'd have thrown my hat into the ring when it was still fresh.

I'm not exactly a god when it comes to Elixir (I know just enough to be dangerous), but I'm totally down to give you whatever data I can if there's anything I can do to help triage this. Worth noting is that I don't see any bugs or errors on my end when this occurs - the only sign of trouble is that I hear a while later that the affected folks I'm replying to just never saw any notice of my response. I have no idea where in the pipeline it happens or why it happens, but if there's anything on my end I can do to resolve it, I'd like to do that.

Apologies for bumping this... 8 month old thread, but I've honestly been suffering from this bug for months now on a public Akkoma node I run, and I somehow missed that this bug report was opened or I'd have thrown my hat into the ring when it was still fresh. I'm not exactly a god when it comes to Elixir (I know just enough to be dangerous), but I'm totally down to give you whatever data I can if there's anything I can do to help triage this. Worth noting is that I don't see any bugs or errors on my end when this occurs - the only sign of trouble is that I hear a while later that the affected folks I'm replying to just never saw any notice of my response. I have no idea where in the pipeline it happens or why it happens, but if there's anything on my end I can do to resolve it, I'd like to do that.

Oneric commented

2025-03-09 15:11:46 +00:00

Member

Just to clarify again, the "mentions differ" issue from OP almost certainly isn’t an Akkoma bug but an issue on the receiver side.

For the other issue of posts not arriving, it may be on either or even both sides, which means at least one and possibly both sides need to assist in debugging. First of all you should ensure there are no server- or user-side blocks/defeds or mutes and that DMs are accepted from all authors (at least Akkoma allows each user to restrict who can send you DMs; other DMs will be dropped). Without any involved admin actually looking into it, there’s nothing we can do.

As a first check on your (Akkoma) side:

are affected instances marked as unreachable and thus excluded from federation? Connect to your db and check with SELECT id, host, unreachable_since, updated_at FROM instances WHERE host = ANY('{example.org,domain2.example}');
If thi is the case, usually it should be marked as reachable again automatically if any activity from the domain is received.
If you have setup Prometheus metrics scraping, do you see PublisherWorker errors occuring at the time of posting an affected message? *(If you have many users there might also be unrelated PublisherWorker errors though, or if you have followers on recently died instances this too might create unrelated errors until the instance is marked as unreachable, so this will need further investigation)

Should neither apply, the issue is most likely on the receiving end.

Just to clarify again, the "mentions differ" issue from OP almost certainly isn’t an Akkoma bug but an issue on the receiver side. For the other issue of posts not arriving, it may be on either or even both sides, which means at least one and possibly both sides need to assist in debugging. First of all you should ensure there are no server- or user-side blocks/defeds or mutes and that DMs are accepted from all authors *(at least Akkoma allows each user to restrict who can send you DMs; other DMs will be dropped)*. Without any involved admin actually looking into it, there’s nothing we can do. As a first check on your (Akkoma) side: - are affected instances marked as unreachable and thus excluded from federation? Connect to your db and check with `SELECT id, host, unreachable_since, updated_at FROM instances WHERE host = ANY('{example.org,domain2.example}');` If thi is the case, usually it should be marked as reachable again automatically if any activity from the domain is received. - If you have setup Prometheus metrics scraping, do you see PublisherWorker errors occuring at the time of posting an affected message? *(If you have many users there might also be unrelated PublisherWorker errors though, or if you have followers on recently died instances this too might create unrelated errors until the instance is marked as unreachable, so this will need further investigation) Should neither apply, the issue is most likely on the receiving end.

BlueRaccoon commented

2025-03-09 17:20:27 +00:00

Checking the instances marked as unreachable, none of them are ones I actively know this issue is occurring with.

As of this moment, I don't have Prometheus metrics scraping set up yet. I'll look into getting that set up.

Checking the instances marked as unreachable, none of them are ones I actively know this issue is occurring with. As of this moment, I don't have Prometheus metrics scraping set up yet. I'll look into getting that set up.

Oneric commented

2025-03-10 15:06:38 +00:00

Member

On the latest release you might also be able to use the Oban dashboard (got o /akkoma/oban while logged in as an admin account). However, the dashboard might not keep data around for as long as victoriametrics/prometheus do, so you might need to be quick. On the upside it revelas more information about the failed jobs.
If you spot a failed publisher job in the Oban dashboard, please also check the job details to confirm it’s related to your DM and why it failed.

On the latest release you might also be able to use the Oban dashboard *(got o `/akkoma/oban` while logged in as an admin account)*. However, the dashboard might not keep data around for as long as victoriametrics/prometheus do, so you might need to be quick. On the upside it revelas more information about the failed jobs. If you spot a failed publisher job in the Oban dashboard, please also check the job details to confirm it’s related to your DM and why it failed.

BlueRaccoon commented

2025-03-10 17:32:02 +00:00

I attempted sending a message to someone I know has been experiencing the related issue, and I'm not seeing any errors pop up in the dashboard. I tried drilling down to see if I could find more details about what succeeded towards federating to their node (maybe if some value was off?), but the events disappear from that dashboard after only 90 seconds, so I wasn't able to find the specific task where it federated to their instance in time. (All the more reason for me to set up more long-term monitoring...)

Oneric commented

2025-03-10 18:47:29 +00:00

Member

you may be able to extend the time it remains with:

config :pleroma, Oban,
  # default set of plugins but with custom pruner max_age
  # (default is 60s making it hard to check errors after seeing them in dashboard etc, but for busy instances keeping it around longer may create too much bloat)
  plugins: [
    {Oban.Plugins.Pruner, max_age: 600},
    {Oban.Plugins.Reindexer, schedule: "@weekly"}
  ]

you may be able to extend the time it remains with: ```elixir config :pleroma, Oban, # default set of plugins but with custom pruner max_age # (default is 60s making it hard to check errors after seeing them in dashboard etc, but for busy instances keeping it around longer may create too much bloat) plugins: [ {Oban.Plugins.Pruner, max_age: 600}, {Oban.Plugins.Reindexer, schedule: "@weekly"} ] ```

mirabilos commented

2025-03-13 16:51:02 +00:00

Author

happened today again: SO sent a DM to TP and me, putting the @ for TP first because apparently TP isn’t getting the DM (so it’s not a GtS problem), and I don’t get the post

"2.7.2 (compatible; Akkoma 3.15.1)"

happened today again: SO sent a DM to TP and me, putting the @ for TP first because apparently TP isn’t getting the DM (so it’s not a GtS problem), and I don’t get the post "2.7.2 (compatible; Akkoma 3.15.1)"

Oneric commented

2025-03-13 18:35:50 +00:00

Member

well, what’s in your logs? did you receive the document into your personal inbox, but failed processing, did you never get a inbox POST at all, something else?
As I wrote before just "doesn't work" is in no way actionable and involved admins actually need to debug this to provide specific info.

well, what’s in your logs? did you receive the document into your personal inbox, but failed processing, did you never get a inbox POST at all, something else? As I wrote before just "doesn't work" is in no way actionable and involved admins actually need to debug this to provide specific info.

mirabilos commented

2025-03-13 22:21:43 +00:00

Author

I was hoping to provide a data point so that an Akkoma developer can possibly reproduce it, as I found the “second mention in a DM doesn’t get the post” quite systematic.

There is no admin involved on the Akkoma side, and the DMs in question have a level of privacy in which we’d rather keep it that way.

Looking at my (GtS side) logs, even the Apache httpd in front did not get any connection from the Akkoma instance in the timeframe in question.

SO also just told me something which I interpret as “TP replied to the post but TP’s client’s automatic mentioning on replies also replied to just SO, not me”, so the guess is that Akkoma somehow doesn’t format multiple mentions right in messages it sends out, in a way affecting both GtS and Mastdarm instances.

I *was* hoping to provide a data point so that an Akkoma developer can possibly reproduce it, as I found the “second mention in a DM doesn’t get the post” quite systematic. There is no admin involved on the Akkoma side, and the DMs in question have a level of privacy in which we’d rather keep it that way. Looking at my (GtS side) logs, even the Apache httpd in front did not get *any* connection from the Akkoma instance in the timeframe in question. SO also just told me something which I interpret as “TP replied to the post but TP’s client’s automatic mentioning on replies also replied to just SO, not me”, so the guess is that Akkoma somehow doesn’t format multiple mentions right in messages it sends out, in a way affecting both GtS and Mastdarm instances.

Oneric commented

2025-03-14 00:29:07 +00:00

Member

Looking at my (GtS side) logs, even the Apache httpd in front did not get any connection from the Akkoma instance in the timeframe in question.

Ok, this bit is helpful and suggests either your account or your instance domain is (partially) blocked either by the post author or the author’s instance admin OR your instance is considered "unreachable" by the sending instance (keep in mind not all blocks are published).
Without actually involving the instance admin though, there’s no way to say anything further or if really neither of the above applies detect what else might go wrong. Thus if you don't want to do this, there’s nothing to do here and you’ll just have to hope whatever BlueRacoon experiences and actually debugs will end up helping you too.

> Looking at my (GtS side) logs, even the Apache httpd in front did not get any connection from the Akkoma instance in the timeframe in question. Ok, this bit is helpful and suggests either your account or your instance domain is (partially) blocked either by the post author or the author’s instance admin OR your instance is considered "unreachable" by the sending instance *(keep in mind not all blocks are published)*. Without actually involving the instance admin though, there’s no way to say anything further or if really neither of the above applies detect what else might go wrong. Thus if you don't want to do this, there’s nothing to do here and you’ll just have to hope whatever BlueRacoon experiences and actually debugs will end up helping you too.

mirabilos commented

2025-03-14 18:24:31 +00:00

Author

Ok, this bit is helpful and suggests either your account or your instance domain is (partially) blocked either by the post author or the author’s instance admin OR your instance is considered "unreachable" by the sending instance (keep in mind not all blocks are published).

Pretty implausible, we communicate over this all the time.

I just asked SO to send a DM to just me, and it arrived, both in the timeline and in notifications.

Incidentally, SO then sent a test DM to TP and me in that order again, and it did arrive this time.

It also worked when adding a picture, as in the lost post.

> Ok, this bit is helpful and suggests either your account or your instance domain is (partially) blocked either by the post author or the author’s instance admin OR your instance is considered "unreachable" by the sending instance *(keep in mind not all blocks are published).* Pretty implausible, we communicate over this all the time. I just asked SO to send a DM to just me, and it arrived, both in the timeline and in notifications. Incidentally, SO then sent a test DM to TP and me in that order again, and it did arrive this time. It also worked when adding a picture, as in the lost post.

mirabilos commented

2025-03-14 18:47:49 +00:00

Author

I have another extra debugging thing, in case it helps.

SO navigated to the post in question (which did not federate to me) in the stock webfrontend with F12→Network open and sent me the HAR, and I extracted the JSON object in question and (hopefully) anonymised it sufficiently to post but still show all included detail (I mostly just inserted lots of Xes and 999s instead of IDs and used {so,tp,my}_name@{so,tp,my}-domain.example for handles via search/replace).

{
  "muted": false,
  "tags": [],
  "edited_at": null,
  "id": "As170yN4eVuhoXXxxX",
  "content": "<span class=\"h-card\"><a class=\"u-url mention\" data-user=\"AXHyaHGsPS5pXXXxXX\" href=\"https://tp-domain.example/@tp_name\" rel=\"ugc\">@<span>tp_name</span></a></span> <span class=\"h-card\"><a class=\"u-url mention\" data-user=\"ASDtwUbfjRJbxXXxXX\" href=\"https://my-domain.example/@my_name\" rel=\"ugc\">@<span>my_name</span></a></span>",
  "account": {
    "id": "AXGZYcA1538hxXXXxX",
    "header": "https://so-domain.example/images/banner.png",
    "fields": [],
    "source": {
      "sensitive": false,
      "fields": [],
      "pleroma": {
        "show_role": true,
        "actor_type": "Person",
        "no_rich_text": false,
        "discoverable": true
      },
      "note": "XXXXXXXXXXXX",
      "privacy": "public"
    },
    "pleroma": {
      "email": "XXXXXXXXXXX@XXXXXXXXXXXX",
      "background_image": null,
      "skip_thread_containment": false,
      "ap_id": "https://so-domain.example/users/so_name",
      "is_admin": false,
      "is_moderator": false,
      "hide_follows_count": false,
      "hide_followers_count": false,
      "hide_follows": false,
      "hide_followers": false,
      "tags": [],
      "is_confirmed": true,
      "is_suggested": false,
      "hide_favorites": true,
      "allow_following_move": true,
      "favicon": "https://so-domain.example/favicon.png",
      "also_known_as": [],
      "relationship": {},
      "notification_settings": {
        "block_from_strangers": false,
        "hide_notification_contents": false
      },
      "unread_notifications_count": 999,
      "unread_conversation_count": 999
    },
    "url": "https://so-domain.example/users/so_name",
    "username": "so_name",
    "following_count": 999,
    "created_at": "1970-01-01T00:00:00.000Z",
    "emojis": [],
    "akkoma": {
      "instance": {
        "name": "so-domain.example",
        "favicon": "https://so-domain.example/favicon.png",
        "nodeinfo": {
          "version": "2.1",
          "protocols": [
            "activitypub"
          ],
          "metadata": {
            "private": false,
            "features": [
              "pleroma_api",
              "akkoma_api",
              "mastodon_api",
              "mastodon_api_streaming",
              "polls",
              "v2_suggestions",
              "pleroma_explicit_addressing",
              "shareable_emoji_packs",
              "multifetch",
              "pleroma:api/v1/notifications:include_types_filter",
              "quote_posting",
              "editing",
              "relay",
              "safe_dm_mentions",
              "pleroma_emoji_reactions",
              "exposable_reactions",
              "profile_directory",
              "custom_emoji_reactions",
              "pleroma:get:main/ostatus"
            ],
            "suggestions": {
              "enabled": false
            },
            "federation": {
              "enabled": true,
              "mrf_simple": {
                "accept": [],
                "reject": [
                  "technodon.org",
                  "elonsucks.org"
                ],
                "media_removal": [],
                "media_nsfw": [],
                "federated_timeline_removal": [],
                "report_removal": [],
                "followers_only": [],
                "avatar_removal": [],
                "banner_removal": [],
                "background_removal": [],
                "reject_deletes": []
              },
              "mrf_hashtag": {
                "sensitive": [
                  "nsfw"
                ],
                "reject": [],
                "federated_timeline_removal": []
              },
              "mrf_object_age": {
                "threshold": 604800,
                "actions": [
                  "delist",
                  "strip_followers"
                ]
              },
              "quarantined_instances": [],
              "quarantined_instances_info": {
                "quarantined_instances": {}
              },
              "exclusions": false,
              "mrf_policies": [
                "ObjectAgePolicy",
                "SimplePolicy",
                "HashtagPolicy",
                "InlineQuotePolicy",
                "NormalizeMarkup",
                "DirectMessageDisabledPolicy"
              ],
              "mrf_simple_info": {}
            },
            "accountActivationRequired": true,
            "federatedTimelineAvailable": true,
            "fieldsLimits": {
              "maxFields": 10,
              "maxRemoteFields": 20,
              "nameLength": 512,
              "valueLength": 2048
            },
            "invitesEnabled": false,
            "localBubbleInstances": [],
            "mailerEnabled": true,
            "nodeDescription": "XXXXXXXXXXXXXXXXXXX",
            "nodeName": "so-domain.example",
            "pollLimits": {
              "max_options": 20,
              "max_option_chars": 200,
              "min_expiration": 0,
              "max_expiration": 31536000
            },
            "postFormats": [
              "text/plain",
              "text/html",
              "text/markdown",
              "text/bbcode",
              "text/x.misskeymarkdown"
            ],
            "privilegedStaff": false,
            "publicTimelineVisibility": {
              "local": true,
              "federated": true,
              "bubble": false
            },
            "restrictedNicknames": [
              ".well-known",
              "~",
              "about",
              "activities",
              "api",
              "auth",
              "check_password",
              "dev",
              "friend-requests",
              "inbox",
              "internal",
              "main",
              "media",
              "nodeinfo",
              "notice",
              "oauth",
              "objects",
              "ostatus_subscribe",
              "pleroma",
              "proxy",
              "push",
              "registration",
              "relay",
              "settings",
              "status",
              "tag",
              "user-search",
              "user_exists",
              "users",
              "web",
              "verify_credentials",
              "update_credentials",
              "relationships",
              "search",
              "confirmation_resend",
              "mfa"
            ],
            "skipThreadContainment": true,
            "staffAccounts": [
              "https://so-domain.example/users/xxx"
            ],
            "uploadLimits": {
              "background": 4000000,
              "avatar": 2000000,
              "general": 30000000,
              "banner": 4000000
            }
          },
          "usage": {
            "users": {
              "total": 999,
              "activeHalfyear": 99,
              "activeMonth": 99
            },
            "localPosts": 999
          },
          "services": {
            "inbound": [],
            "outbound": []
          },
          "software": {
            "name": "akkoma",
            "version": "3.15.1",
            "repository": "https://akkoma.dev/AkkomaGang/akkoma"
          },
          "openRegistrations": true
        }
      },
      "status_ttl_days": null,
      "permit_followback": false
    },
    "last_status_at": "2025-03-14",
    "note": "XXXXXXXXXXXXXXXXXX",
    "acct": "so_name",
    "avatar": "https://media.so-domain.example/so-domain-example/xxx.jpg",
    "avatar_static": "https://media.so-domain.example/so-domain-example/xxx.jpg",
    "bot": false,
    "display_name": "so_name",
    "followers_count": 999,
    "fqn": "so_name@so-domain.example",
    "header_static": "https://so-domain.example/images/banner.png",
    "locked": false,
    "statuses_count": 999,
    "accepts_direct_messages_from": "everybody",
    "follow_requests_count": 0
  },
  "pleroma": {
    "local": true,
    "context": "https://so-domain.example/contexts/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "content": {
      "text/plain": "@tp_name @my_name"
    },
    "spoiler_text": {
      "text/plain": ""
    },
    "pinned_at": null,
    "direct_conversation_id": 999,
    "conversation_id": 9999999999,
    "emoji_reactions": [],
    "expires_at": null,
    "in_reply_to_account_acct": null,
    "parent_visible": false,
    "thread_muted": false
  },
  "bookmarked": false,
  "application": null,
  "in_reply_to_id": null,
  "sensitive": false,
  "quote_id": null,
  "reblogs_count": 0,
  "url": "https://so-domain.example/notice/As170yN4eVuhoXXxxX",
  "poll": null,
  "favourites_count": 0,
  "card": null,
  "reblog": null,
  "favourited": false,
  "reblogged": false,
  "media_attachments": [
    {
      "id": "99999999",
      "type": "image",
      "description": "",
      "pleroma": {
        "mime_type": "image/jpeg"
      },
      "url": "https://media.so-domain.example/so-domain-example/xxx.jpeg",
      "preview_url": "https://media.so-domain.example/so-domain-example/xxx.jpeg",
      "blurhash": null,
      "remote_url": "https://media.so-domain.example/so-domain-example/xxx.jpeg",
      "text_url": "https://media.so-domain.example/so-domain-example/xxx.jpeg"
    }
  ],
  "akkoma": {
    "source": {
      "content": "@tp_name@tp-domain.example @my_name@my-domain.example",
      "mediaType": "text/plain"
    }
  },
  "uri": "https://so-domain.example/objects/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "emoji_reactions": [],
  "mentions": [
    {
      "id": "AXHyaHGsPS5pXXXxXX",
      "url": "https://tp-domain.example/@tp_name",
      "username": "tp_name",
      "acct": "tp_name@tp-domain.example"
    }
  ],
  "created_at": "2025-03-13T16:09:33.000Z",
  "language": "en",
  "visibility": "direct",
  "replies_count": 3,
  "text": null,
  "quote": null,
  "emojis": [],
  "spoiler_text": "",
  "pinned": false,
  "in_reply_to_account_id": null
}

I have another extra debugging thing, in case it helps. SO navigated to the post in question (which did not federate to me) in the stock webfrontend with F12→Network open and sent me the HAR, and I extracted the JSON object in question and (hopefully) anonymised it sufficiently to post but still show all included detail (I mostly just inserted lots of Xes and 999s instead of IDs and used `{so,tp,my}_name@{so,tp,my}-domain.example` for handles via search/replace). ``` { "muted": false, "tags": [], "edited_at": null, "id": "As170yN4eVuhoXXxxX", "content": "<a class=\"u-url mention\" data-user=\"AXHyaHGsPS5pXXXxXX\" href=\"https://tp-domain.example/@tp_name\" rel=\"ugc\">@tp_name</a> <a class=\"u-url mention\" data-user=\"ASDtwUbfjRJbxXXxXX\" href=\"https://my-domain.example/@my_name\" rel=\"ugc\">@my_name</a>", "account": { "id": "AXGZYcA1538hxXXXxX", "header": "https://so-domain.example/images/banner.png", "fields": [], "source": { "sensitive": false, "fields": [], "pleroma": { "show_role": true, "actor_type": "Person", "no_rich_text": false, "discoverable": true }, "note": "XXXXXXXXXXXX", "privacy": "public" }, "pleroma": { "email": "XXXXXXXXXXX@XXXXXXXXXXXX", "background_image": null, "skip_thread_containment": false, "ap_id": "https://so-domain.example/users/so_name", "is_admin": false, "is_moderator": false, "hide_follows_count": false, "hide_followers_count": false, "hide_follows": false, "hide_followers": false, "tags": [], "is_confirmed": true, "is_suggested": false, "hide_favorites": true, "allow_following_move": true, "favicon": "https://so-domain.example/favicon.png", "also_known_as": [], "relationship": {}, "notification_settings": { "block_from_strangers": false, "hide_notification_contents": false }, "unread_notifications_count": 999, "unread_conversation_count": 999 }, "url": "https://so-domain.example/users/so_name", "username": "so_name", "following_count": 999, "created_at": "1970-01-01T00:00:00.000Z", "emojis": [], "akkoma": { "instance": { "name": "so-domain.example", "favicon": "https://so-domain.example/favicon.png", "nodeinfo": { "version": "2.1", "protocols": [ "activitypub" ], "metadata": { "private": false, "features": [ "pleroma_api", "akkoma_api", "mastodon_api", "mastodon_api_streaming", "polls", "v2_suggestions", "pleroma_explicit_addressing", "shareable_emoji_packs", "multifetch", "pleroma:api/v1/notifications:include_types_filter", "quote_posting", "editing", "relay", "safe_dm_mentions", "pleroma_emoji_reactions", "exposable_reactions", "profile_directory", "custom_emoji_reactions", "pleroma:get:main/ostatus" ], "suggestions": { "enabled": false }, "federation": { "enabled": true, "mrf_simple": { "accept": [], "reject": [ "technodon.org", "elonsucks.org" ], "media_removal": [], "media_nsfw": [], "federated_timeline_removal": [], "report_removal": [], "followers_only": [], "avatar_removal": [], "banner_removal": [], "background_removal": [], "reject_deletes": [] }, "mrf_hashtag": { "sensitive": [ "nsfw" ], "reject": [], "federated_timeline_removal": [] }, "mrf_object_age": { "threshold": 604800, "actions": [ "delist", "strip_followers" ] }, "quarantined_instances": [], "quarantined_instances_info": { "quarantined_instances": {} }, "exclusions": false, "mrf_policies": [ "ObjectAgePolicy", "SimplePolicy", "HashtagPolicy", "InlineQuotePolicy", "NormalizeMarkup", "DirectMessageDisabledPolicy" ], "mrf_simple_info": {} }, "accountActivationRequired": true, "federatedTimelineAvailable": true, "fieldsLimits": { "maxFields": 10, "maxRemoteFields": 20, "nameLength": 512, "valueLength": 2048 }, "invitesEnabled": false, "localBubbleInstances": [], "mailerEnabled": true, "nodeDescription": "XXXXXXXXXXXXXXXXXXX", "nodeName": "so-domain.example", "pollLimits": { "max_options": 20, "max_option_chars": 200, "min_expiration": 0, "max_expiration": 31536000 }, "postFormats": [ "text/plain", "text/html", "text/markdown", "text/bbcode", "text/x.misskeymarkdown" ], "privilegedStaff": false, "publicTimelineVisibility": { "local": true, "federated": true, "bubble": false }, "restrictedNicknames": [ ".well-known", "~", "about", "activities", "api", "auth", "check_password", "dev", "friend-requests", "inbox", "internal", "main", "media", "nodeinfo", "notice", "oauth", "objects", "ostatus_subscribe", "pleroma", "proxy", "push", "registration", "relay", "settings", "status", "tag", "user-search", "user_exists", "users", "web", "verify_credentials", "update_credentials", "relationships", "search", "confirmation_resend", "mfa" ], "skipThreadContainment": true, "staffAccounts": [ "https://so-domain.example/users/xxx" ], "uploadLimits": { "background": 4000000, "avatar": 2000000, "general": 30000000, "banner": 4000000 } }, "usage": { "users": { "total": 999, "activeHalfyear": 99, "activeMonth": 99 }, "localPosts": 999 }, "services": { "inbound": [], "outbound": [] }, "software": { "name": "akkoma", "version": "3.15.1", "repository": "https://akkoma.dev/AkkomaGang/akkoma" }, "openRegistrations": true } }, "status_ttl_days": null, "permit_followback": false }, "last_status_at": "2025-03-14", "note": "XXXXXXXXXXXXXXXXXX", "acct": "so_name", "avatar": "https://media.so-domain.example/so-domain-example/xxx.jpg", "avatar_static": "https://media.so-domain.example/so-domain-example/xxx.jpg", "bot": false, "display_name": "so_name", "followers_count": 999, "fqn": "so_name@so-domain.example", "header_static": "https://so-domain.example/images/banner.png", "locked": false, "statuses_count": 999, "accepts_direct_messages_from": "everybody", "follow_requests_count": 0 }, "pleroma": { "local": true, "context": "https://so-domain.example/contexts/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "content": { "text/plain": "@tp_name @my_name" }, "spoiler_text": { "text/plain": "" }, "pinned_at": null, "direct_conversation_id": 999, "conversation_id": 9999999999, "emoji_reactions": [], "expires_at": null, "in_reply_to_account_acct": null, "parent_visible": false, "thread_muted": false }, "bookmarked": false, "application": null, "in_reply_to_id": null, "sensitive": false, "quote_id": null, "reblogs_count": 0, "url": "https://so-domain.example/notice/As170yN4eVuhoXXxxX", "poll": null, "favourites_count": 0, "card": null, "reblog": null, "favourited": false, "reblogged": false, "media_attachments": [ { "id": "99999999", "type": "image", "description": "", "pleroma": { "mime_type": "image/jpeg" }, "url": "https://media.so-domain.example/so-domain-example/xxx.jpeg", "preview_url": "https://media.so-domain.example/so-domain-example/xxx.jpeg", "blurhash": null, "remote_url": "https://media.so-domain.example/so-domain-example/xxx.jpeg", "text_url": "https://media.so-domain.example/so-domain-example/xxx.jpeg" } ], "akkoma": { "source": { "content": "@tp_name@tp-domain.example @my_name@my-domain.example", "mediaType": "text/plain" } }, "uri": "https://so-domain.example/objects/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "emoji_reactions": [], "mentions": [ { "id": "AXHyaHGsPS5pXXXxXX", "url": "https://tp-domain.example/@tp_name", "username": "tp_name", "acct": "tp_name@tp-domain.example" } ], "created_at": "2025-03-13T16:09:33.000Z", "language": "en", "visibility": "direct", "replies_count": 3, "text": null, "quote": null, "emojis": [], "spoiler_text": "", "pinned": false, "in_reply_to_account_id": null } ```

Rows
Columns

[bug] loss of posts from Akkoma #822

Your setup

Extra details

Version

PostgreSQL version

What were you trying to do?

What did you expect to happen?

What actually happened?

Logs

Severity

Have you searched for this issue?