[bug] Lemmy and kbin federation does not work

Clovis commented

2023-06-18 16:09:01 +00:00

Contributor

Your setup

OTP

Extra details

No response

Version

v3.0.9

PostgreSQL version

12

What were you trying to do?

I was trying to follow some communities and users on https://kbin.sh (https://kbin.sh/u/clovis, https://kbin.sh/m/oblivion) and lemmy.world (https://lemmy.world/c/maliciouscompliance)

What did you expect to happen?

I expected the accounts and communities to be followed and appear in my timelines

What actually happened?

The follow request were stuck waiting, no activities appeared on my timeline.

I made two experimental fixes on my fork and I am now able to follow Lemmy communities

382a3e3a16
(See followers and following specs: https://www.w3.org/TR/activitypub/#followers-property)

629dfd23bc

I'm not sure I did not break anything so I'll wait a bit before making PRs.

Kbin federation still not work, with no error this time. Follow requests are just pending.

Logs

Two errors are preventing the follow to work correctly:
```
[error] Follower/Following counter update for https://kbin.social/m/kbinMeta failed.
{:error, "id must be a string"}
```
(If follow count sync option is enabled)

```
[error] Internal server error: %Phoenix.NotAcceptableError{message: "no supported media type in accept header.\n\nExpected one of [\"json\", \"jrd+json\", \"xml\", \"xrd+xml\"] but got the following formats:\n\n  * \" application/ld+json\" with extensions: [\"activity+json\", \"application/ld+json\"]\n  \"application/activity+json\" with extensions: [\"activity+json\", \"\"]\n\nTo accept custom formats, register them under the :mime library\nin your config/config.exs file:\n\n    config :mime, :types, %{\n      \"application/xml\" => [\"xml\"]\n    }\n\nAnd then run `mix deps.clean --build mim 
` to force it to be recompiled.\n", accepts: ["json", "jrd+json", "xml", "xrd+xml"], plug_status: 406}
```

Severity

I can manage

Have you searched for this issue?

I have double-checked and have not found this issue mentioned anywhere.

### Your setup OTP ### Extra details _No response_ ### Version v3.0.9 ### PostgreSQL version 12 ### What were you trying to do? I was trying to follow some communities and users on https://kbin.sh (https://kbin.sh/u/clovis, https://kbin.sh/m/oblivion) and lemmy.world (https://lemmy.world/c/maliciouscompliance) ### What did you expect to happen? I expected the accounts and communities to be followed and appear in my timelines ### What actually happened? The follow request were stuck waiting, no activities appeared on my timeline. I made two experimental fixes on my fork and I am now able to follow Lemmy communities https://github.com/BDX-town/Akkoma/commit/382a3e3a160e8244d3753511e2b4bada292cb4ee (See followers and following specs: https://www.w3.org/TR/activitypub/#followers-property) https://github.com/BDX-town/Akkoma/commit/629dfd23bc6292079d762c663d9ebb454b12ce53 I'm not sure I did not break anything so I'll wait a bit before making PRs. Kbin federation still not work, with no error this time. Follow requests are just pending. ### Logs ````shell Two errors are preventing the follow to work correctly: ``` [error] Follower/Following counter update for https://kbin.social/m/kbinMeta failed. {:error, "id must be a string"} ``` (If follow count sync option is enabled) ``` [error] Internal server error: %Phoenix.NotAcceptableError{message: "no supported media type in accept header.\n\nExpected one of [\"json\", \"jrd+json\", \"xml\", \"xrd+xml\"] but got the following formats:\n\n * \" application/ld+json\" with extensions: [\"activity+json\", \"application/ld+json\"]\n \"application/activity+json\" with extensions: [\"activity+json\", \"\"]\n\nTo accept custom formats, register them under the :mime library\nin your config/config.exs file:\n\n config :mime, :types, %{\n \"application/xml\" => [\"xml\"]\n }\n\nAnd then run `mix deps.clean --build mim ` to force it to be recompiled.\n", accepts: ["json", "jrd+json", "xml", "xrd+xml"], plug_status: 406} ``` ```` ### Severity I can manage ### Have you searched for this issue? - [x] I have double-checked and have not found this issue mentioned anywhere.

Clovis added the

bug

label 2023-06-18 16:09:01 +00:00

Clovis commented

2023-06-19 13:57:06 +00:00

Author

Contributor

There is also some issues regarding the transmogrifier:

Jun 19 13:52:11 bdx.town docker[1200230]: 13:52:11.500 [error] Error while fetching https://lemmy.ca/activities/like/8c04e987-7634-465e-91ff-acd229285df7: {:error, {:transmogrifier, :error}}
Jun 19 13:52:12 bdx.town docker[1200230]: 13:52:12.446 [error] Error while fetching https://lemmy.fmhy.ml/activities/like/10801c4f-0aeb-468b-9f8b-12e61736eff5: {:error, {:transmogrifier, :error}}

There is also some issues regarding the transmogrifier: ``` Jun 19 13:52:11 bdx.town docker[1200230]: 13:52:11.500 [error] Error while fetching https://lemmy.ca/activities/like/8c04e987-7634-465e-91ff-acd229285df7: {:error, {:transmogrifier, :error}} Jun 19 13:52:12 bdx.town docker[1200230]: 13:52:12.446 [error] Error while fetching https://lemmy.fmhy.ml/activities/like/10801c4f-0aeb-468b-9f8b-12e61736eff5: {:error, {:transmogrifier, :error}} ```

csolisr commented

2023-06-19 20:45:49 +00:00

Yeah, I already commented about the transmogrifier issues in upstream: https://git.pleroma.social/pleroma/pleroma/-/issues/3133

Also: whenever I follow a PeerTube or Lemmy/Kbin account, it takes forever to load the corresponding data when I manually browse each account from my instance.

Yeah, I already commented about the transmogrifier issues in upstream: https://git.pleroma.social/pleroma/pleroma/-/issues/3133 Also: whenever I follow a PeerTube or Lemmy/Kbin account, it takes forever to load the corresponding data when I manually browse each account from my instance.

XxXCertifiedForkliftDriverXxX commented

2023-06-20 14:50:48 +00:00

Contributor

Yeah, I already commented about the transmogrifier issues in upstream: https://git.pleroma.social/pleroma/pleroma/-/issues/3133

Also: whenever I follow a PeerTube or Lemmy/Kbin account, it takes forever to load the corresponding data when I manually browse each account from my instance.

Hi there, as a certified forklift driver, I'd like to clarify that Akkoma is not a "downstream" or "upstream" fork of Pleroma. Instead, it's an independent fork that doesn't try to follow or track Pleroma's development. It's like how Mobilizon uses Pleroma code, but has its own distinct goals and needs. It's important to avoid referring to Pleroma as "upstream", as the two projects are unrelated. I hope this clears up any confusion!

If you encounter any issues with Pleroma, please bring them up with the Pleroma team. Similarly, if you have any issues with Akkoma, please bring them up with the Akkoma team. Thank you!

> Yeah, I already commented about the transmogrifier issues in upstream: https://git.pleroma.social/pleroma/pleroma/-/issues/3133 > > Also: whenever I follow a PeerTube or Lemmy/Kbin account, it takes forever to load the corresponding data when I manually browse each account from my instance. Hi there, as a certified forklift driver, I'd like to clarify that Akkoma is not a "downstream" or "upstream" fork of Pleroma. Instead, it's an independent fork that doesn't try to follow or track Pleroma's development. It's like how Mobilizon uses Pleroma code, but has its own distinct goals and needs. It's important to avoid referring to Pleroma as "upstream", as the two projects are unrelated. I hope this clears up any confusion! If you encounter any issues with Pleroma, please bring them up with the Pleroma team. Similarly, if you have any issues with Akkoma, please bring them up with the Akkoma team. Thank you!

XxXCertifiedForkliftDriverXxX commented

2023-06-20 15:18:22 +00:00

Contributor

I made two experimental fixes on my fork and I am now able to follow Lemmy communities

382a3e3a16
(See followers and following specs: https://www.w3.org/TR/activitypub/#followers-property)

629dfd23bc

I'm not sure I did not break anything so I'll wait a bit before making PRs.

Kbin federation still not work, with no error this time. Follow requests are just pending.

Hello,

Thank you for submitting this pull request. However, I would advise against accepting the activity+json MIME type for WebFinger queries. The reason being that the returned data is not activity+json or even ld+json, but either jrd+json or xrd+xml. WebFinger has no support for returning activity+json data, and I am against returning the wrong data type to an HTTP request, which federation heavily relies on. Therefore, I suggest that Lemmy or kbin should fix their software to request the proper MIME types.

Regarding the fix for the followers/following collections, it should already be handled by maybe_update_follow_information/1, suggesting that the bug is somewhere else. Therefore, I would not suggest accepting this patch and would suggest trying to find the root of the issue.

Thank you again for your contribution.

Best regards,
Certified Forklift Driver

> I made two experimental fixes on my fork and I am now able to follow Lemmy communities > > https://github.com/BDX-town/Akkoma/commit/382a3e3a160e8244d3753511e2b4bada292cb4ee > (See followers and following specs: https://www.w3.org/TR/activitypub/#followers-property) > > https://github.com/BDX-town/Akkoma/commit/629dfd23bc6292079d762c663d9ebb454b12ce53 > > > > I'm not sure I did not break anything so I'll wait a bit before making PRs. > > Kbin federation still not work, with no error this time. Follow requests are just pending. Hello, Thank you for submitting this pull request. However, I would advise against accepting the `activity+json` MIME type for WebFinger queries. The reason being that the returned data is not `activity+json` or even `ld+json`, but either `jrd+json` or `xrd+xml`. WebFinger has no support for returning `activity+json` data, and I am against returning the wrong data type to an HTTP request, which federation heavily relies on. Therefore, I suggest that Lemmy or kbin should fix their software to request the proper MIME types. Regarding the fix for the followers/following collections, it should already be handled by `maybe_update_follow_information/1`, suggesting that the bug is somewhere else. Therefore, I would not suggest accepting this patch and would suggest trying to find the root of the issue. Thank you again for your contribution. Best regards, Certified Forklift Driver

Clovis commented

2023-06-20 16:12:25 +00:00

Author

Contributor

Hello,

Thank you for your review, indeed those patches are experimental, and I didn't submit a pull request because of that. My objective being to provide as much information as I can.
Those fixes do not allow to fix federation with Lemmy/Kbin since there is still issues even with them (Notably with the transmogriefier with Like/Dislike activities).

Concerning the MIME type, I'll take a look at Lemmy's code to understand what's going on.
Made an issue on their side: https://github.com/LemmyNet/lemmy/issues/3222

Thanks for pointing about maybe_update_follow_information/1, indeed the check made L1681 does not seems to work as expected.

Hello, Thank you for your review, indeed those patches are experimental, and I didn't submit a pull request because of that. My objective being to provide as much information as I can. Those fixes do not allow to fix federation with Lemmy/Kbin since there is still issues even with them (Notably with the transmogriefier with Like/Dislike activities). Concerning the MIME type, I'll take a look at Lemmy's code to understand what's going on. Made an issue on their side: https://github.com/LemmyNet/lemmy/issues/3222 Thanks for pointing about `maybe_update_follow_information/1`, indeed the check made L1681 does not seems to work as expected.

csolisr commented

2023-06-20 16:36:19 +00:00

Looks like Clovis has already posted the issue into the proper upstream - that is, the Lemmy issue tracker. Should I do the same thing on Kbin or do I defer to Clovis?

Looks like Clovis has already posted the issue into the proper upstream - that is, [the Lemmy issue tracker](https://github.com/LemmyNet/lemmy/issues/3222). Should I do the same thing on [Kbin](https://codeberg.org/Kbin/kbin-core/issues) or do I defer to Clovis?

Clovis commented

2023-06-27 21:20:55 +00:00

Author

Contributor

hi ! don't hesitate to create the issue for kbin !

Clovis commented

2023-07-26 08:10:41 +00:00

Author

Contributor

Lemmy has apparently fixed the issue on their side regarding MIME type

https://github.com/LemmyNet/lemmy/issues/3222

On akkoma's side there is still this maybe_update_follow_information/1 check on L1681 that does not seems to work as expected.

Lemmy has apparently fixed the issue on their side regarding MIME type https://github.com/LemmyNet/lemmy/issues/3222 On akkoma's side there is still this `maybe_update_follow_information/1` check on L1681 that does not seems to work as expected.

floatingghost commented

2023-07-27 13:00:01 +00:00

Owner

maybe_update_follow_information can fail without issue, that isn't really the biggest deal

subtype commented

2023-10-04 05:05:37 +00:00

First of all, I've disabled authfetch on my side, and I think it's the only way to federate with Lemmy at all currently.

Posting to lemmy works reliably if I already have a post to reply to on Akkoma side: https://awful.systems/comment/385

Searching for posts that weren't already fetched is wonky.

I can get posts from https://szmer.info reliably, from what I've seen.

For https://awful.systems it sometimes works and sometimes doesn't:
(UPDATE nevermind, it looks like Lemmy just doesn't serve .well-known/host-meta at all so that's not it)

Oct 04 04:59:40 vps-9a7f686e mix[52720]: 04:59:40.796 [warning] Can't find LRDD template in "https://awful.systems/.well-known/host-meta": {:ok, %Tesla.Env{method: :get, url: "https://awful.systems/.well-known/host-meta", query: [], headers: [{"server", "nginx"}, {"date", "Wed, 04 Oct 2023 04:59:40 GMT"}, {"content-length", "0"}, {"vary", "Origin, Access-Control-Request-Method, Access-Control-Request-Headers"}], body: "", status: 404, opts: [adapter: [receive_timeout: 15000, pool_timeout: 5000, name: MyFinch]], __module__: Tesla, __client__: %Tesla.Client{fun: nil, pre: [{Tesla.Middleware.FollowRedirects, :call, [[]]}, {Tesla.Middleware.Telemetry, :call, [[]]}], post: [], adapter: nil}}}

Fetching from https://lemmy.blahaj.zone doesn't seem to work at all, I'm getting errors like:

Oct 04 05:02:35 vps-9a7f686e mix[52720]: 05:02:35.771 [error] Could not decode user at fetch https://lemmy.blahaj.zone/post/3747482, {:ok, %Tesla.Env{method: :get, url: "https://lemmy.blahaj.zone/post/3747482", query: [], headers: [{"access-control-expose-headers", "vary, content-type, content-encoding"}, {"alt-svc", "h3=\":443\"; ma=2592000"}, {"content-type", "application/json"}, {"date", "Wed, 04 Oct 2023 05:02:35 GMT"}, {"referrer-policy", "no-referrer-when-downgrade"}, {"strict-transport-security", "max-age=31536000; include-subdomains;"}, {"vary", "accept-encoding, Origin, Access-Control-Request-Method, Access-Control-Request-Headers"}, {"x-content-type-options", "nosniff"}, {"x-frame-options", "DENY"}, {"x-robots-tag", "none"}, {"x-xss-protection", "1; mode=block"}, {"transfer-encoding", "chunked"}], body: "{\"error\":\"Object not local, fetch it from original instance\"}", status: 400, opts: [adapter: [receive_timeout: 15000, pool_timeout: 5000, name: MyFinch]], __module__: Tesla, __client__: %Tesla.Client{fun: nil, pre: [{Tesla.Middleware.FollowRedirects, :call, [[]]}, {Tesla.Middleware.Telemetry, :call, [[]]}], post: [], adapter: nil}}}
Oct 04 05:02:35 vps-9a7f686e mix[52720]: 05:02:35.772 [error] Could not fetch user https://lemmy.blahaj.zone/post/3747482, {nil, {:error, {:ok, %Tesla.Env{method: :get, url: "https://lemmy.blahaj.zone/post/3747482", query: [], headers: [{"access-control-expose-headers", "vary, content-type, content-encoding"}, {"alt-svc", "h3=\":443\"; ma=2592000"}, {"content-type", "application/json"}, {"date", "Wed, 04 Oct 2023 05:02:35 GMT"}, {"referrer-policy", "no-referrer-when-downgrade"}, {"strict-transport-security", "max-age=31536000; include-subdomains;"}, {"vary", "accept-encoding, Origin, Access-Control-Request-Method, Access-Control-Request-Headers"}, {"x-content-type-options", "nosniff"}, {"x-frame-options", "DENY"}, {"x-robots-tag", "none"}, {"x-xss-protection", "1; mode=block"}, {"transfer-encoding", "chunked"}], body: "{\"error\":\"Object not local, fetch it from original instance\"}", status: 400, opts: [adapter: [receive_timeout: 15000, pool_timeout: 5000, name: MyFinch]], __module__: Tesla, __client__: %Tesla.Client{fun: nil, pre: [{Tesla.Middleware.FollowRedirects, :call, [[]]}, {Tesla.Middleware.Telemetry, :call, [[]]}], post: [], adapter: nil}}}}}

I'll look more into this.

First of all, I've disabled authfetch on my side, and I think it's the only way to federate with Lemmy at all currently. Posting to lemmy works reliably if I already have a post to reply to on Akkoma side: https://awful.systems/comment/385 Searching for posts that weren't already fetched is wonky. I can get posts from https://szmer.info reliably, from what I've seen. For https://awful.systems it sometimes works and sometimes doesn't: (**UPDATE** nevermind, it looks like Lemmy just doesn't serve `.well-known/host-meta` at all so that's not it) ``` Oct 04 04:59:40 vps-9a7f686e mix[52720]: 04:59:40.796 [warning] Can't find LRDD template in "https://awful.systems/.well-known/host-meta": {:ok, %Tesla.Env{method: :get, url: "https://awful.systems/.well-known/host-meta", query: [], headers: [{"server", "nginx"}, {"date", "Wed, 04 Oct 2023 04:59:40 GMT"}, {"content-length", "0"}, {"vary", "Origin, Access-Control-Request-Method, Access-Control-Request-Headers"}], body: "", status: 404, opts: [adapter: [receive_timeout: 15000, pool_timeout: 5000, name: MyFinch]], __module__: Tesla, __client__: %Tesla.Client{fun: nil, pre: [{Tesla.Middleware.FollowRedirects, :call, [[]]}, {Tesla.Middleware.Telemetry, :call, [[]]}], post: [], adapter: nil}}} ``` Fetching from https://lemmy.blahaj.zone doesn't seem to work at all, I'm getting errors like: ``` Oct 04 05:02:35 vps-9a7f686e mix[52720]: 05:02:35.771 [error] Could not decode user at fetch https://lemmy.blahaj.zone/post/3747482, {:ok, %Tesla.Env{method: :get, url: "https://lemmy.blahaj.zone/post/3747482", query: [], headers: [{"access-control-expose-headers", "vary, content-type, content-encoding"}, {"alt-svc", "h3=\":443\"; ma=2592000"}, {"content-type", "application/json"}, {"date", "Wed, 04 Oct 2023 05:02:35 GMT"}, {"referrer-policy", "no-referrer-when-downgrade"}, {"strict-transport-security", "max-age=31536000; include-subdomains;"}, {"vary", "accept-encoding, Origin, Access-Control-Request-Method, Access-Control-Request-Headers"}, {"x-content-type-options", "nosniff"}, {"x-frame-options", "DENY"}, {"x-robots-tag", "none"}, {"x-xss-protection", "1; mode=block"}, {"transfer-encoding", "chunked"}], body: "{\"error\":\"Object not local, fetch it from original instance\"}", status: 400, opts: [adapter: [receive_timeout: 15000, pool_timeout: 5000, name: MyFinch]], __module__: Tesla, __client__: %Tesla.Client{fun: nil, pre: [{Tesla.Middleware.FollowRedirects, :call, [[]]}, {Tesla.Middleware.Telemetry, :call, [[]]}], post: [], adapter: nil}}} Oct 04 05:02:35 vps-9a7f686e mix[52720]: 05:02:35.772 [error] Could not fetch user https://lemmy.blahaj.zone/post/3747482, {nil, {:error, {:ok, %Tesla.Env{method: :get, url: "https://lemmy.blahaj.zone/post/3747482", query: [], headers: [{"access-control-expose-headers", "vary, content-type, content-encoding"}, {"alt-svc", "h3=\":443\"; ma=2592000"}, {"content-type", "application/json"}, {"date", "Wed, 04 Oct 2023 05:02:35 GMT"}, {"referrer-policy", "no-referrer-when-downgrade"}, {"strict-transport-security", "max-age=31536000; include-subdomains;"}, {"vary", "accept-encoding, Origin, Access-Control-Request-Method, Access-Control-Request-Headers"}, {"x-content-type-options", "nosniff"}, {"x-frame-options", "DENY"}, {"x-robots-tag", "none"}, {"x-xss-protection", "1; mode=block"}, {"transfer-encoding", "chunked"}], body: "{\"error\":\"Object not local, fetch it from original instance\"}", status: 400, opts: [adapter: [receive_timeout: 15000, pool_timeout: 5000, name: MyFinch]], __module__: Tesla, __client__: %Tesla.Client{fun: nil, pre: [{Tesla.Middleware.FollowRedirects, :call, [[]]}, {Tesla.Middleware.Telemetry, :call, [[]]}], post: [], adapter: nil}}}}} ``` I'll look more into this.

snan commented

2024-02-06 11:23:58 +00:00

Contributor

Huh, I have been able to post to Lemmy and Kbin from Akkoma for months; it doesn't always but it it usually does.

Here is one example: https://lemmy.zip/post/2441566

But it stopped working early January, maybe because I turned on AF. I turned it off today but not sure of that really helped.

It doesn't work on Kbin or PieFed either. PieFed I've never gotten it to work but it worked fine on Kbin and Lemmy before 🤷🏻‍♀️

I see and boost their threads and comments on my own instance just fine and my own replies on my own instance, they just don't show up on the remote.

Huh, I have been able to post to Lemmy and Kbin from Akkoma for months; it doesn't always but it it usually does. Here is one example: https://lemmy.zip/post/2441566 But it stopped working early January, maybe because I turned on AF. I turned it off today but not sure of that really helped. It doesn't work on Kbin or PieFed either. PieFed I've never gotten it to work but it worked fine on Kbin and Lemmy before 🤷🏻‍♀️ I see and boost their threads and comments on my own instance just fine and my own replies on my own instance, they just don't show up on the remote.

snan commented

2024-02-07 12:38:40 +00:00

Contributor

https://github.com/LemmyNet/lemmy/issues/4336#issuecomment-1931801089 🤷🏻‍♀️

Oneric commented

2024-02-07 22:39:03 +00:00

Member

@snan the second post you referenced on GitHUb also doesn’t show up on akkoma-fe, are you sure this is a publicly-visible, non-deleted post?

As for getting the activity, you can directly get all related activities (in case it becomes relevant) from the database like below, assuming default user and DB names:

Connect to the PostgreSQL shell, e.g. sudo -u postgres psql (adjust for Docker)
In the psql shell connect to the akkoma database: \c akkoma akkoma

Query all related (non-array, but should be fine for here) activities:

SELECT data ->> 'id' AS activity, data ->> 'type' AS type
FROM activities
WHERE data ->> 'object' = 'https://idiomdrottning.org/objects/6db9e809-b898-4fa2-89b7-e4f06c439a91'
;

And as a suggestion, it may be helpful for Lemmy devs if you also provide an example of an older post which did federate successfully as a point of comparison :)

@snan the second post you referenced on GitHUb also doesn’t show up on akkoma-fe, are you sure this is a publicly-visible, non-deleted post? As for getting the activity, you can directly get *all* related activities (in case it becomes relevant) from the database like below, assuming default user and DB names: 1. Connect to the PostgreSQL shell, e.g. `sudo -u postgres psql` (adjust for Docker) 2. In the psql shell connect to the akkoma database: `\c akkoma akkoma` 3. Query all related (non-array, but should be fine for here) activities: ```sql SELECT data ->> 'id' AS activity, data ->> 'type' AS type FROM activities WHERE data ->> 'object' = 'https://idiomdrottning.org/objects/6db9e809-b898-4fa2-89b7-e4f06c439a91' ; ``` And as a suggestion, it may be helpful for Lemmy devs if you also provide an example of an older post which *did* federate successfully as a point of comparison :)

snan commented

2024-02-08 11:18:31 +00:00

Contributor

Wonderful advice, thank you ♥︎
I'll get on that 🙏🏻

Wonderful advice, thank you ♥︎ I'll get on that 🙏🏻