AkkomaGang/akkoma

Fork 92

[bug] Akkoma unable to parse non-ASCII usernames from other instances #787

New issue

Open

opened 2024-05-31 06:39:09 +00:00 by norm · 4 comments

norm commented

2024-05-31 06:39:09 +00:00

Contributor

Your setup

From source

Extra details

Fedora 40

Version

8afc3bee7a

PostgreSQL version

What were you trying to do?

Fetching a post made by a user with a non-ASCII username (example: https://mastodon.vierkantor.com/@%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1/112525442573810213)

What did you expect to happen?

Post gets fetched and Akkoma runs happily.

What actually happened?

Akkoma keeps retrying to fetch the post and it associated user, but is unable to. Instead it keeps retrying until it runs out of memory and the OOM killer terminates beam.smp.

This doesn't seem to hardlock the instance though once it gets restarted by systemd.

Logs

13:50:37.080 [error] Could not fetch user https://mastodon.vierkantor.com/users/%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1, {nil, {:error, #Ecto.Changeset<action: :insert, changes: %{name: ":anne_triangle: αἰμ0", fields: [%{"name" => "pronouns?", "value" => "she/her!"}, %{"name" => "adult?", "value" => "yep!"}, %{"name" => "languages?", "value" => "english mostly!"}, %{"name" => "can i?", "value" => "you must! (call me a good girl! hehe awawawawa)"}], uri: "https://mastodon.vierkantor.com/@%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1", public_key: "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsXt6mFGiT4viEL0x601h\nx18+AHB22s2eNzaVT6VvDdwUURGf3BFeElAcYUMFkwzYbFfV0CBGulPabVBPEmnv\n8go64fW/xcfL8giG9NglMMSLrqQL1JMoErosTuQ20DYd2o4I1W7kBP+O6X8m4CUH\n6Fd3xX7sVIMmEw8gHxQQPYtamjYK/zANGFgR/I4eWbYgnN6YI4KpsXdvT9Vci1Jk\nPuTfvfFQ+kE1rAsXlUdCZ7vhtW6BgFRvrEwMxEqFMkB0lToyqugDF2DR2ehhIYr+\nYdMCPy2U5tUMm4fOrureVe6hsHYgqYcmwJxOt3GZZUtHpUP9gLq+v/zNNlSxlZGp\nbwIDAQAB\n-----END PUBLIC KEY-----\n", emoji: %{"anne_triangle" => "https://mastodon.vierkantor.com/system/custom_emojis/images/000/052/114/original/5d3152f88443ed3e.png"}, background: nil, nickname: "εὐδαιμονία@mastodon.vierkantor.com", ap_id: "https://mastodon.vierkantor.com/users/%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1", bio: "<p>hi!!! hello! I&#39;m αἰμ0, which is short for εὐδαιμονία! I like being a girl and breaking most fediverse instances with my username!</p><p>I am not on here a lot, only when I&#39;m happy but perhaps I&#39;ll learn to be happy more often one day! awawawawawawa ← oh yeah I like stimming so you should expect a lot of awawawawa :D</p><p>you can request a follow for a price! :O :O you must call me a good girl!! hehe :D :D 💚 💚</p>", is_locked: true, pinned_objects: %{"https://mastodon.vierkantor.com/users/%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1/statuses/112525442573810213" => ~N[2024-05-29 17:50:36.842730]}, featured_address: "https://mastodon.vierkantor.com/users/%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1/collections/featured", avatar: %{"type" => "Image", "url" => [%{"href" => "https://mastodon.vierkantor.com/system/accounts/avatars/112/509/705/929/410/879/original/162371bc06cf8f61.jpg"}]}, follower_address: "https://mastodon.vierkantor.com/users/%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1/followers", banner: nil, ap_enabled: true, following_address: "https://mastodon.vierkantor.com/users/%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1/following", inbox: "https://mastodon.vierkantor.com/users/%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1/inbox", last_refreshed_at: ~N[2024-05-29 17:50:37.080372], shared_inbox: "https://mastodon.vierkantor.com/inbox"}, errors: [nickname: {"has invalid format", [validation: :format]}], data: #Pleroma.User<>, valid?: false>}}

13:51:23.216 [error] Object rejected while fetching https://mastodon.vierkantor.com/users/%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1/statuses/112525442573810213 {:transmogrifier, {:error, :not_found}}

Out of memory: Killed process 132381 (beam.smp) total-vm:11295672kB, anon-rss:2088160kB, file-rss:128kB, shmem-rss:16128kB, UID:985 pgtables:17408kB oom_score_adj:0

Severity

I cannot use it as easily as I'd like

Have you searched for this issue?

I have double-checked and have not found this issue mentioned anywhere.

### Your setup From source ### Extra details Fedora 40 ### Version 8afc3bee7a ### PostgreSQL version 16 ### What were you trying to do? Fetching a post made by a user with a non-ASCII username (example: <https://mastodon.vierkantor.com/@%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1/112525442573810213>) ### What did you expect to happen? Post gets fetched and Akkoma runs happily. ### What actually happened? Akkoma keeps retrying to fetch the post and it associated user, but is unable to. Instead it keeps retrying until it runs out of memory and the OOM killer terminates beam.smp. This doesn't seem to hardlock the instance though once it gets restarted by systemd. ### Logs ````shell 13:50:37.080 [error] Could not fetch user https://mastodon.vierkantor.com/users/%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1, {nil, {:error, #Ecto.Changeset<action: :insert, changes: %{name: ":anne_triangle: αἰμ0", fields: [%{"name" => "pronouns?", "value" => "she/her!"}, %{"name" => "adult?", "value" => "yep!"}, %{"name" => "languages?", "value" => "english mostly!"}, %{"name" => "can i?", "value" => "you must! (call me a good girl! hehe awawawawa)"}], uri: "https://mastodon.vierkantor.com/@%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1", public_key: "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsXt6mFGiT4viEL0x601h\nx18+AHB22s2eNzaVT6VvDdwUURGf3BFeElAcYUMFkwzYbFfV0CBGulPabVBPEmnv\n8go64fW/xcfL8giG9NglMMSLrqQL1JMoErosTuQ20DYd2o4I1W7kBP+O6X8m4CUH\n6Fd3xX7sVIMmEw8gHxQQPYtamjYK/zANGFgR/I4eWbYgnN6YI4KpsXdvT9Vci1Jk\nPuTfvfFQ+kE1rAsXlUdCZ7vhtW6BgFRvrEwMxEqFMkB0lToyqugDF2DR2ehhIYr+\nYdMCPy2U5tUMm4fOrureVe6hsHYgqYcmwJxOt3GZZUtHpUP9gLq+v/zNNlSxlZGp\nbwIDAQAB\n-----END PUBLIC KEY-----\n", emoji: %{"anne_triangle" => "https://mastodon.vierkantor.com/system/custom_emojis/images/000/052/114/original/5d3152f88443ed3e.png"}, background: nil, nickname: "εὐδαιμονία@mastodon.vierkantor.com", ap_id: "https://mastodon.vierkantor.com/users/%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1", bio: "hi!!! hello! I'm αἰμ0, which is short for εὐδαιμονία! I like being a girl and breaking most fediverse instances with my username!I am not on here a lot, only when I'm happy but perhaps I'll learn to be happy more often one day! awawawawawawa ← oh yeah I like stimming so you should expect a lot of awawawawa :Dyou can request a follow for a price! :O :O you must call me a good girl!! hehe :D :D 💚 💚", is_locked: true, pinned_objects: %{"https://mastodon.vierkantor.com/users/%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1/statuses/112525442573810213" => ~N[2024-05-29 17:50:36.842730]}, featured_address: "https://mastodon.vierkantor.com/users/%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1/collections/featured", avatar: %{"type" => "Image", "url" => [%{"href" => "https://mastodon.vierkantor.com/system/accounts/avatars/112/509/705/929/410/879/original/162371bc06cf8f61.jpg"}]}, follower_address: "https://mastodon.vierkantor.com/users/%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1/followers", banner: nil, ap_enabled: true, following_address: "https://mastodon.vierkantor.com/users/%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1/following", inbox: "https://mastodon.vierkantor.com/users/%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1/inbox", last_refreshed_at: ~N[2024-05-29 17:50:37.080372], shared_inbox: "https://mastodon.vierkantor.com/inbox"}, errors: [nickname: {"has invalid format", [validation: :format]}], data: #Pleroma.User<>, valid?: false>}} 13:51:23.216 [error] Object rejected while fetching https://mastodon.vierkantor.com/users/%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1/statuses/112525442573810213 {:transmogrifier, {:error, :not_found}} Out of memory: Killed process 132381 (beam.smp) total-vm:11295672kB, anon-rss:2088160kB, file-rss:128kB, shmem-rss:16128kB, UID:985 pgtables:17408kB oom_score_adj:0 ```` ### Severity I cannot use it as easily as I'd like ### Have you searched for this issue? - [x] I have double-checked and have not found this issue mentioned anywhere.

norm added the

bug

label 2024-05-31 06:39:09 +00:00

norm referenced this issue

2024-05-31 06:48:51 +00:00

[bug] Akkoma getting OOM killed #736

norm commented

2024-05-31 06:53:34 +00:00

Author

Contributor

Seems like the server OOMing when the queue gets stuck is a bigger issue (see #736), but I think I'll leave this open as I do think we should do something about non-ASCII usernames.

floatingghost commented

2024-05-31 06:55:47 +00:00

Owner

by all accounts the actual rejection is correct, we (follow masto in that regard)[https://github.com/mastodon/mastodon/blob/main/app/models/account.rb#L70]

the question is, why doesn't it just throw the fetch away

by all accounts the actual rejection is correct, we (follow masto in that regard)[https://github.com/mastodon/mastodon/blob/main/app/models/account.rb#L70] the question is, why doesn't it just throw the fetch away

floatingghost commented

2024-05-31 10:21:41 +00:00

Owner

The OOM part of this is fixed by #788

Oneric commented

2024-05-31 15:59:44 +00:00

Member

Curiously the actor from AkkomaGang/akkoma-fe#372 (@你好@i18n.viii.fi / https://i18n.viii.fi/%E4%BD%A0%E5%A5%BD) with a CJK username and also URL-encode sequences in the AP ID can successfully be fetched on current develop. While the one mentioned here cannot (@εὐδαιμονία@mastodon.vierkantor.com / https://mastodon.vierkantor.com/@%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1)
Both use non-ASCII letters in both name and preferredUsername

The name of our validation regex makes it sound like it is only supposed to apply to local actors, which would explain how one of these passes at all. I’m not completely sure how to read ruby db schema definitions, but the linked Mastodon regex might also mostly only apply to local actors: d326ad0ed9/app/models/account.rb (L100)

Note how some of the error logs dropped the percentage signs from the URL:

[error] Could not decode user at fetch https://mastodon.vierkantor.com/@CEB5E1BD90CEB4CEB1CEB9CEBCCEBFCEBDCEAFCEB1, :not_found
[error] Could not fetch user https://mastodon.vierkantor.com/@CEB5E1BD90CEB4CEB1CEB9CEBCCEBFCEBDCEAFCEB1, {nil, {:error, :not_found}}
[error] Object rejected while fetching https://mastodon.vierkantor.com/@%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1 {:transmogrifier, :error}
[error] Could not fetch user https://mastodon.vierkantor.com/users/%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1, {nil, {:error, ....

Curiously the actor from https://akkoma.dev/AkkomaGang/akkoma-fe/issues/372 (`@你好@i18n.viii.fi` / `https://i18n.viii.fi/%E4%BD%A0%E5%A5%BD`) with a CJK username and also URL-encode sequences in the AP ID can successfully be fetched on current develop. While the one mentioned here cannot (`@εὐδαιμονία@mastodon.vierkantor.com ` / `https://mastodon.vierkantor.com/@%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1`) Both use non-ASCII letters in both `name` and `preferredUsername` The name of our validation regex makes it sound like it is only supposed to apply to local actors, which would explain how one of these passes at all. I’m not completely sure how to read ruby db schema definitions, but the linked Mastodon regex might also mostly only apply to local actors: https://github.com/mastodon/mastodon/blob/d326ad0ed9a2e874213c9313f25b973638e4b94d/app/models/account.rb#L100 Note how some of the error logs dropped the percentage signs from the URL: ``` [error] Could not decode user at fetch https://mastodon.vierkantor.com/@CEB5E1BD90CEB4CEB1CEB9CEBCCEBFCEBDCEAFCEB1, :not_found [error] Could not fetch user https://mastodon.vierkantor.com/@CEB5E1BD90CEB4CEB1CEB9CEBCCEBFCEBDCEAFCEB1, {nil, {:error, :not_found}} [error] Object rejected while fetching https://mastodon.vierkantor.com/@%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1 {:transmogrifier, :error} [error] Could not fetch user https://mastodon.vierkantor.com/users/%CE%B5%E1%BD%90%CE%B4%CE%B1%CE%B9%CE%BC%CE%BF%CE%BD%CE%AF%CE%B1, {nil, {:error, .... ```

No milestone

No project

No assignees

3 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: AkkomaGang/akkoma#787

No description provided.