[bug] Remote instances that reuse domains and nicknames cause Akkoma to generate excessive requests #419

New issue

Closed

opened 2023-01-04 19:10:47 +00:00 by Dee · 1 comment

Dee commented

2023-01-04 19:10:47 +00:00

Your setup

From source

Extra details

NixOS 22.11

Version

3.5.0

PostgreSQL version

15.1

What were you trying to do?

A remote instance, at some point in early January, has installed Mastodon (Hometown fork) on a domain that previous held a Misskey install, with no migration. At least one nickname (webfinger address) was reused: it used to point to a Misskey user, and now it is pointing to a Hometown user.

My instance was aware of the remote instance prior to the reinstall. My instances table includes an entry for the instance with cached nodeinfo data that mentions Misskey. My users table also includes two different rows for the user.

What did you expect to happen?

I expected things to break, I did not expect them to break as terribly.

What actually happened?

At some point my instance apparently tried to refresh its data about the remote user. It appears to have hit a database constraint related to the nickname being duplicated. It then continued to attempt to fetch the data, apparently multiple times per second, apparently with outgoing Web traffic for every attempt. It only stopped after a VM reboot (for unrelated reasons).

Logs

Jan 03 11:48:19 perihelion postgres[525703]: [525703] STATEMENT:  UPDATE "users" SET "also_known_as" = $1, "avatar" = $2, "banner" = $3, "bio" = $4, "fields" = $5, "follower_address" = $6, "last_refreshed_at" = $7, "name" = $8, "nickname" = $9, "pinned_objects" = $10, "public_key" = $11, "shared_inbox" = $12, "updated_at" = $13 WHERE "id" = $14
Jan 03 11:48:20 perihelion postgres[525709]: [525709] ERROR:  duplicate key value violates unique constraint "users_nickname_index"
Jan 03 11:48:20 perihelion postgres[525709]: [525709] DETAIL:  Key (nickname)=(hatch@creacher.club) already exists.
Jan 03 11:48:20 perihelion postgres[525709]: [525709] STATEMENT:  UPDATE "users" SET "also_known_as" = $1, "avatar" = $2, "banner" = $3, "bio" = $4, "fields" = $5, "follower_address" = $6, "last_refreshed_at" = $7, "name" = $8, "nickname" = $9, "pinned_objects" = $10, "public_key" = $11, "shared_inbox" = $12, "updated_at" = $13 WHERE "id" = $14
Jan 03 11:48:19 perihelion akkoma[38830]: [error] Could not fetch page https://creacher.club/users/hatch/statuses/109620471780924323/replies?min_id=109620676793607866&page=true - {:ok, %Tesla.Env{__client__: %Tesla.Client{adapter: nil, fun: nil, post: [], pre: [{Tesla.Middleware.FollowRedirects, :call, [[]]}]}, __module__: Tesla, body: "{\"error\":\"Too many requests\"}", headers: [{"date", "Tue, 03 Jan 2023 11:48:19 GMT"}, {"content-type", "application/json"}, {"transfer-encoding", "chunked"}, {"connection", "keep-alive"}, {"x-ratelimit-limit", "300"}, {"x-ratelimit-remaining", "0"}, {"x-ratelimit-reset", "2023-01-03T12:00:00.760188Z"}, {"cache-control", "no-cache"}, {"content-security-policy", "base-uri 'none'; default-src 'none'; frame-ancestors 'none'; font-src 'self' https://creacher.club; img-src 'self' https: data: blob: https://creacher.club; style-src 'self' https://creacher.club 'nonce-rhKRekDWZOOe+e7FrGTiDA=='; media-src 'self' https: data: https://creacher.club; frame-src 'self' https:; manifest-src 'self' https://creacher.club; connect-src 'self' data: blob: https://creacher.club https://creacher.club wss://creacher.club; script-src 'self' https://creacher.club 'wasm-unsafe-eval'; child-src 'self' blob: https://creacher.club; worker-src 'self' blob: https://creacher.club"}, {"x-request-id", "464ed17a-942e-4616-94e7-bf796a6ab7cf"}, {"x-runtime", "0.001403"}, {"strict-transport-security", "max-age=63072000; includeSubDomains"}], method: :get, opts: [adapter: [receive_timeout: 15000, pool_timeout: 5000, name: MyFinch]], query: [], status: 429, url: "https://creacher.club/users/hatch/statuses/109620471780924323/replies?min_id=109620676793607866&page=true"}}
Jan 03 11:48:20 perihelion postgres[525126]: [525126] ERROR:  duplicate key value violates unique constraint "users_nickname_index"
Jan 03 11:48:20 perihelion postgres[525126]: [525126] DETAIL:  Key (nickname)=(hatch@creacher.club) already exists.
Jan 03 11:48:20 perihelion postgres[525126]: [525126] STATEMENT:  UPDATE "users" SET "also_known_as" = $1, "avatar" = $2, "banner" = $3, "bio" = $4, "fields" = $5, "follower_address" = $6, "last_refreshed_at" = $7, "name" = $8, "nickname" = $9, "pinned_objects" = $10, "public_key" = $11, "shared_inbox" = $12, "updated_at" = $13 WHERE "id" = $14
Jan 03 11:48:19 perihelion akkoma[38830]: [error] Could not fetch page https://creacher.club/users/hatch/statuses/109620471780924323/replies?min_id=109620676793607866&page=true - {:ok, %Tesla.Env{__client__: %Tesla.Client{adapter: nil, fun: nil, post: [], pre: [{Tesla.Middleware.FollowRedirects, :call, [[]]}]}, __module__: Tesla, body: "{\"error\":\"Too many requests\"}", headers: [{"date", "Tue, 03 Jan 2023 11:48:19 GMT"}, {"content-type", "application/json"}, {"transfer-encoding", "chunked"}, {"connection", "keep-alive"}, {"x-ratelimit-limit", "300"}, {"x-ratelimit-remaining", "0"}, {"x-ratelimit-reset", "2023-01-03T12:00:00.795245Z"}, {"cache-control", "no-cache"}, {"content-security-policy", "base-uri 'none'; default-src 'none'; frame-ancestors 'none'; font-src 'self' https://creacher.club; img-src 'self' https: data: blob: https://creacher.club; style-src 'self' https://creacher.club 'nonce-N6OOzFmwIqHZAcRyTHtZ8g=='; media-src 'self' https: data: https://creacher.club; frame-src 'self' https:; manifest-src 'self' https://creacher.club; connect-src 'self' data: blob: https://creacher.club https://creacher.club wss://creacher.club; script-src 'self' https://creacher.club 'wasm-unsafe-eval'; child-src 'self' blob: https://creacher.club; worker-src 'self' blob: https://creacher.club"}, {"x-request-id", "b3016292-45ef-44de-98fb-650c6eca457d"}, {"x-runtime", "0.001668"}, {"strict-transport-security", "max-age=63072000; includeSubDomains"}], method: :get, opts: [adapter: [receive_timeout: 15000, pool_timeout: 5000, name: MyFinch]], query: [], status: 429, url: "https://creacher.club/users/hatch/statuses/109620471780924323/replies?min_id=109620676793607866&page=true"}}
Jan 03 11:48:20 perihelion postgres[525709]: [525709] ERROR:  duplicate key value violates unique constraint "users_nickname_index"
Jan 03 11:48:20 perihelion postgres[525709]: [525709] DETAIL:  Key (nickname)=(hatch@creacher.club) already exists.
Jan 03 11:48:20 perihelion postgres[525709]: [525709] STATEMENT:  UPDATE "users" SET "also_known_as" = $1, "avatar" = $2, "banner" = $3, "bio" = $4, "fields" = $5, "follower_address" = $6, "last_refreshed_at" = $7, "name" = $8, "nickname" = $9, "pinned_objects" = $10, "public_key" = $11, "shared_inbox" = $12, "updated_at" = $13 WHERE "id" = $14
Jan 03 11:48:19 perihelion akkoma[38830]: [error] Could not fetch page https://creacher.club/users/hatch/statuses/109620471780924323/replies?min_id=109620676793607866&page=true - {:ok, %Tesla.Env{__client__: %Tesla.Client{adapter: nil, fun: nil, post: [], pre: [{Tesla.Middleware.FollowRedirects, :call, [[]]}]}, __module__: Tesla, body: "{\"error\":\"Too many requests\"}", headers: [{"date", "Tue, 03 Jan 2023 11:48:19 GMT"}, {"content-type", "application/json"}, {"transfer-encoding", "chunked"}, {"connection", "keep-alive"}, {"x-ratelimit-limit", "300"}, {"x-ratelimit-remaining", "0"}, {"x-ratelimit-reset", "2023-01-03T12:00:00.867446Z"}, {"cache-control", "no-cache"}, {"content-security-policy", "base-uri 'none'; default-src 'none'; frame-ancestors 'none'; font-src 'self' https://creacher.club; img-src 'self' https: data: blob: https://creacher.club; style-src 'self' https://creacher.club 'nonce-lp4Jhgr/s0DDf68WcdyiGg=='; media-src 'self' https: data: https://creacher.club; frame-src 'self' https:; manifest-src 'self' https://creacher.club; connect-src 'self' data: blob: https://creacher.club https://creacher.club wss://creacher.club; script-src 'self' https://creacher.club 'wasm-unsafe-eval'; child-src 'self' blob: https://creacher.club; worker-src 'self' blob: https://creacher.club"}, {"x-request-id", "025f84da-616c-4195-b1bd-268c4794d1ce"}, {"x-runtime", "0.001741"}, {"strict-transport-security", "max-age=63072000; includeSubDomains"}], method: :get, opts: [adapter: [receive_timeout: 15000, pool_timeout: 5000, name: MyFinch]], query: [], status: 429, url: "https://creacher.club/users/hatch/statuses/109620471780924323/replies?min_id=109620676793607866&page=true"}}

I do not have logs of this starting, since this has produced dozens of lines per second for hours and earlier logs got pruned.

Severity

I cannot use it as easily as I'd like

Have you searched for this issue?

I have double-checked and have not found this issue mentioned anywhere.

### Your setup From source ### Extra details NixOS 22.11 ### Version 3.5.0 ### PostgreSQL version 15.1 ### What were you trying to do? A remote instance, at some point in early January, has installed Mastodon (Hometown fork) on a domain that previous held a Misskey install, with no migration. At least one nickname (webfinger address) was reused: it used to point to a Misskey user, and now it is pointing to a Hometown user. My instance was aware of the remote instance prior to the reinstall. My `instances` table includes an entry for the instance with cached nodeinfo data that mentions Misskey. My `users` table also includes two different rows for the user. ### What did you expect to happen? I expected things to break, I did not expect them to break as terribly. ### What actually happened? At some point my instance apparently tried to refresh its data about the remote user. It appears to have hit a database constraint related to the nickname being duplicated. It then continued to attempt to fetch the data, apparently multiple times per second, apparently with outgoing Web traffic for every attempt. It only stopped after a VM reboot (for unrelated reasons). ### Logs ``` Jan 03 11:48:19 perihelion postgres[525703]: [525703] STATEMENT: UPDATE "users" SET "also_known_as" = $1, "avatar" = $2, "banner" = $3, "bio" = $4, "fields" = $5, "follower_address" = $6, "last_refreshed_at" = $7, "name" = $8, "nickname" = $9, "pinned_objects" = $10, "public_key" = $11, "shared_inbox" = $12, "updated_at" = $13 WHERE "id" = $14 Jan 03 11:48:20 perihelion postgres[525709]: [525709] ERROR: duplicate key value violates unique constraint "users_nickname_index" Jan 03 11:48:20 perihelion postgres[525709]: [525709] DETAIL: Key (nickname)=(hatch@creacher.club) already exists. Jan 03 11:48:20 perihelion postgres[525709]: [525709] STATEMENT: UPDATE "users" SET "also_known_as" = $1, "avatar" = $2, "banner" = $3, "bio" = $4, "fields" = $5, "follower_address" = $6, "last_refreshed_at" = $7, "name" = $8, "nickname" = $9, "pinned_objects" = $10, "public_key" = $11, "shared_inbox" = $12, "updated_at" = $13 WHERE "id" = $14 Jan 03 11:48:19 perihelion akkoma[38830]: [error] Could not fetch page https://creacher.club/users/hatch/statuses/109620471780924323/replies?min_id=109620676793607866&page=true - {:ok, %Tesla.Env{__client__: %Tesla.Client{adapter: nil, fun: nil, post: [], pre: [{Tesla.Middleware.FollowRedirects, :call, [[]]}]}, __module__: Tesla, body: "{\"error\":\"Too many requests\"}", headers: [{"date", "Tue, 03 Jan 2023 11:48:19 GMT"}, {"content-type", "application/json"}, {"transfer-encoding", "chunked"}, {"connection", "keep-alive"}, {"x-ratelimit-limit", "300"}, {"x-ratelimit-remaining", "0"}, {"x-ratelimit-reset", "2023-01-03T12:00:00.760188Z"}, {"cache-control", "no-cache"}, {"content-security-policy", "base-uri 'none'; default-src 'none'; frame-ancestors 'none'; font-src 'self' https://creacher.club; img-src 'self' https: data: blob: https://creacher.club; style-src 'self' https://creacher.club 'nonce-rhKRekDWZOOe+e7FrGTiDA=='; media-src 'self' https: data: https://creacher.club; frame-src 'self' https:; manifest-src 'self' https://creacher.club; connect-src 'self' data: blob: https://creacher.club https://creacher.club wss://creacher.club; script-src 'self' https://creacher.club 'wasm-unsafe-eval'; child-src 'self' blob: https://creacher.club; worker-src 'self' blob: https://creacher.club"}, {"x-request-id", "464ed17a-942e-4616-94e7-bf796a6ab7cf"}, {"x-runtime", "0.001403"}, {"strict-transport-security", "max-age=63072000; includeSubDomains"}], method: :get, opts: [adapter: [receive_timeout: 15000, pool_timeout: 5000, name: MyFinch]], query: [], status: 429, url: "https://creacher.club/users/hatch/statuses/109620471780924323/replies?min_id=109620676793607866&page=true"}} Jan 03 11:48:20 perihelion postgres[525126]: [525126] ERROR: duplicate key value violates unique constraint "users_nickname_index" Jan 03 11:48:20 perihelion postgres[525126]: [525126] DETAIL: Key (nickname)=(hatch@creacher.club) already exists. Jan 03 11:48:20 perihelion postgres[525126]: [525126] STATEMENT: UPDATE "users" SET "also_known_as" = $1, "avatar" = $2, "banner" = $3, "bio" = $4, "fields" = $5, "follower_address" = $6, "last_refreshed_at" = $7, "name" = $8, "nickname" = $9, "pinned_objects" = $10, "public_key" = $11, "shared_inbox" = $12, "updated_at" = $13 WHERE "id" = $14 Jan 03 11:48:19 perihelion akkoma[38830]: [error] Could not fetch page https://creacher.club/users/hatch/statuses/109620471780924323/replies?min_id=109620676793607866&page=true - {:ok, %Tesla.Env{__client__: %Tesla.Client{adapter: nil, fun: nil, post: [], pre: [{Tesla.Middleware.FollowRedirects, :call, [[]]}]}, __module__: Tesla, body: "{\"error\":\"Too many requests\"}", headers: [{"date", "Tue, 03 Jan 2023 11:48:19 GMT"}, {"content-type", "application/json"}, {"transfer-encoding", "chunked"}, {"connection", "keep-alive"}, {"x-ratelimit-limit", "300"}, {"x-ratelimit-remaining", "0"}, {"x-ratelimit-reset", "2023-01-03T12:00:00.795245Z"}, {"cache-control", "no-cache"}, {"content-security-policy", "base-uri 'none'; default-src 'none'; frame-ancestors 'none'; font-src 'self' https://creacher.club; img-src 'self' https: data: blob: https://creacher.club; style-src 'self' https://creacher.club 'nonce-N6OOzFmwIqHZAcRyTHtZ8g=='; media-src 'self' https: data: https://creacher.club; frame-src 'self' https:; manifest-src 'self' https://creacher.club; connect-src 'self' data: blob: https://creacher.club https://creacher.club wss://creacher.club; script-src 'self' https://creacher.club 'wasm-unsafe-eval'; child-src 'self' blob: https://creacher.club; worker-src 'self' blob: https://creacher.club"}, {"x-request-id", "b3016292-45ef-44de-98fb-650c6eca457d"}, {"x-runtime", "0.001668"}, {"strict-transport-security", "max-age=63072000; includeSubDomains"}], method: :get, opts: [adapter: [receive_timeout: 15000, pool_timeout: 5000, name: MyFinch]], query: [], status: 429, url: "https://creacher.club/users/hatch/statuses/109620471780924323/replies?min_id=109620676793607866&page=true"}} Jan 03 11:48:20 perihelion postgres[525709]: [525709] ERROR: duplicate key value violates unique constraint "users_nickname_index" Jan 03 11:48:20 perihelion postgres[525709]: [525709] DETAIL: Key (nickname)=(hatch@creacher.club) already exists. Jan 03 11:48:20 perihelion postgres[525709]: [525709] STATEMENT: UPDATE "users" SET "also_known_as" = $1, "avatar" = $2, "banner" = $3, "bio" = $4, "fields" = $5, "follower_address" = $6, "last_refreshed_at" = $7, "name" = $8, "nickname" = $9, "pinned_objects" = $10, "public_key" = $11, "shared_inbox" = $12, "updated_at" = $13 WHERE "id" = $14 Jan 03 11:48:19 perihelion akkoma[38830]: [error] Could not fetch page https://creacher.club/users/hatch/statuses/109620471780924323/replies?min_id=109620676793607866&page=true - {:ok, %Tesla.Env{__client__: %Tesla.Client{adapter: nil, fun: nil, post: [], pre: [{Tesla.Middleware.FollowRedirects, :call, [[]]}]}, __module__: Tesla, body: "{\"error\":\"Too many requests\"}", headers: [{"date", "Tue, 03 Jan 2023 11:48:19 GMT"}, {"content-type", "application/json"}, {"transfer-encoding", "chunked"}, {"connection", "keep-alive"}, {"x-ratelimit-limit", "300"}, {"x-ratelimit-remaining", "0"}, {"x-ratelimit-reset", "2023-01-03T12:00:00.867446Z"}, {"cache-control", "no-cache"}, {"content-security-policy", "base-uri 'none'; default-src 'none'; frame-ancestors 'none'; font-src 'self' https://creacher.club; img-src 'self' https: data: blob: https://creacher.club; style-src 'self' https://creacher.club 'nonce-lp4Jhgr/s0DDf68WcdyiGg=='; media-src 'self' https: data: https://creacher.club; frame-src 'self' https:; manifest-src 'self' https://creacher.club; connect-src 'self' data: blob: https://creacher.club https://creacher.club wss://creacher.club; script-src 'self' https://creacher.club 'wasm-unsafe-eval'; child-src 'self' blob: https://creacher.club; worker-src 'self' blob: https://creacher.club"}, {"x-request-id", "025f84da-616c-4195-b1bd-268c4794d1ce"}, {"x-runtime", "0.001741"}, {"strict-transport-security", "max-age=63072000; includeSubDomains"}], method: :get, opts: [adapter: [receive_timeout: 15000, pool_timeout: 5000, name: MyFinch]], query: [], status: 429, url: "https://creacher.club/users/hatch/statuses/109620471780924323/replies?min_id=109620676793607866&page=true"}} ``` I do not have logs of this starting, since this has produced dozens of lines per second for hours and earlier logs got pruned. ### Severity I cannot use it as easily as I'd like ### Have you searched for this issue? - [x] I have double-checked and have not found this issue mentioned anywhere.