Backoff on HTTP requests when 429 is recieved #762

Merged
floatingghost merged 9 commits from backoff-http into develop 2024-05-11 04:38:48 +00:00

Fixes #269

Fixes #269
floatingghost added 3 commits 2024-04-26 18:05:41 +00:00
ci/woodpecker/push/build-amd64 Pipeline is pending Details
ci/woodpecker/push/build-arm64 Pipeline is pending Details
ci/woodpecker/push/docs Pipeline is pending Details
ci/woodpecker/push/lint Pipeline is pending Details
ci/woodpecker/push/test Pipeline is pending Details
cc1efd2182
add test for backoff
ci/woodpecker/push/build-amd64 Pipeline is pending Details
ci/woodpecker/push/build-arm64 Pipeline is pending Details
ci/woodpecker/push/docs Pipeline is pending Details
ci/woodpecker/push/lint Pipeline is pending Details
ci/woodpecker/push/test Pipeline is pending Details
ci/woodpecker/pr/build-amd64 Pipeline is pending Details
ci/woodpecker/pr/build-arm64 Pipeline is pending Details
ci/woodpecker/pr/docs Pipeline is pending Details
ci/woodpecker/pr/lint Pipeline is pending Details
ci/woodpecker/pr/test Pipeline is pending Details
14ff522b67
Add ratelimit backoff to HTTP get
floatingghost changed title from backoff-http to Backoff on HTTP requests when 429 is recieved 2024-04-26 18:05:56 +00:00
floatingghost force-pushed backoff-http from 14ff522b67 to f531484063 2024-04-26 18:06:36 +00:00 Compare
floatingghost added 1 commit 2024-04-26 18:10:26 +00:00
ci/woodpecker/push/build-amd64 Pipeline is pending Details
ci/woodpecker/push/build-arm64 Pipeline is pending Details
ci/woodpecker/push/docs Pipeline is pending Details
ci/woodpecker/push/lint Pipeline is pending Details
ci/woodpecker/push/test Pipeline is pending Details
ci/woodpecker/pr/lint Pipeline failed Details
ci/woodpecker/pr/test unknown status Details
ci/woodpecker/pr/build-arm64 unknown status Details
ci/woodpecker/pr/build-amd64 unknown status Details
ci/woodpecker/pr/docs unknown status Details
9671cdecdf
changelog entry
floatingghost added 1 commit 2024-04-26 18:28:11 +00:00
ci/woodpecker/push/build-amd64 Pipeline is pending Details
ci/woodpecker/push/build-arm64 Pipeline is pending Details
ci/woodpecker/push/docs Pipeline is pending Details
ci/woodpecker/push/lint Pipeline is pending Details
ci/woodpecker/push/test Pipeline is pending Details
ci/woodpecker/pr/lint Pipeline was successful Details
ci/woodpecker/pr/test Pipeline was successful Details
ci/woodpecker/pr/build-arm64 unknown status Details
ci/woodpecker/pr/build-amd64 unknown status Details
ci/woodpecker/pr/docs unknown status Details
010e8c7bb2
where were you when lint fail
Member

A suggestion for the backoff-http branch, it currently handles only 429 and Mastodon’s custom X-Ratelimit-Reset header.

GoToSocial uses the same for too many requests from a single server, but also a 503 response and the more standard Retry-After header if the server is just generally overloaded. We should backoff based on this too

MDN documents Retry-After to either contain a timestamp or a offset in seconds and that it may be used for both 503 and 429. It’s probably ok to also check X-Ratelimit-Reset for 503 and an offset for X-Ratelimit-Reset, i.e. something like this: 96d3f3b4fa (Though this changes the log message on an unparsable header and ig it’s also debateable whether 429 and 503 should use the same log message. Converting a second offset to a date and then back to an offset seems a bit redundant but i didn't wnat to change logs etc further)

> A suggestion for the backoff-http branch, it currently handles only 429 and Mastodon’s custom `X-Ratelimit-Reset` header. > > [GoToSocial uses](https://docs.gotosocial.org/en/latest/federation/federating_with_gotosocial/#request-throttling-rate-limiting) the same for too many requests from a single server, but also a [`503` response](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/503) and the more standard `Retry-After` header if the server is just generally overloaded. We should backoff based on this too MDN documents `Retry-After` to either contain a timestamp or a offset in seconds and that it may be used for both 503 and 429. It’s probably ok to also check `X-Ratelimit-Reset` for 503 and an offset for `X-Ratelimit-Reset`, i.e. something like this: https://akkoma.dev/Oneric/akkoma/commit/96d3f3b4fa668c94e31256d6201ecb9b065e7394 *(Though this changes the log message on an unparsable header and ig it’s also debateable whether 429 and 503 should use the same log message. Converting a second offset to a date and then back to an offset seems a bit redundant but i didn't wnat to change logs etc further)*
floatingghost added 1 commit 2024-05-06 22:35:01 +00:00
ci/woodpecker/push/build-amd64 Pipeline is pending Details
ci/woodpecker/push/build-arm64 Pipeline is pending Details
ci/woodpecker/push/docs Pipeline is pending Details
ci/woodpecker/push/lint Pipeline is pending Details
ci/woodpecker/push/test Pipeline is pending Details
ci/woodpecker/pr/build-amd64 Pipeline is pending Details
ci/woodpecker/pr/build-arm64 Pipeline is pending Details
ci/woodpecker/pr/docs Pipeline is pending Details
ci/woodpecker/pr/lint Pipeline is pending Details
ci/woodpecker/pr/test Pipeline is pending Details
bd74693db6
additionally support retry-after values
floatingghost added 1 commit 2024-05-06 22:36:15 +00:00
ci/woodpecker/push/build-amd64 Pipeline is pending Details
ci/woodpecker/push/build-arm64 Pipeline is pending Details
ci/woodpecker/push/docs Pipeline is pending Details
ci/woodpecker/push/lint Pipeline is pending Details
ci/woodpecker/push/test Pipeline is pending Details
ci/woodpecker/pr/lint Pipeline was successful Details
ci/woodpecker/pr/test Pipeline was successful Details
ci/woodpecker/pr/build-amd64 Pipeline was successful Details
ci/woodpecker/pr/build-arm64 Pipeline was successful Details
ci/woodpecker/pr/docs Pipeline was successful Details
ea6bc8a7c5
add a test for 503-rate-limiting
Author
Owner

added support for Retry-After, and checks for 503 as well (we can internally treat it as a ratelimit, because it's probably having troubles anyhow)

added support for `Retry-After`, and checks for 503 as well (we can internally treat it as a ratelimit, because it's probably having troubles anyhow)
floatingghost merged commit 76ded10a70 into develop 2024-05-11 04:38:48 +00:00
floatingghost deleted branch backoff-http 2024-05-11 04:38:48 +00:00
Sign in to join this conversation.
No description provided.