Backoff on HTTP requests when 429 is recieved #762

Merged

floatingghost merged 9 commits from backoff-http into develop 2024-05-11 04:38:48 +00:00

Conversation 2 Commits 9 Files changed 6 +233 -4

floatingghost commented 2024-04-26 18:05:41 +00:00

Owner

Copy link

Fixes #269

Fixes #269

floatingghost added 3 commits 2024-04-26 18:05:41 +00:00

Add HTTP backoff cache to respect 429s 666d6238b3

add test for backoff cc1efd2182

Add ratelimit backoff to HTTP get 14ff522b67

floatingghost changed title from backoff-http to Backoff on HTTP requests when 429 is recieved 2024-04-26 18:05:56 +00:00

floatingghost force-pushed backoff-http from 14ff522b67 to f531484063 2024-04-26 18:06:36 +00:00 Compare

floatingghost added 1 commit 2024-04-26 18:10:26 +00:00

changelog entry 9671cdecdf

floatingghost added 1 commit 2024-04-26 18:28:11 +00:00

where were you when lint fail 010e8c7bb2

Oneric commented 2024-04-26 19:38:10 +00:00

Member

Copy link

A suggestion for the backoff-http branch, it currently handles only 429 and Mastodon’s custom X-Ratelimit-Reset header.

GoToSocial uses the same for too many requests from a single server, but also a 503 response and the more standard Retry-After header if the server is just generally overloaded. We should backoff based on this too

MDN documents Retry-After to either contain a timestamp or a offset in seconds and that it may be used for both 503 and 429. It’s probably ok to also check X-Ratelimit-Reset for 503 and an offset for X-Ratelimit-Reset, i.e. something like this: 96d3f3b4fa (Though this changes the log message on an unparsable header and ig it’s also debateable whether 429 and 503 should use the same log message. Converting a second offset to a date and then back to an offset seems a bit redundant but i didn't wnat to change logs etc further)

> A suggestion for the backoff-http branch, it currently handles only 429 and Mastodon’s custom `X-Ratelimit-Reset` header. > > [GoToSocial uses](https://docs.gotosocial.org/en/latest/federation/federating_with_gotosocial/#request-throttling-rate-limiting) the same for too many requests from a single server, but also a [`503` response](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/503) and the more standard `Retry-After` header if the server is just generally overloaded. We should backoff based on this too MDN documents `Retry-After` to either contain a timestamp or a offset in seconds and that it may be used for both 503 and 429. It’s probably ok to also check `X-Ratelimit-Reset` for 503 and an offset for `X-Ratelimit-Reset`, i.e. something like this: https://akkoma.dev/Oneric/akkoma/commit/96d3f3b4fa668c94e31256d6201ecb9b065e7394 *(Though this changes the log message on an unparsable header and ig it’s also debateable whether 429 and 503 should use the same log message. Converting a second offset to a date and then back to an offset seems a bit redundant but i didn't wnat to change logs etc further)*

floatingghost added 1 commit 2024-05-06 22:35:01 +00:00

additionally support retry-after values bd74693db6

floatingghost added 1 commit 2024-05-06 22:36:15 +00:00

add a test for 503-rate-limiting ea6bc8a7c5

floatingghost commented 2024-05-06 22:36:57 +00:00

Author

Owner

Copy link

added support for Retry-After, and checks for 503 as well (we can internally treat it as a ratelimit, because it's probably having troubles anyhow)

added support for `Retry-After`, and checks for 503 as well (we can internally treat it as a ratelimit, because it's probably having troubles anyhow)

❤️ 1

floatingghost merged commit 76ded10a70 into develop 2024-05-11 04:38:48 +00:00

floatingghost deleted branch backoff-http 2024-05-11 04:38:48 +00:00

floatingghost referenced this pull request from a commit 2024-05-11 04:38:49 +00:00

Merge pull request 'Backoff on HTTP requests when 429 is recieved' (#762) from backoff-http into develop

Oneric referenced this pull request 2024-05-25 14:43:09 +00:00

meta: closeable issues #758

norm referenced this pull request 2024-05-31 06:51:44 +00:00

[bug] Akkoma getting OOM killed #736

Mynacol referenced this pull request 2024-07-19 14:14:30 +00:00

[bug] Akkoma getting OOM killed #736

Oneric referenced this pull request 2024-11-14 23:38:53 +00:00

WIP: Add backfilling of posts #846

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

approved, awaiting change

  

bug

Something is not working

  

configuration

This requires config changes

  

documentation

This is about human-readable docs

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

extremely low priority

  

feature request

Something new?

  

Fix it yourself

Support has been attempted

  

help wanted

Need some help

  

invalid

Something is wrong

  

mastodon_api

  

needs docs

  

needs tests

  

not a bug

  

planned

  

pleroma_api

  

privacy

  

question

More information is needed

  

static_fe

  

triage

  

wontfix

This won't be fixed

No labels

approved, awaiting change

bug

configuration

documentation

duplicate

enhancement

extremely low priority

feature request

Fix it yourself

help wanted

invalid

mastodon_api

needs docs

needs tests

not a bug

planned

pleroma_api

privacy

question

static_fe

triage

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: AkkomaGang/akkoma#762

No description provided.