Certain outbound requests seem to queue up #259

New issue

Closed

opened 2022-11-09 00:05:54 +00:00 by nninja · 3 comments

nninja commented

2022-11-09 00:05:54 +00:00

Anecdotally, I've noticed certain requests would block other (unrelated) requests

Examples:

Browsing timeline generates multiple requests to fetch remote users' avatars, which seems to block other unrelated requests, such as Elasticsearch.
Fetching user's banner image has the same effect as (1)
Request to Follow remote user sometimes take a long time - Possible to make this a background task?

Appreciate any pointers to help identify the root cause, and possible tweaks/improvements.

p.s. Instance is running outbound HTTP proxy (tinyproxy), Media proxy, MediaProxyWarmingPolicy and the associated nginx caching as described here: https://docs.akkoma.dev/stable/configuration/storing_remote_media/

Attached the current Job Queue settings for reference:

Anecdotally, I've noticed certain requests would block other (unrelated) requests Examples: 1. Browsing timeline generates multiple requests to fetch remote users' avatars, which seems to block other unrelated requests, such as Elasticsearch. 2. Fetching user's banner image has the same effect as (1) 3. Request to Follow remote user sometimes take a long time - Possible to make this a background task? Appreciate any pointers to help identify the root cause, and possible tweaks/improvements. p.s. Instance is running outbound HTTP proxy (`tinyproxy`), Media proxy, `MediaProxyWarmingPolicy` and the associated nginx caching as described here: https://docs.akkoma.dev/stable/configuration/storing_remote_media/ Attached the current Job Queue settings for reference: ![image](/attachments/15036b18-999c-45dd-9857-8ea06177e79e)

image.png

232 KiB

floatingghost commented

2022-11-25 12:35:31 +00:00

Owner

yeah, HTTP uses a pool setup - it should allow 50 outbound requests at any one time by default

are you consistently hitting this limit?

yeah, HTTP uses a pool setup - it should allow 50 outbound requests at any one time by default are you consistently hitting this limit?

nninja commented

2022-11-28 01:18:44 +00:00

Author

@floatingghost would hitting limit trigger some form of logs?

Oneric commented

2025-08-18 11:41:51 +00:00

Owner

Request to Follow remote user sometimes take a long time - Possible to make this a background task?

iinm it intetionally waits a while after federating the follow request to check if the remote already replied (in case follows are not restricted and automatically approved), so the HTTP response can report back with an accepted/denied instead of just pending status. There might be some config setting to control the wait duration.

would hitting limit trigger some form of logs?

Not sure if Finch logs anything about that on default logging levels (it presumably does with debug), but you can just check Prometheus metrics to see how many outgoing requests your instance generates and the Oban Web dashboard to see if any queue is overloaded and has consistently many jobs in the backlog. If needed, more detailed instructions on those metrics can be found here atm: #966

EDIT: oh wait, just noticed this was from two years ago, is this even still an issue?

EDIT 2: you can now also track the amount of queued up jobs in Prometheus stats; see: monitoring docs

> Request to Follow remote user sometimes take a long time - Possible to make this a background task? iinm it _intetionally_ waits a while after federating the follow request to check if the remote already replied *(in case follows are not restricted and automatically approved)*, so the HTTP response can report back with an `accepted`/`denied` instead of just `pending` status. There might be some config setting to control the wait duration. > would hitting limit trigger some form of logs? Not sure if Finch logs anything about that on default logging levels (it presumably does with `debug`), but you can just check Prometheus metrics to see how many outgoing requests your instance generates and the Oban Web dashboard to see if any queue is overloaded and has consistently many jobs in the backlog. If needed, more detailed instructions on those metrics can be found here atm: https://akkoma.dev/AkkomaGang/akkoma/pulls/966 **EDIT**: oh wait, just noticed this was from two years ago, is this even still an issue? **EDIT 2:** you can now also track the amount of queued up jobs in Prometheus stats; see: [monitoring docs](https://docs.akkoma.dev/develop/administration/monitoring/#prometheus)