Massive Performance impact when delivery Queue starts working

puniko commented

2022-11-24 10:43:00 +00:00

Contributor

Every time, when a bunch of jobs have piled up as delayed and the delivery queue starts rerunning them, the frontend gets very unresponsive, to a point where sending a post takes minutes.

i upped the worked count, i lowered the worker count, but regardless of what i do, it keeps getting unresponsive. even now, where foundkey only manages to use 40 - 50 % of cpu for the worker stuff, it still looks up depsite there being enough resources available to serve frontend normally.

memory, disk io, network and stuff all seem fine and there is plenty for foundkey to use, but it seems to refuse using it.

Sooooo, i'm out of ideas. idk how to fix this, idk how to work around this, let alone why this happens.

Every time, when a bunch of jobs have piled up as delayed and the delivery queue starts rerunning them, the frontend gets very unresponsive, to a point where sending a post takes minutes. i upped the worked count, i lowered the worker count, but regardless of what i do, it keeps getting unresponsive. even now, where foundkey only manages to use 40 - 50 % of cpu for the worker stuff, it still looks up depsite there being enough resources available to serve frontend normally. memory, disk io, network and stuff all seem fine and there is plenty for foundkey to use, but it seems to refuse using it. Sooooo, i'm out of ideas. idk how to fix this, idk how to work around this, let alone why this happens.

Johann150 added a new dependency 2022-11-25 12:03:17 +00:00

#252 implement separate web workers

Johann150 commented

2022-11-25 12:14:29 +00:00

Owner

I separated web and queue workers in #252 but I can't check if it really helps. If this doesn't do the trick the queue workers could be re-niced even more from PRIORITY_BELOW_NORMAL to PRIORITY_LOW.

I separated web and queue workers in #252 but I can't check if it really helps. If this doesn't do the trick the queue workers could be re-niced even more from `PRIORITY_BELOW_NORMAL` to `PRIORITY_LOW`.

puniko commented

2022-11-25 13:39:06 +00:00

Author

Contributor

Thanks, will merge it in, tomorrow or on sunday maybe and let you know

puniko commented

2022-11-28 07:10:01 +00:00

Author

Contributor

Just for completness. merged it yesterday, seems to run well so far. will keep it for a week without the charts stuff, than enable charts again to see how it runs with them

puniko commented

2022-12-03 12:07:16 +00:00

Author

Contributor

can confirm that the split works well with on my instance. now lockdowns of fe anymore even when queue is doing its thing.
i still have to disable charts tho, but thats another issue. at least it also doesn't block web, when gathering stuff for the charts (but it does stop the queue while it does so)

can confirm that the split works well with on my instance. now lockdowns of fe anymore even when queue is doing its thing. i still have to disable charts tho, but thats another issue. at least it also doesn't block web, when gathering stuff for the charts (but it does stop the queue while it does so)

👍 1

toast commented

2022-12-03 13:21:44 +00:00

Owner

Can you make a new issue for charts?
In the meanwhile closing this one since we got there 🎉

Can you make a new issue for charts? In the meanwhile closing this one since we got there 🎉

Johann150 commented

2022-12-03 13:29:59 +00:00

Owner

@toast #252 is not merged yet because there is an outstanding review comment by you. After that is merged this can be closed.

Chart problems are already tracked in #237 and #253.

@toast #252 is not merged yet because there is an outstanding review comment by you. After that is merged this can be closed. Chart problems are already tracked in #237 and #253.

👍 1

Johann150 closed this issue

2022-12-03 13:33:47 +00:00

Massive Performance impact when delivery Queue starts working #249