Replace the database queue system with an in-memory solution #346

New issue

Open

opened 2022-12-06 12:28:31 +00:00 by fristi · 2 comments

fristi commented

2022-12-06 12:28:31 +00:00

Contributor

Personally I have never understood why this system was implemented in the first place, considering database queues are notable for being slow and unreliable, generally not suited for production environments, especially if scaling is required.

PS: I'm mostly posting this one as a mental note, I know this is a big one and that I cannot help with it myself. This is mainly so that there is a place to discuss this.

The fact that pleroma uses a db queue has seriously irked me ever since it was introduced in its 2.0 release. It seriously increased disk IO due to the constant read/write operations done when running background tasks, which is especially apparent if you happen to run the database on a hard disk drive. In that case it can literally turn your server into a ticking clock. Personally I have never understood why this system was implemented in the first place, considering database queues are notable for being slow and unreliable, generally not suited for production environments, especially if scaling is required. --- PS: I'm mostly posting this one as a mental note, I know this is a big one and that I cannot help with it myself. This is mainly so that there is a place to discuss this.

alyssa commented

2022-12-12 01:07:11 +00:00

would be great to have a redis queue, so a server admin can still configure persistence without needing the elixir process to handle it directly

floatingghost commented

2022-12-15 01:06:57 +00:00

Owner

proof of concept @ https://akkoma.dev/AkkomaGang/akkoma/src/branch/rabbit

uses rabbitmq

seeing 50% IO savings on an admittedly small test run

reliability isn't as good and it's buggy
but it works

proof of concept @ https://akkoma.dev/AkkomaGang/akkoma/src/branch/rabbit uses rabbitmq seeing 50% IO savings on an admittedly small test run reliability isn't as good and it's buggy but it works

No milestone

No project

No assignees

3 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: AkkomaGang/akkoma#346

No description provided.