deleting a note locks up the server

dwarf commented

2022-10-31 14:29:09 +00:00

Contributor

I see that #198 implemented notifying all known instances about removed notes. It seems this also works for remote notes, which I'm not sure is intended, but as mentioned in the ticket, it locks up the server considerably making it unavailable to the users. I currently run an instance for a mere two people so the load is otherwise fine. Worth noting is that I've configured a relay server, so the server knows a lot of servers.

I've attached some screenshots for context.

I see that #198 implemented notifying all known instances about removed notes. It seems this also works for remote notes, which I'm not sure is intended, but as mentioned in the ticket, it locks up the server considerably making it unavailable to the users. I currently run an instance for a mere two people so the load is otherwise fine. Worth noting is that I've configured a relay server, so the server knows a lot of servers. I've attached some screenshots for context.

Screenshot_20221031_150952_1.png

56 KiB

Screenshot_20221031_150852_1.png

114 KiB

Johann150 commented

2022-10-31 15:07:43 +00:00

Owner

If another instance delivers their delete instance to us, that delete activity will not be forwarded. What you may be seeing is that our database schema is unable to store notes without storing their replies. If a remote user deletes a note, local replies/renotes will be cascade-deleted. These cascaded activities will then be delivered.

There was already 91a4f38871 and #215 to try to mitigate the negative performance effects. I think in this special case of cascading we could optimize by not delivering each cascaded delete in a separate HTTP request (if there are multiple notes being deleted).

You may also want to take a look at deliverJobPerSec in the configuration file and lowering it, it should lower the impact of delivering a large number of activities at once, the delivery should be a bit more spread out.

If another instance delivers their delete instance to us, that delete activity will not be forwarded. What you may be seeing is that our database schema is unable to store notes without storing their replies. If a remote user deletes a note, local replies/renotes will be cascade-deleted. These cascaded activities will then be delivered. There was already 91a4f38871e4f39fc30e943fa913dd9ab43cc600 and #215 to try to mitigate the negative performance effects. I think in this special case of cascading we could optimize by not delivering each cascaded delete in a separate HTTP request (if there are multiple notes being deleted). You may also want to take a look at `deliverJobPerSec` in the configuration file and lowering it, it should lower the impact of delivering a large number of activities at once, the delivery should be a bit more spread out.

dwarf commented

2022-10-31 17:32:13 +00:00

Author

Contributor

I think part of it is also the SQL queries it does, it does at least two per instance from what I've seen, ideally that'd be optimized somehow. It takes between 300-500msec per SQL query on my instance (which is odd, it all runs from a nvme SSD connected using USB3, so that shouldn't be the bottleneck).

toast commented

2022-12-03 13:17:39 +00:00

Owner

@dwarf this should be greatly improved now with the many optimizations we made (and the separate worker threads).
I'm closing this, but do feel free to ask for a reopen if you still experience issues.

@dwarf this should be greatly improved now with the many optimizations we made (and the separate worker threads). I'm closing this, but do feel free to ask for a reopen if you still experience issues.

toast closed this issue

2022-12-03 13:17:41 +00:00

deleting a note locks up the server #220