Possible higher memory usage due to timeouts #331

New Issue

Open

opened 2022-12-03 10:25:45 +00:00 by pimeys · 1 comment

pimeys commented

2022-12-03 10:25:45 +00:00

Dec 03 09:01:03 social pleroma[63558]: 09:01:03.605 request_id=Fy089F99bUmiKfoAHuhB [error] Could not decode user at fetch https://0lb.be/actor, "timeout"
Dec 03 09:01:03 social pleroma[63558]: 09:01:03.616 request_id=Fy089F4p7nz3Q48AHufB [error] Could not decode user at fetch https://social.hackerspace.pl/actor, "timeout"
Dec 03 09:01:03 social pleroma[63558]: 09:01:03.640 request_id=Fy089GIqqRdVWlsAHHQS [error] Could not decode user at fetch https://beach.city/actor, "timeout"
Dec 03 09:01:03 social pleroma[63558]: 09:01:03.718 request_id=Fy089F9H3ftuPlYAHugh [error] Could not decode user at fetch https://mutual.tls.zone/actor, "timeout"
Dec 03 09:01:03 social pleroma[63558]: 09:01:03.722 request_id=Fy089Frp2dRIk4AAHudR [error] Could not decode user at fetch https://social.furry.nz/actor, "timeout"
Dec 03 09:01:03 social pleroma[63558]: 09:01:03.736 request_id=Fy089GDBuOEdeG4AHHOS [error] Could not decode user at fetch https://glauca.space/actor, "timeout"
Dec 03 09:01:03 social pleroma[63558]: 09:01:03.738 request_id=Fy089GTt-dxQ--0AHHSS [error] Could not decode user at fetch https://mastodon.indie.host/actor, "timeout"
Dec 03 09:01:03 social pleroma[63558]: 09:01:03.742 request_id=Fy089HPgpQz8OQcAHukh [error] Could not decode user at fetch https://social.emi.cx/actor, "timeout"
Dec 03 09:01:03 social pleroma[63558]: 09:01:03.748 request_id=Fy089HC_mWbFgEEAHHZi [error] Could not decode user at fetch https://mythago.space/actor, "timeout"
Dec 03 09:01:03 social pleroma[63558]: 09:01:03.752 request_id=Fy089HePx3_pEqcAHHdi [error] Could not decode user at fetch https://freeradical.zone/actor, "timeout"
Dec 03 09:01:03 social pleroma[63558]: 09:01:03.762 request_id=Fy089HUL_iy7Bi0AHulR [error] Could not decode user at fetch https://stop.voring.me/users/8vr53c3zja, "timeout"
Dec 03 09:01:03 social pleroma[63558]: 09:01:03.805 request_id=Fy089IPWe0i8nEIAHHly [error] Could not decode user at fetch https://trans.house/actor, "timeout"
lines 940-1000

I think we had some request queued up in the memory, timing out eventually. I do not know why the timeouts happened, but if similar thing would happen with a larger server, this could mean a crash due OOM.

Saw this in my server monitoring today. Typically memory usage is quite flat, but here we have some nice bumps. Luckily it didn't swap or OOM, but I checked from the logs and they're filled with messages such as: ``` Dec 03 09:01:03 social pleroma[63558]: 09:01:03.605 request_id=Fy089F99bUmiKfoAHuhB [error] Could not decode user at fetch https://0lb.be/actor, "timeout" Dec 03 09:01:03 social pleroma[63558]: 09:01:03.616 request_id=Fy089F4p7nz3Q48AHufB [error] Could not decode user at fetch https://social.hackerspace.pl/actor, "timeout" Dec 03 09:01:03 social pleroma[63558]: 09:01:03.640 request_id=Fy089GIqqRdVWlsAHHQS [error] Could not decode user at fetch https://beach.city/actor, "timeout" Dec 03 09:01:03 social pleroma[63558]: 09:01:03.718 request_id=Fy089F9H3ftuPlYAHugh [error] Could not decode user at fetch https://mutual.tls.zone/actor, "timeout" Dec 03 09:01:03 social pleroma[63558]: 09:01:03.722 request_id=Fy089Frp2dRIk4AAHudR [error] Could not decode user at fetch https://social.furry.nz/actor, "timeout" Dec 03 09:01:03 social pleroma[63558]: 09:01:03.736 request_id=Fy089GDBuOEdeG4AHHOS [error] Could not decode user at fetch https://glauca.space/actor, "timeout" Dec 03 09:01:03 social pleroma[63558]: 09:01:03.738 request_id=Fy089GTt-dxQ--0AHHSS [error] Could not decode user at fetch https://mastodon.indie.host/actor, "timeout" Dec 03 09:01:03 social pleroma[63558]: 09:01:03.742 request_id=Fy089HPgpQz8OQcAHukh [error] Could not decode user at fetch https://social.emi.cx/actor, "timeout" Dec 03 09:01:03 social pleroma[63558]: 09:01:03.748 request_id=Fy089HC_mWbFgEEAHHZi [error] Could not decode user at fetch https://mythago.space/actor, "timeout" Dec 03 09:01:03 social pleroma[63558]: 09:01:03.752 request_id=Fy089HePx3_pEqcAHHdi [error] Could not decode user at fetch https://freeradical.zone/actor, "timeout" Dec 03 09:01:03 social pleroma[63558]: 09:01:03.762 request_id=Fy089HUL_iy7Bi0AHulR [error] Could not decode user at fetch https://stop.voring.me/users/8vr53c3zja, "timeout" Dec 03 09:01:03 social pleroma[63558]: 09:01:03.805 request_id=Fy089IPWe0i8nEIAHHly [error] Could not decode user at fetch https://trans.house/actor, "timeout" lines 940-1000 ``` I think we had some request queued up in the memory, timing out eventually. I do not know why the timeouts happened, but if similar thing would happen with a larger server, this could mean a crash due OOM.

Screenshot 2022-12-03 at 11-23-44 Node Exporter Full - Dashboards - Dashboards - Grafana.png

35 KiB