Well, I have a backup, but I can't restore it when the database index is corrupt
So I figured out that the whole database got corrupted. Rebuilding the index resulted in the whole database being yeeted.
koyu@koyu:~$ dd if=/dev/urandom of=test.img bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.6276 s, 191 MB/s
koyu@koyu:~$…please benchmark your disk to ensure it has the read and write speeds necessary to comfortably run a database
How do I do that? I mean Pleroma and bloaty Mastodon worked before.
I also tried regenerating the entire config. That didn't help.
This is very interesting. I'm occasionally getting Oct 06 19:28:24 koyu.space mix[622149]: 19:28:24.173 [notice] :alarm_handler: {:clear, :system_memory_high_watermark} even though I still…
Now I'm getting these funny things with a few requests:
Oct 06 19:24:47 koyu.space mix[622149]: 19:24:47.607 request_id=FxuRVH9dchMu0pwAAG9D [error] Internal server error: {:timeout,…Running a vacuum didn't improve performance though. I had high hopes.
I might have found the issue. Running a vacuum using mix takes a long time to finish and has a very high IO load.
pg_repack returned ERROR: query failed: ERROR: could not create unique index "index_371024" and the server is still slow
I did an SQL vacuum and tried debug logging, but there's nothing of value. That's tough.
iotop is safe so I don't think it's disk IO. How do I enable debug logging?
check your running queries again - is there a specific one that's taking a long time?
This one has been trying to do something since start:
DO $$BEGIN LISTEN "public.oban_insert"; END$$
I'm also getting these from time to time
Oct 05 18:23:32 koyu.space mix[20932]: 18:23:32.108 [notice] Application ex_aws exited: :stopped
Oct 05 18:23:32 koyu.space mix[20932]: 18:23:32.10…Removing masto-fe related settings from the database made it kinda faster 🤔
Yes, it seems to have synced up, but timelines still load slow
What I find ironic is that the timelines load super slow, but the rest like config etc. loads as it should
It's getting better, but it's still very slow. Some requests even don't get throguh now. I'm running koyu.space on a KVM VPS with the following specs:
10GB RAM 4 Cores (AMD EPYC) 200GB SSD