A lot of errors when migrating from Pleroma

bubblineyuri commented

2022-09-16 11:29:25 +00:00

Hi,

I get the following PostgreSQL error when trying to migrate from Pleroma to Akkoma

column u1.mastofe_settings does not exist

I hope you can help me with that.

~Leonie

Hi, I get the following PostgreSQL error when trying to migrate from Pleroma to Akkoma ``` column u1.mastofe_settings does not exist ``` I hope you can help me with that. ~Leonie

floatingghost commented

2022-09-16 11:55:16 +00:00

Owner

Ah, this could occur if migrating from develop pleroma where you'd already applied the removal of mastofe patch

Should be fixed via #216

if you're from source, git pull and try again

if you're OTP, wait for https://ci.akkoma.dev/AkkomaGang/akkoma/build/930 to complete then update and try again

Ah, this could occur if migrating from develop pleroma where you'd already applied the removal of mastofe patch Should be fixed via https://akkoma.dev/AkkomaGang/akkoma/pulls/216 if you're from source, git pull and try again if you're OTP, wait for https://ci.akkoma.dev/AkkomaGang/akkoma/build/930 to complete then update and try again

bubblineyuri commented

2022-09-16 13:18:42 +00:00

Author

This didn't fix anything, now I get these fun messages http://content.koyu.space/1MPAN

floatingghost commented

2022-09-16 13:21:36 +00:00

Owner

ok so it actually did fix your error since these are totally different

sounds like you've got your pool target set really low - it should never be that low by default

you'll want to increase it by changing queue_target like so:

config :pleroma, Pleroma.Repo,
  adapter: Ecto.Adapters.Postgres,
  ... creds are probably here...
  timeout: 90_000,
  queue_target: 20_000, # THIS ONE
  queue_interval: 2_000,
  pool_size: 10

ok so it actually _did_ fix your error since these are totally different sounds like you've got your pool target set really low - it should never be that low by default you'll want to increase it by changing `queue_target` like so: ```elixir config :pleroma, Pleroma.Repo, adapter: Ecto.Adapters.Postgres, ... creds are probably here... timeout: 90_000, queue_target: 20_000, # THIS ONE queue_interval: 2_000, pool_size: 10 ```

bubblineyuri commented

2022-10-05 11:38:19 +00:00

Author

Getting this beauty here

Oct 05 11:35:12 koyu.space mix[2454618]: 11:35:12.038 [info] Postgrex.Protocol (#PID<0.5170.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.6331.0> exited

Getting this beauty here `Oct 05 11:35:12 koyu.space mix[2454618]: 11:35:12.038 [info] Postgrex.Protocol (#PID<0.5170.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.6331.0> exited`

floatingghost commented

2022-10-05 13:15:07 +00:00

Owner

that alone doesn't give an awful lot to go on, there should be more logs above it that indicate what used the exit

bubblineyuri commented

2022-10-05 13:23:34 +00:00

Author

Is this a little better? http://content.koyu.space/A6tGK

floatingghost commented

2022-10-05 13:24:53 +00:00

Owner

hm, that doesn't look fatal - does the instance die or does it recover after doing that?

bubblineyuri commented

2022-10-05 13:33:06 +00:00

Author

It loads the UI and responds to requests very slowly

floatingghost commented

2022-10-05 13:43:48 +00:00

Owner

hm, that sounds like you might have too big of a database for your system

have you done a vacuum/pg_repack to remove stuff?

additionally, how long is your remote object retention? that may be causing object table inflation

hm, that sounds like you might have too big of a database for your system have you done a vacuum/pg_repack to remove stuff? additionally, how long is your remote object retention? that may be causing object table inflation

bubblineyuri commented

2022-10-05 13:59:09 +00:00

Author

I have no idea what those two are. I also store configuration in the database, so how do I check that if I've blown off half of the server?

floatingghost commented

2022-10-05 14:01:54 +00:00

Owner

vacuum - https://docs.akkoma.dev/stable/administration/CLI_tasks/database/#prune-old-remote-posts-from-the-database

pg_repack - https://github.com/reorg/pg_repack

show options in the database - https://docs.akkoma.dev/stable/administration/CLI_tasks/config/#dump-all-of-the-config-settings-defined-in-the-database

vacuum - https://docs.akkoma.dev/stable/administration/CLI_tasks/database/#prune-old-remote-posts-from-the-database pg_repack - https://github.com/reorg/pg_repack show options in the database - https://docs.akkoma.dev/stable/administration/CLI_tasks/config/#dump-all-of-the-config-settings-defined-in-the-database

bubblineyuri commented

2022-10-05 14:07:54 +00:00

Author

There is no object retention configuration. Should I try configuring it or assume it has some sort of default?

Doing a pg_repack does ERROR: pg_repack failed with error: pg_repack 1.4.7 is not installed in the database

There is no object retention configuration. Should I try configuring it or assume it has some sort of default? Doing a pg_repack does `ERROR: pg_repack failed with error: pg_repack 1.4.7 is not installed in the database`

floatingghost commented

2022-10-05 14:08:55 +00:00

Owner

it'll have a default then, you should just be able to run the prune command from the docs.akkoma and it'll remove stuff

bubblineyuri commented

2022-10-05 14:11:06 +00:00

Author

This is getting more interesting now after pruning

Oct 05 14:10:26 koyu.space mix[2497179]: 14:10:26.757 [error] Error while fetching https://fedi.absturztau.be/objects/b51ef47d-46bf-420d-b3d2-4ac22a1126d1: {:error, {:transmogrifier, {:error, {:validate, {:error, #Ecto.Changeset<action: :insert, changes: %{actor: "https://fedi.absturztau.be/users/khaosgrille", cc: ["https://www.w3.org/ns/activitystreams#Public"], context: "https://brotka.st/contexts/bc241591-d030-4d42-8868-4948f437cc86", object: "https://fedi.absturztau.be/objects/b51ef47d-46bf-420d-b3d2-4ac22a1126d1", to: ["https://brotka.st/users/kaia", "https://fedi.absturztau.be/users/khaosgrille/followers", "https://freespeechextremist.com/users/xue"], type: "Create"}, errors: [object: {"The object to create already exists", []}], data: #Pleroma.Web.ActivityPub.ObjectValidators.CreateGenericValidator<>, valid?: false>}}}}}
Oct 05 14:10:26 koyu.space mix[2497179]: 14:10:26.757 [warning] Couldn't fetch "https://fedi.absturztau.be/objects/b51ef47d-46bf-420d-b3d2-4ac22a1126d1", error: nil

This is getting more interesting now after pruning ``` Oct 05 14:10:26 koyu.space mix[2497179]: 14:10:26.757 [error] Error while fetching https://fedi.absturztau.be/objects/b51ef47d-46bf-420d-b3d2-4ac22a1126d1: {:error, {:transmogrifier, {:error, {:validate, {:error, #Ecto.Changeset<action: :insert, changes: %{actor: "https://fedi.absturztau.be/users/khaosgrille", cc: ["https://www.w3.org/ns/activitystreams#Public"], context: "https://brotka.st/contexts/bc241591-d030-4d42-8868-4948f437cc86", object: "https://fedi.absturztau.be/objects/b51ef47d-46bf-420d-b3d2-4ac22a1126d1", to: ["https://brotka.st/users/kaia", "https://fedi.absturztau.be/users/khaosgrille/followers", "https://freespeechextremist.com/users/xue"], type: "Create"}, errors: [object: {"The object to create already exists", []}], data: #Pleroma.Web.ActivityPub.ObjectValidators.CreateGenericValidator<>, valid?: false>}}}}} Oct 05 14:10:26 koyu.space mix[2497179]: 14:10:26.757 [warning] Couldn't fetch "https://fedi.absturztau.be/objects/b51ef47d-46bf-420d-b3d2-4ac22a1126d1", error: nil ```

floatingghost commented

2022-10-05 14:14:19 +00:00

Owner

ok, that's fine, that error isn't fatal at all, just a refetch
you can safely ignore it

ok, that's fine, that error isn't fatal at all, just a refetch you can safely ignore it

bubblineyuri commented

2022-10-05 14:15:57 +00:00

Author

How about this? http://content.koyu.space/XIXb4

The server is still super slow

How about this? http://content.koyu.space/XIXb4 The server is still super slow

floatingghost commented

2022-10-05 14:21:32 +00:00

Owner

that's an interesting one, seems it doesn't like some of your config

that sounds like you may have an outdated schema in your config

can you add

config :pleroma, :instance,
  staff_transparency: []

in your config?

if you run from source, does your config/config.exs match the one in our version control?

that's an interesting one, seems it doesn't like some of your config that sounds like you may have an outdated schema in your config can you add ``` config :pleroma, :instance, staff_transparency: [] ``` in your config? if you run from source, does your `config/config.exs` match the one in our version control?

bubblineyuri commented

2022-10-05 14:27:28 +00:00

Author

Now it tries to eat itself http://content.koyu.space/Mj6z8

Copied config from version control and added the config flag you mentioned

Now it tries to eat itself http://content.koyu.space/Mj6z8 Copied config from version control and added the config flag you mentioned

floatingghost commented

2022-10-05 14:33:24 +00:00

Owner

nice! that means we're past the worst of it, we've got standard inbound requests and cron activating

looks like you might have some long-running requests

when the server is up and the timeouts are occuring, run SELECT * FROM pg_stat_activity; on your db and see if it throws anything interesting

nice! that means we're past the worst of it, we've got standard inbound requests and cron activating looks like you might have some long-running requests when the server is up and the timeouts are occuring, run `SELECT * FROM pg_stat_activity;` on your db and see if it throws anything interesting

bubblineyuri commented

2022-10-05 14:40:55 +00:00

Author

These are the first 20 seconds or so and it's clogging up with queries. This looks awful I guess.

These are the first 20 seconds or so and it's clogging up with queries. This looks awful I guess. ![grafik](/attachments/af8d47c4-8ac7-41a1-9180-1a5d67e813ee)

grafik.png

20 KiB

floatingghost commented

2022-10-05 14:41:43 +00:00

Owner

yeah ok that's what i'd expect to see in this case

what sort of size box are you running this on? does the IO max out?

yeah ok that's what i'd expect to see in this case what sort of size box are you running this on? does the IO max out?

bubblineyuri commented

2022-10-05 15:39:10 +00:00

Author

It's getting better, but it's still very slow. Some requests even don't get throguh now. I'm running koyu.space on a KVM VPS with the following specs:

10GB RAM
4 Cores (AMD EPYC)
200GB SSD

It's getting better, but it's still very slow. Some requests even don't get throguh now. I'm running koyu.space on a KVM VPS with the following specs: 10GB RAM 4 Cores (AMD EPYC) 200GB SSD

floatingghost commented

2022-10-05 15:47:44 +00:00

Owner

you may just have a very large backlog of tasks that is slowly processing

try leaving it online for an hour or so and see if it improves

you may just have a very large backlog of tasks that is slowly processing try leaving it online for an hour or so and see if it improves

bubblineyuri commented

2022-10-05 16:16:19 +00:00

Author

What I find ironic is that the timelines load super slow, but the rest like config etc. loads as it should

bubblineyuri commented

2022-10-05 16:25:23 +00:00

Author

Yes, it seems to have synced up, but timelines still load slow

bubblineyuri commented

2022-10-05 16:34:48 +00:00

Author

I'll let it sink in a little longer. Will report back.

floatingghost commented

2022-10-05 16:46:01 +00:00

Owner

other things that can cause slow timelines include having thread containment turned on, so check if that's on

it's off by default

other things that can cause slow timelines include having thread containment turned on, so check if that's on it's off by default

bubblineyuri commented

2022-10-05 17:55:31 +00:00

Author

Removing masto-fe related settings from the database made it kinda faster 🤔

bubblineyuri commented

2022-10-05 18:24:14 +00:00

Author

I'm also getting these from time to time

Oct 05 18:23:32 koyu.space mix[20932]: 18:23:32.108 [notice] Application ex_aws exited: :stopped
Oct 05 18:23:32 koyu.space mix[20932]: 18:23:32.108 [notice] Application web_push_encryption exited: :stopped
Oct 05 18:23:34 koyu.space mix[20932]: 18:23:34.453 [info] Postgrex.Protocol (#PID<0.10863.1>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.13157.1> exited

And the thread containment setting does nothing

I'm also getting these from time to time ``` Oct 05 18:23:32 koyu.space mix[20932]: 18:23:32.108 [notice] Application ex_aws exited: :stopped Oct 05 18:23:32 koyu.space mix[20932]: 18:23:32.108 [notice] Application web_push_encryption exited: :stopped Oct 05 18:23:34 koyu.space mix[20932]: 18:23:34.453 [info] Postgrex.Protocol (#PID<0.10863.1>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.13157.1> exited ``` And the thread containment setting does nothing

floatingghost commented

2022-10-05 18:31:07 +00:00

Owner

hm

well keep it off anyhow

https://pgtune.leopard.in.ua/ might be of use, maybe your Db isn't using as much of your hardware as it could

hm well keep it off anyhow https://pgtune.leopard.in.ua/ might be of use, maybe your Db isn't using as much of your hardware as it could

bubblineyuri commented

2022-10-05 18:39:49 +00:00

Author

This is also doing nothing, it's a real hard one

floatingghost commented

2022-10-05 20:00:25 +00:00

Owner

check your running queries again - is there a specific one that's taking a long time?

bubblineyuri commented

2022-10-05 20:32:30 +00:00

Author

check your running queries again - is there a specific one that's taking a long time?

This one has been trying to do something since start:

SELECT a0."id", a0."data", a0."local", a0."actor", a0."recipients", a0."inserted_at", a0."updated_at", o1."id", o1."data", o1."inserted_at", o1."updated_at" FROM "activities" AS a0 LEFT OUTER JOIN "objects" AS o1 ON (o1."data"->>'id') = COALESCE(a0."data"->'object'->>'id', a0."data"->>'object') WHERE (coalesce((a0."data")->'object'->>'id', (a0."data")->>'object') = ANY($1)) AND ((a0."data")->>'type' = $2)

> check your running queries again - is there a specific one that's taking a long time? This one has been trying to do something since start: ``` SELECT a0."id", a0."data", a0."local", a0."actor", a0."recipients", a0."inserted_at", a0."updated_at", o1."id", o1."data", o1."inserted_at", o1."updated_at" FROM "activities" AS a0 LEFT OUTER JOIN "objects" AS o1 ON (o1."data"->>'id') = COALESCE(a0."data"->'object'->>'id', a0."data"->>'object') WHERE (coalesce((a0."data")->'object'->>'id', (a0."data")->>'object') = ANY($1)) AND ((a0."data")->>'type' = $2) ```

floatingghost commented

2022-10-05 21:16:01 +00:00

Owner

none of that is particularly unusual

you could try turning on debug logging and seeing if you get anything?

you might also consider checking iotop to check if your IO is doing anything untoward

none of that is particularly unusual you could try turning on debug logging and seeing if you get anything? you might also consider checking `iotop` to check if your IO is doing anything untoward

bubblineyuri commented

2022-10-05 21:34:17 +00:00

Author

iotop is safe so I don't think it's disk IO. How do I enable debug logging?

floatingghost commented

2022-10-05 22:40:55 +00:00

Owner

there's a bunch of level stuff in the config

floatingghost commented

2022-10-05 22:45:07 +00:00

Owner

also, in case you didn't run it earlier

https://www.postgresql.org/docs/current/sql-vacuum.html

this may help

also, in case you didn't run it earlier https://www.postgresql.org/docs/current/sql-vacuum.html this may help

bubblineyuri commented

2022-10-06 13:34:53 +00:00

Author

I did an SQL vacuum and tried debug logging, but there's nothing of value. That's tough.

floatingghost commented

2022-10-06 15:54:31 +00:00

Owner

you can also try the pg_repack thing

you'll need to run CREATE EXTENSION pg_repack on your database before you run it (that's why you ran into the not installed thing)

you can also try the pg_repack thing you'll need to run `CREATE EXTENSION pg_repack` on your database before you run it (that's why you ran into the not installed thing)

bubblineyuri commented

2022-10-06 19:13:36 +00:00

Author

pg_repack returned ERROR: query failed: ERROR: could not create unique index "index_371024" and the server is still slow

pg_repack returned `ERROR: query failed: ERROR: could not create unique index "index_371024"` and the server is still slow

bubblineyuri changed title from ~~masto-fe errors when migrating from Pleroma~~ to A lot of errors when migrating from Pleroma

2022-10-06 19:13:57 +00:00

bubblineyuri commented

2022-10-06 19:21:27 +00:00

Author

I might have found the issue. Running a vacuum using mix takes a long time to finish and has a very high IO load.

bubblineyuri commented

2022-10-06 19:23:27 +00:00

Author

Running a vacuum didn't improve performance though. I had high hopes.

bubblineyuri commented

2022-10-06 19:25:33 +00:00

Author

Now I'm getting these funny things with a few requests:

Oct 06 19:24:47 koyu.space mix[622149]: 19:24:47.607 request_id=FxuRVH9dchMu0pwAAG9D [error] Internal server error: {:timeout, {GenServer, :call, [Pleroma.Stats, :get_state, 5000]}}
Oct 06 19:24:47 koyu.space mix[622149]: 19:24:47.617 request_id=FxuRVIHE4U-zvBcAAIKi [error] Internal server error: {:timeout, {GenServer, :call, [Pleroma.Stats, :get_state, 5000]}}
Oct 06 19:24:47 koyu.space mix[622149]: 19:24:47.627 [error] #PID<0.4003.0> running Pleroma.Web.Endpoint (connection #PID<0.4001.0>, stream id 1) terminated
Oct 06 19:24:47 koyu.space mix[622149]: Server: fedi.koyu.space:80 (http)
Oct 06 19:24:47 koyu.space mix[622149]: Request: GET /static/font/css/animation.css
Oct 06 19:24:47 koyu.space mix[622149]: ** (exit) exited in: GenServer.call(Pleroma.Stats, :get_state, 5000)
Oct 06 19:24:47 koyu.space mix[622149]:     ** (EXIT) time out
Oct 06 19:24:53 koyu.space mix[622149]: 19:24:53.580 request_id=FxuRVqXJCWEvLSkAAIcC [error] Internal server error: {:timeout, {GenServer, :call, [Pleroma.Stats, :get_state, 5000]}}
Oct 06 19:24:53 koyu.space mix[622149]: 19:24:53.580 [error] #PID<0.4164.0> running Pleroma.Web.Endpoint (connection #PID<0.4153.0>, stream id 1) terminated
Oct 06 19:24:53 koyu.space mix[622149]: Server: fedi.koyu.space:80 (http)
Oct 06 19:24:53 koyu.space mix[622149]: Request: GET /static/themes/mammal.json
Oct 06 19:24:53 koyu.space mix[622149]: ** (exit) exited in: GenServer.call(Pleroma.Stats, :get_state, 5000)
Oct 06 19:24:53 koyu.space mix[622149]:     ** (EXIT) time out

Now I'm getting these funny things with a few requests: ``` Oct 06 19:24:47 koyu.space mix[622149]: 19:24:47.607 request_id=FxuRVH9dchMu0pwAAG9D [error] Internal server error: {:timeout, {GenServer, :call, [Pleroma.Stats, :get_state, 5000]}} Oct 06 19:24:47 koyu.space mix[622149]: 19:24:47.617 request_id=FxuRVIHE4U-zvBcAAIKi [error] Internal server error: {:timeout, {GenServer, :call, [Pleroma.Stats, :get_state, 5000]}} Oct 06 19:24:47 koyu.space mix[622149]: 19:24:47.627 [error] #PID<0.4003.0> running Pleroma.Web.Endpoint (connection #PID<0.4001.0>, stream id 1) terminated Oct 06 19:24:47 koyu.space mix[622149]: Server: fedi.koyu.space:80 (http) Oct 06 19:24:47 koyu.space mix[622149]: Request: GET /static/font/css/animation.css Oct 06 19:24:47 koyu.space mix[622149]: ** (exit) exited in: GenServer.call(Pleroma.Stats, :get_state, 5000) Oct 06 19:24:47 koyu.space mix[622149]: ** (EXIT) time out Oct 06 19:24:53 koyu.space mix[622149]: 19:24:53.580 request_id=FxuRVqXJCWEvLSkAAIcC [error] Internal server error: {:timeout, {GenServer, :call, [Pleroma.Stats, :get_state, 5000]}} Oct 06 19:24:53 koyu.space mix[622149]: 19:24:53.580 [error] #PID<0.4164.0> running Pleroma.Web.Endpoint (connection #PID<0.4153.0>, stream id 1) terminated Oct 06 19:24:53 koyu.space mix[622149]: Server: fedi.koyu.space:80 (http) Oct 06 19:24:53 koyu.space mix[622149]: Request: GET /static/themes/mammal.json Oct 06 19:24:53 koyu.space mix[622149]: ** (exit) exited in: GenServer.call(Pleroma.Stats, :get_state, 5000) Oct 06 19:24:53 koyu.space mix[622149]: ** (EXIT) time out ```

bubblineyuri commented

2022-10-06 19:29:44 +00:00

Author

This is very interesting. I'm occasionally getting Oct 06 19:28:24 koyu.space mix[622149]: 19:28:24.173 [notice] :alarm_handler: {:clear, :system_memory_high_watermark} even though I still have 6.9 GB (nice) available. Why is it not using my entire RAM to work with? Timeout on static assets went away once I restarted the PostgreSQL server.

This is very interesting. I'm occasionally getting `Oct 06 19:28:24 koyu.space mix[622149]: 19:28:24.173 [notice] :alarm_handler: {:clear, :system_memory_high_watermark}` even though I still have 6.9 GB (nice) available. Why is it not using my entire RAM to work with? Timeout on static assets went away once I restarted the PostgreSQL server.

floatingghost commented

2022-10-06 19:41:01 +00:00

Owner

hold up, your static assets were timing out?

that would heavily indicate that your server does not have the disk throughput to run a database

hold up, your _static_ assets were timing out? that would heavily indicate that your server does not have the disk throughput to run a database

floatingghost commented

2022-10-06 19:44:27 +00:00

Owner

please benchmark your disk to ensure it has the read and write speeds necessary to comfortably run a database

bubblineyuri commented

2022-10-06 19:46:48 +00:00

Author

I also tried regenerating the entire config. That didn't help.

bubblineyuri commented

2022-10-06 19:47:22 +00:00

Author

please benchmark your disk to ensure it has the read and write speeds necessary to comfortably run a database

How do I do that? I mean Pleroma and bloaty Mastodon worked before.

> please benchmark your disk to ensure it has the read and write speeds necessary to comfortably run a database How do I do that? I mean Pleroma and bloaty Mastodon worked before.

floatingghost commented

2022-10-06 19:51:25 +00:00

Owner

use hdparm and dd

bubblineyuri commented

2022-10-06 20:24:44 +00:00

Author

koyu@koyu:~$ dd if=/dev/urandom of=test.img bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.6276 s, 191 MB/s

koyu@koyu:~$ sudo hdparm -T /dev/sda

/dev/sda:
 Timing cached reads:   13338 MB in  2.00 seconds = 6674.36 MB/sec

``` koyu@koyu:~$ dd if=/dev/urandom of=test.img bs=1M count=1024 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.6276 s, 191 MB/s ``` ``` koyu@koyu:~$ sudo hdparm -T /dev/sda /dev/sda: Timing cached reads: 13338 MB in 2.00 seconds = 6674.36 MB/sec ```

floatingghost commented

2022-10-06 20:52:30 +00:00

Owner

that really should be sufficient

but I've given you all the resources I can, there's very little else I can do remotely to diagnose

that really should be sufficient but I've given you all the resources I can, there's very little else I can do remotely to diagnose

bubblineyuri commented

2022-10-06 23:56:48 +00:00

Author

So I figured out that the whole database got corrupted. Rebuilding the index resulted in the whole database being yeeted.

floatingghost commented

2022-10-08 10:32:29 +00:00

Owner

if you've got a backup somewhere it should still be ok

bubblineyuri commented

2022-10-08 11:41:40 +00:00

Author

Well, I have a backup, but I can't restore it when the database index is corrupt

Clovis commented

2023-06-16 08:31:32 +00:00

Contributor

Had the same issue, it was indeed related to database corruption.
I was able to restore a working state by:

(psql, connected in your akkoma database)

VACUUM FULL;
REINDEX database your-akkoma-database;

Took some time, but now the random timeout are fixed !

PS: your database will be busy during those operations so warn your users / close the service for some time.

Had the same issue, it was indeed related to database corruption. I was able to restore a working state by: (psql, connected in your akkoma database) ``` VACUUM FULL; REINDEX database your-akkoma-database; ``` Took some time, but now the random timeout are fixed ! PS: your database will be busy during those operations so warn your users / close the service for some time.

A lot of errors when migrating from Pleroma #215