AkkomaGang/akkoma
AkkomaGang/akkoma
15
66
Fork 94
Code Issues 180 Pull requests 28 Projects 2 Releases 14 Packages Wiki Activity

A lot of errors when migrating from Pleroma #215

New issue
Closed
opened 2022-09-16 11:29:25 +00:00 by bubblineyuri · 55 comments
bubblineyuri commented 2022-09-16 11:29:25 +00:00
Copy link

Hi,

I get the following PostgreSQL error when trying to migrate from Pleroma to Akkoma

column u1.mastofe_settings does not exist

I hope you can help me with that.

~Leonie

Hi, I get the following PostgreSQL error when trying to migrate from Pleroma to Akkoma ``` column u1.mastofe_settings does not exist ``` I hope you can help me with that. ~Leonie
floatingghost commented 2022-09-16 11:55:16 +00:00
Owner
Copy link

Ah, this could occur if migrating from develop pleroma where you'd already applied the removal of mastofe patch

Should be fixed via #216

if you're from source, git pull and try again

if you're OTP, wait for https://ci.akkoma.dev/AkkomaGang/akkoma/build/930 to complete then update and try again

Ah, this could occur if migrating from develop pleroma where you'd already applied the removal of mastofe patch Should be fixed via https://akkoma.dev/AkkomaGang/akkoma/pulls/216 if you're from source, git pull and try again if you're OTP, wait for https://ci.akkoma.dev/AkkomaGang/akkoma/build/930 to complete then update and try again
bubblineyuri commented 2022-09-16 13:18:42 +00:00
Author
Copy link

This didn't fix anything, now I get these fun messages http://content.koyu.space/1MPAN

This didn't fix anything, now I get these fun messages http://content.koyu.space/1MPAN
floatingghost commented 2022-09-16 13:21:36 +00:00
Owner
Copy link

ok so it actually did fix your error since these are totally different

sounds like you've got your pool target set really low - it should never be that low by default

you'll want to increase it by changing queue_target like so:

config :pleroma, Pleroma.Repo,
  adapter: Ecto.Adapters.Postgres,
  ... creds are probably here...
  timeout: 90_000,
  queue_target: 20_000, # THIS ONE
  queue_interval: 2_000,
  pool_size: 10
ok so it actually _did_ fix your error since these are totally different sounds like you've got your pool target set really low - it should never be that low by default you'll want to increase it by changing `queue_target` like so: ```elixir config :pleroma, Pleroma.Repo, adapter: Ecto.Adapters.Postgres, ... creds are probably here... timeout: 90_000, queue_target: 20_000, # THIS ONE queue_interval: 2_000, pool_size: 10 ```
bubblineyuri commented 2022-10-05 11:38:19 +00:00
Author
Copy link

Getting this beauty here

Oct 05 11:35:12 koyu.space mix[2454618]: 11:35:12.038 [info] Postgrex.Protocol (#PID<0.5170.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.6331.0> exited

Getting this beauty here `Oct 05 11:35:12 koyu.space mix[2454618]: 11:35:12.038 [info] Postgrex.Protocol (#PID<0.5170.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.6331.0> exited`
floatingghost commented 2022-10-05 13:15:07 +00:00
Owner
Copy link

that alone doesn't give an awful lot to go on, there should be more logs above it that indicate what used the exit

that alone doesn't give an awful lot to go on, there should be more logs above it that indicate what used the exit
bubblineyuri commented 2022-10-05 13:23:34 +00:00
Author
Copy link

Is this a little better? http://content.koyu.space/A6tGK

Is this a little better? http://content.koyu.space/A6tGK
floatingghost commented 2022-10-05 13:24:53 +00:00
Owner
Copy link

hm, that doesn't look fatal - does the instance die or does it recover after doing that?

hm, that doesn't look fatal - does the instance die or does it recover after doing that?
bubblineyuri commented 2022-10-05 13:33:06 +00:00
Author
Copy link

It loads the UI and responds to requests very slowly

It loads the UI and responds to requests very slowly
floatingghost commented 2022-10-05 13:43:48 +00:00
Owner
Copy link

hm, that sounds like you might have too big of a database for your system

have you done a vacuum/pg_repack to remove stuff?

additionally, how long is your remote object retention? that may be causing object table inflation

hm, that sounds like you might have too big of a database for your system have you done a vacuum/pg_repack to remove stuff? additionally, how long is your remote object retention? that may be causing object table inflation
bubblineyuri commented 2022-10-05 13:59:09 +00:00
Author
Copy link

I have no idea what those two are. I also store configuration in the database, so how do I check that if I've blown off half of the server?

I have no idea what those two are. I also store configuration in the database, so how do I check that if I've blown off half of the server?
floatingghost commented 2022-10-05 14:01:54 +00:00
Owner
Copy link

vacuum - https://docs.akkoma.dev/stable/administration/CLI_tasks/database/#prune-old-remote-posts-from-the-database

pg_repack - https://github.com/reorg/pg_repack

show options in the database - https://docs.akkoma.dev/stable/administration/CLI_tasks/config/#dump-all-of-the-config-settings-defined-in-the-database

vacuum - https://docs.akkoma.dev/stable/administration/CLI_tasks/database/#prune-old-remote-posts-from-the-database pg_repack - https://github.com/reorg/pg_repack show options in the database - https://docs.akkoma.dev/stable/administration/CLI_tasks/config/#dump-all-of-the-config-settings-defined-in-the-database
bubblineyuri commented 2022-10-05 14:07:54 +00:00
Author
Copy link

There is no object retention configuration. Should I try configuring it or assume it has some sort of default?

Doing a pg_repack does ERROR: pg_repack failed with error: pg_repack 1.4.7 is not installed in the database

There is no object retention configuration. Should I try configuring it or assume it has some sort of default? Doing a pg_repack does `ERROR: pg_repack failed with error: pg_repack 1.4.7 is not installed in the database`
floatingghost commented 2022-10-05 14:08:55 +00:00
Owner
Copy link

it'll have a default then, you should just be able to run the prune command from the docs.akkoma and it'll remove stuff

it'll have a default then, you should just be able to run the prune command from the docs.akkoma and it'll remove stuff
bubblineyuri commented 2022-10-05 14:11:06 +00:00
Author
Copy link

This is getting more interesting now after pruning

Oct 05 14:10:26 koyu.space mix[2497179]: 14:10:26.757 [error] Error while fetching https://fedi.absturztau.be/objects/b51ef47d-46bf-420d-b3d2-4ac22a1126d1: {:error, {:transmogrifier, {:error, {:validate, {:error, #Ecto.Changeset<action: :insert, changes: %{actor: "https://fedi.absturztau.be/users/khaosgrille", cc: ["https://www.w3.org/ns/activitystreams#Public"], context: "https://brotka.st/contexts/bc241591-d030-4d42-8868-4948f437cc86", object: "https://fedi.absturztau.be/objects/b51ef47d-46bf-420d-b3d2-4ac22a1126d1", to: ["https://brotka.st/users/kaia", "https://fedi.absturztau.be/users/khaosgrille/followers", "https://freespeechextremist.com/users/xue"], type: "Create"}, errors: [object: {"The object to create already exists", []}], data: #Pleroma.Web.ActivityPub.ObjectValidators.CreateGenericValidator<>, valid?: false>}}}}}
Oct 05 14:10:26 koyu.space mix[2497179]: 14:10:26.757 [warning] Couldn't fetch "https://fedi.absturztau.be/objects/b51ef47d-46bf-420d-b3d2-4ac22a1126d1", error: nil
This is getting more interesting now after pruning ``` Oct 05 14:10:26 koyu.space mix[2497179]: 14:10:26.757 [error] Error while fetching https://fedi.absturztau.be/objects/b51ef47d-46bf-420d-b3d2-4ac22a1126d1: {:error, {:transmogrifier, {:error, {:validate, {:error, #Ecto.Changeset<action: :insert, changes: %{actor: "https://fedi.absturztau.be/users/khaosgrille", cc: ["https://www.w3.org/ns/activitystreams#Public"], context: "https://brotka.st/contexts/bc241591-d030-4d42-8868-4948f437cc86", object: "https://fedi.absturztau.be/objects/b51ef47d-46bf-420d-b3d2-4ac22a1126d1", to: ["https://brotka.st/users/kaia", "https://fedi.absturztau.be/users/khaosgrille/followers", "https://freespeechextremist.com/users/xue"], type: "Create"}, errors: [object: {"The object to create already exists", []}], data: #Pleroma.Web.ActivityPub.ObjectValidators.CreateGenericValidator<>, valid?: false>}}}}} Oct 05 14:10:26 koyu.space mix[2497179]: 14:10:26.757 [warning] Couldn't fetch "https://fedi.absturztau.be/objects/b51ef47d-46bf-420d-b3d2-4ac22a1126d1", error: nil ```
floatingghost commented 2022-10-05 14:14:19 +00:00
Owner
Copy link

ok, that's fine, that error isn't fatal at all, just a refetch
you can safely ignore it

ok, that's fine, that error isn't fatal at all, just a refetch you can safely ignore it
bubblineyuri commented 2022-10-05 14:15:57 +00:00
Author
Copy link

How about this? http://content.koyu.space/XIXb4

The server is still super slow

How about this? http://content.koyu.space/XIXb4 The server is still super slow
floatingghost commented 2022-10-05 14:21:32 +00:00
Owner
Copy link

that's an interesting one, seems it doesn't like some of your config

that sounds like you may have an outdated schema in your config

can you add

config :pleroma, :instance,
  staff_transparency: []

in your config?

if you run from source, does your config/config.exs match the one in our version control?

that's an interesting one, seems it doesn't like some of your config that sounds like you may have an outdated schema in your config can you add ``` config :pleroma, :instance, staff_transparency: [] ``` in your config? if you run from source, does your `config/config.exs` match the one in our version control?
bubblineyuri commented 2022-10-05 14:27:28 +00:00
Author
Copy link

Now it tries to eat itself http://content.koyu.space/Mj6z8

Copied config from version control and added the config flag you mentioned

Now it tries to eat itself http://content.koyu.space/Mj6z8 Copied config from version control and added the config flag you mentioned
floatingghost commented 2022-10-05 14:33:24 +00:00
Owner
Copy link

nice! that means we're past the worst of it, we've got standard inbound requests and cron activating

looks like you might have some long-running requests

when the server is up and the timeouts are occuring, run SELECT * FROM pg_stat_activity; on your db and see if it throws anything interesting

nice! that means we're past the worst of it, we've got standard inbound requests and cron activating looks like you might have some long-running requests when the server is up and the timeouts are occuring, run `SELECT * FROM pg_stat_activity;` on your db and see if it throws anything interesting
bubblineyuri commented 2022-10-05 14:40:55 +00:00
Author
Copy link

These are the first 20 seconds or so and it's clogging up with queries. This looks awful I guess.

grafik

These are the first 20 seconds or so and it's clogging up with queries. This looks awful I guess. ![grafik](/attachments/af8d47c4-8ac7-41a1-9180-1a5d67e813ee)
grafik.png
20 KiB
floatingghost commented 2022-10-05 14:41:43 +00:00
Owner
Copy link

yeah ok that's what i'd expect to see in this case

what sort of size box are you running this on? does the IO max out?

yeah ok that's what i'd expect to see in this case what sort of size box are you running this on? does the IO max out?
bubblineyuri commented 2022-10-05 15:39:10 +00:00
Author
Copy link

It's getting better, but it's still very slow. Some requests even don't get throguh now. I'm running koyu.space on a KVM VPS with the following specs:

10GB RAM
4 Cores (AMD EPYC)
200GB SSD

It's getting better, but it's still very slow. Some requests even don't get throguh now. I'm running koyu.space on a KVM VPS with the following specs: 10GB RAM 4 Cores (AMD EPYC) 200GB SSD
floatingghost commented 2022-10-05 15:47:44 +00:00
Owner
Copy link

you may just have a very large backlog of tasks that is slowly processing

try leaving it online for an hour or so and see if it improves

you may just have a very large backlog of tasks that is slowly processing try leaving it online for an hour or so and see if it improves
bubblineyuri commented 2022-10-05 16:16:19 +00:00
Author
Copy link

What I find ironic is that the timelines load super slow, but the rest like config etc. loads as it should

What I find ironic is that the timelines load super slow, but the rest like config etc. loads as it should
bubblineyuri commented 2022-10-05 16:25:23 +00:00
Author
Copy link

Yes, it seems to have synced up, but timelines still load slow

Yes, it seems to have synced up, but timelines still load slow
bubblineyuri commented 2022-10-05 16:34:48 +00:00
Author
Copy link

I'll let it sink in a little longer. Will report back.

I'll let it sink in a little longer. Will report back.
floatingghost commented 2022-10-05 16:46:01 +00:00
Owner
Copy link

other things that can cause slow timelines include having thread containment turned on, so check if that's on

it's off by default

other things that can cause slow timelines include having thread containment turned on, so check if that's on it's off by default
bubblineyuri commented 2022-10-05 17:55:31 +00:00
Author
Copy link

Removing masto-fe related settings from the database made it kinda faster 🤔

Removing masto-fe related settings from the database made it kinda faster 🤔
bubblineyuri commented 2022-10-05 18:24:14 +00:00
Author
Copy link

I'm also getting these from time to time

Oct 05 18:23:32 koyu.space mix[20932]: 18:23:32.108 [notice] Application ex_aws exited: :stopped
Oct 05 18:23:32 koyu.space mix[20932]: 18:23:32.108 [notice] Application web_push_encryption exited: :stopped
Oct 05 18:23:34 koyu.space mix[20932]: 18:23:34.453 [info] Postgrex.Protocol (#PID<0.10863.1>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.13157.1> exited

And the thread containment setting does nothing

I'm also getting these from time to time ``` Oct 05 18:23:32 koyu.space mix[20932]: 18:23:32.108 [notice] Application ex_aws exited: :stopped Oct 05 18:23:32 koyu.space mix[20932]: 18:23:32.108 [notice] Application web_push_encryption exited: :stopped Oct 05 18:23:34 koyu.space mix[20932]: 18:23:34.453 [info] Postgrex.Protocol (#PID<0.10863.1>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.13157.1> exited ``` And the thread containment setting does nothing
floatingghost commented 2022-10-05 18:31:07 +00:00
Owner
Copy link

hm

well keep it off anyhow

https://pgtune.leopard.in.ua/ might be of use, maybe your Db isn't using as much of your hardware as it could

hm well keep it off anyhow https://pgtune.leopard.in.ua/ might be of use, maybe your Db isn't using as much of your hardware as it could
bubblineyuri commented 2022-10-05 18:39:49 +00:00
Author
Copy link

This is also doing nothing, it's a real hard one

This is also doing nothing, it's a real hard one
floatingghost commented 2022-10-05 20:00:25 +00:00
Owner
Copy link

check your running queries again - is there a specific one that's taking a long time?

check your running queries again - is there a specific one that's taking a long time?
bubblineyuri commented 2022-10-05 20:32:30 +00:00
Author
Copy link

check your running queries again - is there a specific one that's taking a long time?

This one has been trying to do something since start:

SELECT a0."id", a0."data", a0."local", a0."actor", a0."recipients", a0."inserted_at", a0."updated_at", o1."id", o1."data", o1."inserted_at", o1."updated_at" FROM "activities" AS a0 LEFT OUTER JOIN "objects" AS o1 ON (o1."data"->>'id') = COALESCE(a0."data"->'object'->>'id', a0."data"->>'object') WHERE (coalesce((a0."data")->'object'->>'id', (a0."data")->>'object') = ANY($1)) AND ((a0."data")->>'type' = $2)
> check your running queries again - is there a specific one that's taking a long time? This one has been trying to do something since start: ``` SELECT a0."id", a0."data", a0."local", a0."actor", a0."recipients", a0."inserted_at", a0."updated_at", o1."id", o1."data", o1."inserted_at", o1."updated_at" FROM "activities" AS a0 LEFT OUTER JOIN "objects" AS o1 ON (o1."data"->>'id') = COALESCE(a0."data"->'object'->>'id', a0."data"->>'object') WHERE (coalesce((a0."data")->'object'->>'id', (a0."data")->>'object') = ANY($1)) AND ((a0."data")->>'type' = $2) ```
floatingghost commented 2022-10-05 21:16:01 +00:00
Owner
Copy link

none of that is particularly unusual

you could try turning on debug logging and seeing if you get anything?

you might also consider checking iotop to check if your IO is doing anything untoward

none of that is particularly unusual you could try turning on debug logging and seeing if you get anything? you might also consider checking `iotop` to check if your IO is doing anything untoward
bubblineyuri commented 2022-10-05 21:34:17 +00:00
Author
Copy link

iotop is safe so I don't think it's disk IO. How do I enable debug logging?

iotop is safe so I don't think it's disk IO. How do I enable debug logging?
floatingghost commented 2022-10-05 22:40:55 +00:00
Owner
Copy link

there's a bunch of level stuff in the config

there's a bunch of level stuff in the config
floatingghost commented 2022-10-05 22:45:07 +00:00
Owner
Copy link

also, in case you didn't run it earlier

https://www.postgresql.org/docs/current/sql-vacuum.html

this may help

also, in case you didn't run it earlier https://www.postgresql.org/docs/current/sql-vacuum.html this may help
bubblineyuri commented 2022-10-06 13:34:53 +00:00
Author
Copy link

I did an SQL vacuum and tried debug logging, but there's nothing of value. That's tough.

I did an SQL vacuum and tried debug logging, but there's nothing of value. That's tough.
floatingghost commented 2022-10-06 15:54:31 +00:00
Owner
Copy link

you can also try the pg_repack thing

you'll need to run CREATE EXTENSION pg_repack on your database before you run it (that's why you ran into the not installed thing)

you can also try the pg_repack thing you'll need to run `CREATE EXTENSION pg_repack` on your database before you run it (that's why you ran into the not installed thing)
bubblineyuri commented 2022-10-06 19:13:36 +00:00
Author
Copy link

pg_repack returned ERROR: query failed: ERROR: could not create unique index "index_371024" and the server is still slow

pg_repack returned `ERROR: query failed: ERROR: could not create unique index "index_371024"` and the server is still slow
bubblineyuri changed title from masto-fe errors when migrating from Pleroma to A lot of errors when migrating from Pleroma 2022-10-06 19:13:57 +00:00
bubblineyuri commented 2022-10-06 19:21:27 +00:00
Author
Copy link

I might have found the issue. Running a vacuum using mix takes a long time to finish and has a very high IO load.

I might have found the issue. Running a vacuum using mix takes a long time to finish and has a very high IO load.
bubblineyuri commented 2022-10-06 19:23:27 +00:00
Author
Copy link

Running a vacuum didn't improve performance though. I had high hopes.

Running a vacuum didn't improve performance though. I had high hopes.
bubblineyuri commented 2022-10-06 19:25:33 +00:00
Author
Copy link

Now I'm getting these funny things with a few requests:

Oct 06 19:24:47 koyu.space mix[622149]: 19:24:47.607 request_id=FxuRVH9dchMu0pwAAG9D [error] Internal server error: {:timeout, {GenServer, :call, [Pleroma.Stats, :get_state, 5000]}}
Oct 06 19:24:47 koyu.space mix[622149]: 19:24:47.617 request_id=FxuRVIHE4U-zvBcAAIKi [error] Internal server error: {:timeout, {GenServer, :call, [Pleroma.Stats, :get_state, 5000]}}
Oct 06 19:24:47 koyu.space mix[622149]: 19:24:47.627 [error] #PID<0.4003.0> running Pleroma.Web.Endpoint (connection #PID<0.4001.0>, stream id 1) terminated
Oct 06 19:24:47 koyu.space mix[622149]: Server: fedi.koyu.space:80 (http)
Oct 06 19:24:47 koyu.space mix[622149]: Request: GET /static/font/css/animation.css
Oct 06 19:24:47 koyu.space mix[622149]: ** (exit) exited in: GenServer.call(Pleroma.Stats, :get_state, 5000)
Oct 06 19:24:47 koyu.space mix[622149]:     ** (EXIT) time out
Oct 06 19:24:53 koyu.space mix[622149]: 19:24:53.580 request_id=FxuRVqXJCWEvLSkAAIcC [error] Internal server error: {:timeout, {GenServer, :call, [Pleroma.Stats, :get_state, 5000]}}
Oct 06 19:24:53 koyu.space mix[622149]: 19:24:53.580 [error] #PID<0.4164.0> running Pleroma.Web.Endpoint (connection #PID<0.4153.0>, stream id 1) terminated
Oct 06 19:24:53 koyu.space mix[622149]: Server: fedi.koyu.space:80 (http)
Oct 06 19:24:53 koyu.space mix[622149]: Request: GET /static/themes/mammal.json
Oct 06 19:24:53 koyu.space mix[622149]: ** (exit) exited in: GenServer.call(Pleroma.Stats, :get_state, 5000)
Oct 06 19:24:53 koyu.space mix[622149]:     ** (EXIT) time out
Now I'm getting these funny things with a few requests: ``` Oct 06 19:24:47 koyu.space mix[622149]: 19:24:47.607 request_id=FxuRVH9dchMu0pwAAG9D [error] Internal server error: {:timeout, {GenServer, :call, [Pleroma.Stats, :get_state, 5000]}} Oct 06 19:24:47 koyu.space mix[622149]: 19:24:47.617 request_id=FxuRVIHE4U-zvBcAAIKi [error] Internal server error: {:timeout, {GenServer, :call, [Pleroma.Stats, :get_state, 5000]}} Oct 06 19:24:47 koyu.space mix[622149]: 19:24:47.627 [error] #PID<0.4003.0> running Pleroma.Web.Endpoint (connection #PID<0.4001.0>, stream id 1) terminated Oct 06 19:24:47 koyu.space mix[622149]: Server: fedi.koyu.space:80 (http) Oct 06 19:24:47 koyu.space mix[622149]: Request: GET /static/font/css/animation.css Oct 06 19:24:47 koyu.space mix[622149]: ** (exit) exited in: GenServer.call(Pleroma.Stats, :get_state, 5000) Oct 06 19:24:47 koyu.space mix[622149]: ** (EXIT) time out Oct 06 19:24:53 koyu.space mix[622149]: 19:24:53.580 request_id=FxuRVqXJCWEvLSkAAIcC [error] Internal server error: {:timeout, {GenServer, :call, [Pleroma.Stats, :get_state, 5000]}} Oct 06 19:24:53 koyu.space mix[622149]: 19:24:53.580 [error] #PID<0.4164.0> running Pleroma.Web.Endpoint (connection #PID<0.4153.0>, stream id 1) terminated Oct 06 19:24:53 koyu.space mix[622149]: Server: fedi.koyu.space:80 (http) Oct 06 19:24:53 koyu.space mix[622149]: Request: GET /static/themes/mammal.json Oct 06 19:24:53 koyu.space mix[622149]: ** (exit) exited in: GenServer.call(Pleroma.Stats, :get_state, 5000) Oct 06 19:24:53 koyu.space mix[622149]: ** (EXIT) time out ```
bubblineyuri commented 2022-10-06 19:29:44 +00:00
Author
Copy link

This is very interesting. I'm occasionally getting Oct 06 19:28:24 koyu.space mix[622149]: 19:28:24.173 [notice] :alarm_handler: {:clear, :system_memory_high_watermark} even though I still have 6.9 GB (nice) available. Why is it not using my entire RAM to work with? Timeout on static assets went away once I restarted the PostgreSQL server.

This is very interesting. I'm occasionally getting `Oct 06 19:28:24 koyu.space mix[622149]: 19:28:24.173 [notice] :alarm_handler: {:clear, :system_memory_high_watermark}` even though I still have 6.9 GB (nice) available. Why is it not using my entire RAM to work with? Timeout on static assets went away once I restarted the PostgreSQL server.
floatingghost commented 2022-10-06 19:41:01 +00:00
Owner
Copy link

hold up, your static assets were timing out?

that would heavily indicate that your server does not have the disk throughput to run a database

hold up, your _static_ assets were timing out? that would heavily indicate that your server does not have the disk throughput to run a database
floatingghost commented 2022-10-06 19:44:27 +00:00
Owner
Copy link

please benchmark your disk to ensure it has the read and write speeds necessary to comfortably run a database

please benchmark your disk to ensure it has the read and write speeds necessary to comfortably run a database
bubblineyuri commented 2022-10-06 19:46:48 +00:00
Author
Copy link

I also tried regenerating the entire config. That didn't help.

I also tried regenerating the entire config. That didn't help.
bubblineyuri commented 2022-10-06 19:47:22 +00:00
Author
Copy link

please benchmark your disk to ensure it has the read and write speeds necessary to comfortably run a database

How do I do that? I mean Pleroma and bloaty Mastodon worked before.

> please benchmark your disk to ensure it has the read and write speeds necessary to comfortably run a database How do I do that? I mean Pleroma and bloaty Mastodon worked before.
floatingghost commented 2022-10-06 19:51:25 +00:00
Owner
Copy link

use hdparm and dd

use hdparm and dd
bubblineyuri commented 2022-10-06 20:24:44 +00:00
Author
Copy link
koyu@koyu:~$ dd if=/dev/urandom of=test.img bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.6276 s, 191 MB/s

koyu@koyu:~$ sudo hdparm -T /dev/sda

/dev/sda:
 Timing cached reads:   13338 MB in  2.00 seconds = 6674.36 MB/sec
``` koyu@koyu:~$ dd if=/dev/urandom of=test.img bs=1M count=1024 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.6276 s, 191 MB/s ``` ``` koyu@koyu:~$ sudo hdparm -T /dev/sda /dev/sda: Timing cached reads: 13338 MB in 2.00 seconds = 6674.36 MB/sec ```
floatingghost commented 2022-10-06 20:52:30 +00:00
Owner
Copy link

that really should be sufficient

but I've given you all the resources I can, there's very little else I can do remotely to diagnose

that really should be sufficient but I've given you all the resources I can, there's very little else I can do remotely to diagnose
bubblineyuri commented 2022-10-06 23:56:48 +00:00
Author
Copy link

So I figured out that the whole database got corrupted. Rebuilding the index resulted in the whole database being yeeted.

So I figured out that the whole database got corrupted. Rebuilding the index resulted in the whole database being yeeted.
floatingghost commented 2022-10-08 10:32:29 +00:00
Owner
Copy link

if you've got a backup somewhere it should still be ok

if you've got a backup somewhere it should still be ok
bubblineyuri commented 2022-10-08 11:41:40 +00:00
Author
Copy link

Well, I have a backup, but I can't restore it when the database index is corrupt

Well, I have a backup, but I can't restore it when the database index is corrupt
Clovis commented 2023-06-16 08:31:32 +00:00
Contributor
Copy link

Had the same issue, it was indeed related to database corruption.
I was able to restore a working state by:

(psql, connected in your akkoma database)

VACUUM FULL;
REINDEX database your-akkoma-database;

Took some time, but now the random timeout are fixed !

PS: your database will be busy during those operations so warn your users / close the service for some time.

Had the same issue, it was indeed related to database corruption. I was able to restore a working state by: (psql, connected in your akkoma database) ``` VACUUM FULL; REINDEX database your-akkoma-database; ``` Took some time, but now the random timeout are fixed ! PS: your database will be busy during those operations so warn your users / close the service for some time.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
develop
better-stats
stable
translations
customizable-docker-db
static-adminfe
bad-uploader-config-warning
docker
user-expiry-backfill
config-db-keys
ensure-scrubbers-loaded
otp26
code-shaking-does-not-go-brr
elixir1.15
circleci
backup-fixes
mod-task
policy
mrf-coolness
frontend-switcher-9000
contentMap
language-on-posts
rabbit
credo-on-pr
unify-http
normalise-markup-by-default
buildx
v3.13.3
v3.13.2
v2.6.3
v3.13.1
v3.13.0
snac-3.12.1.2
v3.12.2
snac-3.12.1.1
v3.12.1
v3.12.0
v3.11.0
v2.6.2
v2.6.1
2.6.1
v2.6.0
v2.5.5
v3.10.4
v3.10.3
v3.10.2
v3.10.1
v3.10.0
v2.5.4
v2.5.3
v3.9.3
v3.9.2
v3.9.1
V3.9.0
2023.05
v3.8.0
v3.7.1
v3.7.0
v3.6.0
v3.5.0
v2.4.5
v3.4.0
v3.3.1
v2.4.4
v3.3.0
v3.2.3
v3.2.2
v3.2.1
stable-202209
v3.2.0
v3.1.0
stable-2022.07
v3.0.0
v0.0.1
v2.5.2-6
v2.5.2-5
v2.5.2-4
v2.5.2-3
v2.5.2-2
v2.5.2-1
v2.5.2
v2.5.1
v2.5.0-8
v2.5.0-7
v2.5.0-6
v2.5.0-5
v2.5.0-4
v2.5.0-3
v2.5.0-2
v2.5.0-1
v2.5.0
v2.4.3
pre-rebase
v2.4.2
v2.4.1
v2.4.0
soapbox-v1.1.1
soapbox-v1.1.0
soapbox-v1.0.0
v2.3.0
v2.2.2
v2.2.1
v2.2.0
v2.1.2
v2.1.1
v2.1.0
v2.0.7
v2.0.6
v2.0.5
v2.0.4
v2.0.3
v2.0.2
v2.0.1
v2.0.0
v1.1.9
v1.1.8
v1.1.7
v1.1.6
v1.1.5
v1.1.4
v1.1.3
v1.1.2
v1.1.1
v1.1.0
v1.0.91
v1.0.90
v1.0.7
v1.0.6
v1.0.5
v1.0.4
v1.0.3
v1.0.2
v1.0.1
v1.0.0
v0.9.0
Labels
Clear labels   
approved, awaiting change

   
bug

Something is not working

  
configuration

This requires config changes

  
documentation

This is about human-readable docs

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
extremely low priority

   
feature request

Something new?

  
Fix it yourself

Support has been attempted

  
help wanted

Need some help

  
invalid

Something is wrong

  
mastodon_api

   
needs docs

   
needs tests

   
not a bug

   
planned

   
pleroma_api

   
privacy

   
question

More information is needed

  
static_fe

   
triage

   
wontfix

This won't be fixed

No labels
approved, awaiting change
bug
configuration
documentation
duplicate
enhancement
extremely low priority
feature request
Fix it yourself
help wanted
invalid
mastodon_api
needs docs
needs tests
not a bug
planned
pleroma_api
privacy
question
static_fe
triage
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: AkkomaGang/akkoma#215
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?