Unable to access my Home timeline of https://pleroma.envs.net even after repeatedly clicking on "Load older posts".

srijit commented

2023-02-27 14:11:38 +00:00

Your setup

OTP

Extra details

No response

Version

Akkoma stable 2023.02

PostgreSQL version

No response

What were you trying to do?

I logged in https://pleroma.envs.net.

What did you expect to happen?

The Home timeline should get loaded even without clicking "Load older posts".

What actually happened?

I am unable to access my Home timeline. Instead of Home timeline getting loaded, I am repeatedly getting the following error message when I attempt to access the Home timeline by clicking on "Load older posts".

Error fetching timeline: Unexpected status: 502

Other timelines are working fine. This problem did not exist with Akkoma 3.5.0.

Logs

No response

Severity

I cannot use it as easily as I'd like. Without the Home timeline, it is difficult to read the posts of the followed users and hashtags.

Have you searched for this issue?

I have double-checked and have not found this issue mentioned anywhere.

### Your setup OTP ### Extra details _No response_ ### Version Akkoma stable 2023.02 ### PostgreSQL version _No response_ ### What were you trying to do? I logged in https://pleroma.envs.net. ### What did you expect to happen? The [Home timeline](https://pleroma.envs.net/main/friends) should get loaded even without clicking "Load older posts". ### What actually happened? I am unable to access my [Home timeline](https://pleroma.envs.net/main/friends). Instead of [Home timeline](https://pleroma.envs.net/main/friends) getting loaded, I am repeatedly getting the following error message when I attempt to access the [Home timeline](https://pleroma.envs.net/main/friends) by clicking on "Load older posts". Error fetching timeline: Unexpected status: 502 Other timelines are working fine. This problem did not exist with Akkoma 3.5.0. ### Logs _No response_ ### Severity I cannot use it as easily as I'd like. Without the Home timeline, it is difficult to read the posts of the followed users and hashtags. ### Have you searched for this issue? - [x] I have double-checked and have not found this issue mentioned anywhere.

😕 1

srijit added the

bug

label 2023-02-27 14:11:38 +00:00

gamerfort commented

2023-03-05 22:05:37 +00:00

I have occasionally seen similar issues on my instance. I don't have much to go on, but they seem to have increased in frequency slightly ever since I started using the newer mobile apps which seem to use the timeline marker API of Mastodon to try to sync the timeline. I've been turning off that feature in the apps I use when I can to see if it lowers the incidence of it.

I don't have the tools to debug that, though.

I have occasionally seen similar issues on my instance. I don't have much to go on, but they seem to have increased in frequency slightly ever since I started using the newer mobile apps which seem to use the timeline marker API of Mastodon to try to sync the timeline. I've been turning off that feature in the apps I use when I can to see if it lowers the incidence of it. I don't have the tools to debug that, though.

sl commented

2023-03-08 11:04:40 +00:00

Same issue, I got a feeling that it gradually got worse and now home timeline won't load at all.

floatingghost commented

2023-03-08 17:29:31 +00:00

Owner

there's a diagnostics task to help you debug

https://akkoma.dev/AkkomaGang/akkoma/src/branch/develop/lib/mix/tasks/pleroma/diagnostics.ex#L20

there's a diagnostics task to help you debug https://akkoma.dev/AkkomaGang/akkoma/src/branch/develop/lib/mix/tasks/pleroma/diagnostics.ex#L20

❤️ 1

sl commented

2023-03-08 18:35:30 +00:00

The debug output for my account was sent moments ago as direct ActivityPub message. We (admin & I) didn't see any thing that would indicate something being off. We assume that @srijit is experiencing exactly the same problem as I am.

floatingghost commented

2023-03-08 22:45:32 +00:00

Owner

you may also want to check if turning rich media processing off helps you, that is another common cause of this

srijit commented

2023-03-09 04:41:10 +00:00

Author

you may also want to check if turning rich media processing off helps you, that is another common cause of this

> you may also want to check if turning rich media processing off helps you, that is another common cause of this

srijit closed this issue

2023-03-09 04:41:10 +00:00

srijit reopened this issue

2023-03-09 04:41:27 +00:00

srijit commented

2023-03-09 04:42:18 +00:00

Author

The debug output for my account was sent moments ago as direct ActivityPub message. We (admin & I) didn't see any thing that would indicate something being off.

We assume that @srijit is experiencing exactly the same problem as I am.

Yes

> The debug output for my account was sent moments ago as direct ActivityPub message. We (admin & I) didn't see any thing that would indicate something being off. > We assume that @srijit is experiencing exactly the same problem as I am. Yes

srijit commented

2023-03-09 04:44:42 +00:00

Author

you may also want to check if turning rich media processing off helps you, that is another common cause of this

Please let me know which setting I should change to turn off rich media processing.

> > you may also want to check if turning rich media processing off helps you, that is another common cause of this > Please let me know which setting I should change to turn off rich media processing.

floatingghost commented

2023-03-09 06:03:24 +00:00

Owner

I need to make a "let me Google that for you" but it's "let me search the documentation for you'

https://docs.akkoma.dev/stable/configuration/cheatsheet/#rich_media-consumer

I need to make a "let me Google that for you" but it's "let me search the documentation for you' https://docs.akkoma.dev/stable/configuration/cheatsheet/#rich_media-consumer

sl commented

2023-03-09 08:12:17 +00:00

Sorry, @srijit / me do not have direct access to Akkoma configuration. But I just moments ago confirmed with the admin, that turning off the rich media processing didn't change situation at all and the home timeline issue remained. Also when the rich media processing is off, the debug dump is still similar to what I sent privately earlier.

Would you happen to have any more ideas / things / debug things to try? - As mentioned, I guess we all are sorry, because this is frustrating to all of us.

Sorry, @srijit / me do not have direct access to Akkoma configuration. But I just moments ago confirmed with the admin, that turning off the rich media processing didn't change situation at all and the home timeline issue remained. Also when the rich media processing is off, the debug dump is still similar to what I sent privately earlier. Would you happen to have any more ideas / things / debug things to try? - As mentioned, I guess we all are sorry, because this is frustrating to all of us.

sl referenced this issue

2023-03-25 08:25:19 +00:00

[bug] Timeline Internal server error with no logs #508

sl commented

2023-03-26 07:43:18 +00:00

https://pleroma.envs.net has been updated to develop branch, but the issue still persists. Are more users are checking in and saying it isn't working.

https://pleroma.envs.net has been updated to develop branch, but the issue still persists. Are [more users](https://pleroma.envs.net/notice/ATz3IKS3YtJjF2eBbU) are checking in and saying it isn't working.

floatingghost commented

2023-03-27 03:02:42 +00:00

Owner

I'm loath to repeat myself but without logs or you debugging there is quite literally nothing anyone can do to help you

nobody here is psychic and we cannot converse with computers

I'm loath to repeat myself but without logs or you debugging there is quite literally nothing anyone can do to help you nobody here is psychic and we cannot converse with computers

creme commented

2023-03-27 09:54:29 +00:00

hi,

i have here a collection of my logs and the akkoma config. Maybe it helps to find the issue.

FYI: the server runs on the current "develop"-branch

hi, i have here a collection of my logs and the akkoma config. Maybe it helps to find the issue. FYI: the server runs on the current "develop"-branch

postgresql.log

11 MiB

pleroma.log

255 KiB

config.exs

4.6 KiB

floatingghost commented

2023-03-27 10:02:45 +00:00

Owner

wowee these config deprecation warnings have been shouting at you for about 3 years now

please for all that is holy fix those

wowee these config deprecation warnings have been shouting at you for about 3 years now please for all that is holy fix those

floatingghost commented

2023-03-27 10:05:22 +00:00

Owner

config :pleroma, :dangerzone, override_repo_pool_size: true

never override this, this is likely the source of your database issues

>config :pleroma, :dangerzone, override_repo_pool_size: true _never_ override this, this is likely the source of your database issues

creme commented

2023-03-27 10:27:02 +00:00

i update my config..

Mär 27 10:09:52 pleroma.envs.net pleroma[15415]: 10:09:52.932 [warning] !!!DEPRECATION WARNING!!!
Mär 27 10:09:52 pleroma.envs.net pleroma[15415]: Your config is using old namespaces for MRF configuration. They should work for now, but you are advised to change to new namespaces to prevent possible issues later:
Mär 27 10:09:52 pleroma.envs.net pleroma[15415]: * `config :pleroma, :instance, rewrite_policy` is now `config :pleroma, :mrf, policies`

but i can't find this in the file.

i update my config.. ``` Mär 27 10:09:52 pleroma.envs.net pleroma[15415]: 10:09:52.932 [warning] !!!DEPRECATION WARNING!!! Mär 27 10:09:52 pleroma.envs.net pleroma[15415]: Your config is using old namespaces for MRF configuration. They should work for now, but you are advised to change to new namespaces to prevent possible issues later: Mär 27 10:09:52 pleroma.envs.net pleroma[15415]: * `config :pleroma, :instance, rewrite_policy` is now `config :pleroma, :mrf, policies` ``` but i can't find this in the file.

config.exs

4.4 KiB

pleroma.log

231 KiB

floatingghost commented

2023-03-27 10:27:30 +00:00

Owner

it's in your database

creme commented

2023-03-27 10:32:36 +00:00

hmm can't find it.

floatingghost commented

2023-03-27 10:33:03 +00:00

Owner

well it's either in your database or in your config. it's in one of the two.

creme commented

2023-03-27 10:41:31 +00:00

any idea how i can find it?

floatingghost commented

2023-03-27 10:43:03 +00:00

Owner

https://man7.org/linux/man-pages/man1/grep.1.html

https://docs.akkoma.dev/stable/administration/CLI_tasks/config/

https://man7.org/linux/man-pages/man1/grep.1.html https://docs.akkoma.dev/stable/administration/CLI_tasks/config/

creme commented

2023-03-27 10:47:00 +00:00

looks like it is in the DB. but no idea how to change it..

$ su pleroma -s $SHELL -lc "./bin/pleroma_ctl config dump" | grep rewrite_policy
config :pleroma, :instance, [email: "hostmaster@envs.net", notify_email: "noreply@envs.net", limit: 10000, registrations_open: true, account_activation_required: true, invites_enabled: false, rewrite_policy: [Pleroma.Web.ActivityPub.MRF.SimplePolicy], max_pinned_statuses: 5, max_report_comment_size: 10000, healthcheck: true, upload_limit: 33554432, avatar_upload_limit: 33554432, background_upload_limit: 33554432, banner_upload_limit: 33554432, name: "Akkoma on envs.net", description: "A Pleroma/Akkoma instance on envs.net, an alternative fediverse server.", account_approval_required: true]

can't find it in the admin panel.

looks like it is in the DB. but no idea how to change it.. ``` $ su pleroma -s $SHELL -lc "./bin/pleroma_ctl config dump" | grep rewrite_policy config :pleroma, :instance, [email: "hostmaster@envs.net", notify_email: "noreply@envs.net", limit: 10000, registrations_open: true, account_activation_required: true, invites_enabled: false, rewrite_policy: [Pleroma.Web.ActivityPub.MRF.SimplePolicy], max_pinned_statuses: 5, max_report_comment_size: 10000, healthcheck: true, upload_limit: 33554432, avatar_upload_limit: 33554432, background_upload_limit: 33554432, banner_upload_limit: 33554432, name: "Akkoma on envs.net", description: "A Pleroma/Akkoma instance on envs.net, an alternative fediverse server.", account_approval_required: true] ``` can't find it in the admin panel.

floatingghost commented

2023-03-27 10:47:43 +00:00

Owner

please read the documentation page linked above

creme commented

2023-03-27 10:53:12 +00:00

i see no way to change this entry.

floatingghost commented

2023-03-27 10:55:12 +00:00

Owner

![image](/attachments/26d6a5c4-318c-4bd3-ba06-04647442ad67)

image.png

272 KiB

creme commented

2023-03-27 11:06:48 +00:00

i try this:

su pleroma -s $SHELL -lc "./bin/pleroma_ctl config dump_to_file pleroma instance instance.json"
modify the instance.json and remove the rewrite_policy tuple
su pleroma -s $SHELL -lc "./bin/pleroma_ctl config load_from_file instance.json"
restart the instance

but the entry is still present

i try this: - `su pleroma -s $SHELL -lc "./bin/pleroma_ctl config dump_to_file pleroma instance instance.json"` - modify the instance.json and remove the rewrite_policy tuple - `su pleroma -s $SHELL -lc "./bin/pleroma_ctl config load_from_file instance.json"` - restart the instance but the entry is still present

floatingghost commented

2023-03-27 11:07:25 +00:00

Owner

then remove it and reload it?

creme commented

2023-03-27 11:11:09 +00:00

i would like to, but the question is how

floatingghost closed this issue

2023-03-27 11:11:27 +00:00

floatingghost added the

Fix it yourself

label 2023-03-27 11:11:32 +00:00

creme commented

2023-03-27 11:16:29 +00:00

it is a pity that there is no interest in explaining to people who do not understand your system how it works.

floatingghost commented

2023-03-27 11:19:12 +00:00

Owner

you were linked the documentation multiple times, free tech support only goes so far without you taking initiative.

the first time was ok, the second was annoying and the third was taking the mickey more than a little.

running an instance comes with it the tacit understanding that you are capable of managing your own system, and in your case, setting danger zone keys is also agreeing that you understand what you are doing since it is non-standard.

i cannot fix your instance remotely, you have all the resources you need.

you were linked the documentation multiple times, free tech support only goes so far without you taking initiative. the first time was ok, the second was annoying and the third was taking the mickey more than a little. running an instance comes with it the tacit understanding that you are capable of managing your own system, and in your case, setting `danger zone` keys is also agreeing that you understand what you are doing since it is non-standard. i cannot fix your instance remotely, you have all the resources you need.

creme commented

2023-03-27 11:26:19 +00:00

i try now this way:

su pleroma -s $SHELL -lc "./bin/pleroma_ctl config dump_to_file pleroma instance instance.json"
modify the instance.json and remove the rewrite_policy tuple
su pleroma -s $SHELL -lc "./bin/pleroma_ctl config delete pleroma instance"
su pleroma -s $SHELL -lc "./bin/pleroma_ctl config load_from_file instance.json"
restart the instance

it looks like it works. the rewrite_policy is now removed.

the current log is attached..

i try now this way: - `su pleroma -s $SHELL -lc "./bin/pleroma_ctl config dump_to_file pleroma instance instance.json"` - modify the instance.json and remove the rewrite_policy tuple - `su pleroma -s $SHELL -lc "./bin/pleroma_ctl config delete pleroma instance"` - `su pleroma -s $SHELL -lc "./bin/pleroma_ctl config load_from_file instance.json"` - restart the instance it looks like it works. the rewrite_policy is now removed. the current log is attached..

pleroma.log

51 KiB

postgresql.log

11 MiB

creme commented

2023-03-27 12:04:11 +00:00

what i notice is that my home timeline with mastodon-fe loads but not with the pleroma-fe. the "know network" tab load nothing in both fe's.

creme commented

2023-03-27 13:17:53 +00:00

now after ~1 hour, the home timeline has re-operated itself in the pleroma-fe and shows something again. The "known network" tab continues to throw errors.

creme commented

2023-03-27 20:34:14 +00:00

@floatingghost can we reopen the issue please?

floatingghost commented

2023-03-29 13:32:56 +00:00

Owner

this is the last thing it is possible for me to do remotely

https://akkoma.dev/AkkomaGang/akkoma/src/branch/develop/lib/mix/tasks/pleroma/diagnostics.ex#L102

this is the last thing it is possible for me to do remotely https://akkoma.dev/AkkomaGang/akkoma/src/branch/develop/lib/mix/tasks/pleroma/diagnostics.ex#L102

creme commented

2023-03-29 13:55:32 +00:00

okay, thanks.. i just updated and ran the task. here is the output:

https://envs.sh/hcP.txt

okay, thanks.. i just updated and ran the task. here is the output: https://envs.sh/hcP.txt

floatingghost commented

2023-03-29 14:18:43 +00:00

Owner

make sure your debug log is on - it will tell you if it gets stuck rendering a specific status

creme commented

2023-03-29 14:32:36 +00:00

i change the logging now from info to debug..

the diagnostic output looks okay i think..
https://envs.sh/hcw.txt

and this is the pleroma log:
https://envs.sh/hcq.log

i change the logging now from info to debug.. the diagnostic output looks okay i think.. https://envs.sh/hcw.txt and this is the pleroma log: https://envs.sh/hcq.log

floatingghost commented

2023-03-29 14:56:11 +00:00

Owner

it's 100% your backend

the idle time on your transactions is worrying me - you may be being bottlenecked by your IO or database

/phoenix/live_dashboard/ecto_stats should be able to tell you if there's anything it automatically detects as being concerning

it's 100% your backend the idle time on your transactions is worrying me - you may be being bottlenecked by your IO or database `/phoenix/live_dashboard/ecto_stats` should be able to tell you if there's anything it automatically detects as being concerning

creme commented

2023-03-29 15:03:46 +00:00

hmm so resources technically, there should actually be no problems. I'm using an AMD Ryzen 9 5950X 16-core processor. The VM has 16GB RAM of which hardly anything is used and an ssd nvme is installed as a storage medium.

hmm so resources technically, there should actually be no problems. I'm using an AMD Ryzen 9 5950X 16-core processor. The VM has 16GB RAM of which hardly anything is used and an ssd nvme is installed as a storage medium. ![image](/attachments/ab959867-8edb-4dc1-8917-d0158f7ac87c) ![image](/attachments/ca04f05d-c9bf-4857-a5a0-2f2b1e0741b8) ![image](/attachments/669ff33d-e5c9-4440-8959-a81999de0a49) ![image](/attachments/6e8959f6-254f-4212-be15-99125413e83f)

image.png

254 KiB

image.png

134 KiB

image.png

199 KiB

image.png

107 KiB

floatingghost commented

2023-03-30 07:08:55 +00:00

Owner

it appears that your endpoint is indeed working again, it is returning statuses as expected

creme commented

2023-03-30 10:23:46 +00:00

hi, so there is something interesting. I tested everything possible yesterday. Everything worked until the loading of the "known network" timeline.

I optimize my postgresql config with https://pgtune.leopard.in.ua / , gave the machine more RAM and optimized the DB itself:

su pleroma -s $SHELL -lc "./bin/pleroma_ctl database prune_objects --keep-threads --keep-non-public --prune-orphaned-activities"
and
REINDEX (VERBOSE) DATABASE CONCURRENTLY pleroma;

this, at least, obviously, did not improve anything. The timeline still could not be loaded.

But when I log out now, I see the "known network" timeline as an visitor and it shows posts. But the public timeline looks near empty. Which, however, works in the logged-in state.

When I log in again as a user creme, all timelines load without problems and show me all posts and also newly arriving ones.
But as soon as I reload the tab in the browser (via F5) I get the error message "Error fetching timeline: Unexpected status: 502" again and the known network timeline does not load again :/

I can recreate the behavior with 3 accounts that I manage. Both in Firefox and in the google-chrome browser.

i test it with "rich_media" and "media_proxy" enabled and disabled. it makes no difference.

vistor view (public timeline):

visitor view (known network timeline):

fresh login view (known network):

reload tab (known network):

hi, so there is something interesting. I tested everything possible yesterday. Everything worked until the loading of the "known network" timeline. I optimize my postgresql config with https://pgtune.leopard.in.ua / , gave the machine more RAM and optimized the DB itself: `su pleroma -s $SHELL -lc "./bin/pleroma_ctl database prune_objects --keep-threads --keep-non-public --prune-orphaned-activities"` and `REINDEX (VERBOSE) DATABASE CONCURRENTLY pleroma;` this, at least, obviously, did not improve anything. The timeline still could not be loaded. But when I log out now, I see the "known network" timeline as an visitor and it shows posts. But the public timeline looks near empty. Which, however, works in the logged-in state. When I log in again as a user creme, all timelines load without problems and show me all posts and also newly arriving ones. But as soon as I reload the tab in the browser (via F5) I get the error message "Error fetching timeline: Unexpected status: 502" again and the known network timeline does not load again :/ I can recreate the behavior with 3 accounts that I manage. Both in Firefox and in the google-chrome browser. i test it with "rich_media" and "media_proxy" enabled and disabled. it makes no difference. vistor view (public timeline): ![image](/attachments/0f0ef0b1-9dde-46f5-a654-733955e1b70b) visitor view (known network timeline): ![image](/attachments/af02a3e9-c3fc-4f61-b148-1772d48d310d) fresh login view (known network): ![image](/attachments/9ea2a00c-da30-4859-974b-a0c404aef1d3) reload tab (known network): ![image](/attachments/5c045a85-6ac9-4463-bf4c-3a44e1b992d6)

image.png

1.1 MiB

image.png

1.3 MiB

image.png

601 KiB

image.png

813 KiB