AkkomaGang/akkoma
AkkomaGang/akkoma
15
65
Fork 92
Code Issues 177 Pull requests 24 Projects 2 Releases 14 Packages Wiki Activity

[bug] Unusually large amount of "Object has been deleted" errors #355

New issue
Closed
opened 2022-12-09 07:54:10 +00:00 by Ghost · 7 comments
Ghost commented 2022-12-09 07:54:10 +00:00
Copy link

Your setup

Source

Extra details

Arch Linux amd64

Version

3.4.0-194-g59fde45b-develop

PostgreSQL version

14.6

What were you trying to do?

n/a

What did you expect to happen?

Don't think this should be semi-spammed in my logs every few seconds as this seems to be a new issue.

What actually happened?

Noticed in Akkoma logs a significant amount of [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} being logged every few seconds for what seems like hours on end?

Memory and CPU usage seems fine, I don't do fancy monitoring so I can't tell if I see any kind of spikes. Postgres logs don't report any errors or anything concerning associated as far as I can tell.

Restarting Akkoma doesn't seem to reliably solve this weird issue.

Logs

# (snippet)

Dec 09 07:19:47 plem.sapphic.site pleroma[18015]: 07:19:47.576 request_id=Fy8O6pSWOROx39wAGoUx [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:19:47 plem.sapphic.site pleroma[18015]: 07:19:47.698 request_id=Fy8O6pSWOROx39wAGoUx [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:19:48 plem.sapphic.site pleroma[18015]: 07:19:48.023 request_id=Fy8O6piUDdsxAMQAGoVB [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:19:48 plem.sapphic.site pleroma[18015]: 07:19:48.165 request_id=Fy8O6piUDdsxAMQAGoVB [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:20:07 plem.sapphic.site pleroma[18015]: 07:20:07.604 request_id=Fy8O7z3TVH43cPYAGoXR [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:20:07 plem.sapphic.site pleroma[18015]: 07:20:07.734 request_id=Fy8O7z3TVH43cPYAGoXR [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:20:13 plem.sapphic.site pleroma[18015]: 07:20:13.088 request_id=Fy8O8IUeTioVz1UAGoaB [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:20:13 plem.sapphic.site pleroma[18015]: 07:20:13.209 request_id=Fy8O8IUeTioVz1UAGoaB [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:20:33 plem.sapphic.site pleroma[18015]: 07:20:33.630 request_id=Fy8O9U2PT1Yl9pYAGodR [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:20:33 plem.sapphic.site pleroma[18015]: 07:20:33.751 request_id=Fy8O9U2PT1Yl9pYAGodR [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:20:35 plem.sapphic.site pleroma[18015]: 07:20:35.270 request_id=Fy8O9a7t-XjiohkAGodh [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:20:35 plem.sapphic.site pleroma[18015]: 07:20:35.401 request_id=Fy8O9a7t-XjiohkAGodh [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:20:40 plem.sapphic.site pleroma[18015]: 07:20:40.379 request_id=Fy8O9sDgehtIk04AGodx [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:20:40 plem.sapphic.site pleroma[18015]: 07:20:40.519 request_id=Fy8O9sDgehtIk04AGodx [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:20:54 plem.sapphic.site pleroma[18015]: 07:20:54.992 request_id=Fy8O-kbbq0nHHREAGoix [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:20:55 plem.sapphic.site pleroma[18015]: 07:20:55.114 request_id=Fy8O-kbbq0nHHREAGoix [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:20:56 plem.sapphic.site pleroma[18015]: 07:20:56.642 request_id=Fy8O-pbDNh9F0-YAGojB [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:20:56 plem.sapphic.site pleroma[18015]: 07:20:56.791 request_id=Fy8O-pbDNh9F0-YAGojB [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:21:13 plem.sapphic.site pleroma[18015]: 07:21:13.343 request_id=Fy8O_oypjAJa-mcAGorx [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:21:13 plem.sapphic.site pleroma[18015]: 07:21:13.483 request_id=Fy8O_oypjAJa-mcAGorx [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:21:16 plem.sapphic.site pleroma[18015]: 07:21:16.111 request_id=Fy8O_y8LMTQ-X0MAGosB [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:21:16 plem.sapphic.site pleroma[18015]: 07:21:16.258 request_id=Fy8O_y8LMTQ-X0MAGosB [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:21:19 plem.sapphic.site pleroma[18015]: 07:21:19.593 request_id=Fy8PAAE3i9eS8xkAGosh [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:21:19 plem.sapphic.site pleroma[18015]: 07:21:19.733 request_id=Fy8PAAE3i9eS8xkAGosh [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:21:20 plem.sapphic.site pleroma[18015]: 07:21:20.258 request_id=Fy8PACiC1oLFx9kAGosx [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:21:20 plem.sapphic.site pleroma[18015]: 07:21:20.380 request_id=Fy8PACiC1oLFx9kAGosx [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:21:22 plem.sapphic.site pleroma[18015]: 07:21:22.033 request_id=Fy8PAJIz6KgrzoMAGotB [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:21:22 plem.sapphic.site pleroma[18015]: 07:21:22.164 request_id=Fy8PAJIz6KgrzoMAGotB [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:21:23 plem.sapphic.site pleroma[18015]: 07:21:23.143 request_id=Fy8PANQqwTixmokAGouB [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:21:23 plem.sapphic.site pleroma[18015]: 07:21:23.281 request_id=Fy8PANQqwTixmokAGouB [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:21:23 plem.sapphic.site pleroma[18015]: 07:21:23.834 request_id=Fy8PAONeAPMNLd4AGouR [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}
Dec 09 07:21:24 plem.sapphic.site pleroma[18015]: 07:21:24.014 request_id=Fy8PAONeAPMNLd4AGouR [error] Could not fetch user, {nil, {:error, "Object has been deleted"}}

Severity

I can manage

Have you searched for this issue?

  • I have double-checked and have not found this issue mentioned anywhere.
### Your setup Source ### Extra details Arch Linux amd64 ### Version 3.4.0-194-g59fde45b-develop ### PostgreSQL version 14.6 ### What were you trying to do? n/a ### What did you expect to happen? Don't think this should be semi-spammed in my logs every few seconds as this seems to be a new issue. ### What actually happened? Noticed in Akkoma logs a significant amount of `[error] Could not fetch user, {nil, {:error, "Object has been deleted"}}` being logged every few seconds for what seems like hours on end? Memory and CPU usage seems fine, I don't do fancy monitoring so I can't tell if I see any kind of spikes. Postgres logs don't report any errors or anything concerning associated as far as I can tell. Restarting Akkoma doesn't seem to reliably solve this weird issue. ### Logs ```shell # (snippet) Dec 09 07:19:47 plem.sapphic.site pleroma[18015]: 07:19:47.576 request_id=Fy8O6pSWOROx39wAGoUx [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:19:47 plem.sapphic.site pleroma[18015]: 07:19:47.698 request_id=Fy8O6pSWOROx39wAGoUx [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:19:48 plem.sapphic.site pleroma[18015]: 07:19:48.023 request_id=Fy8O6piUDdsxAMQAGoVB [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:19:48 plem.sapphic.site pleroma[18015]: 07:19:48.165 request_id=Fy8O6piUDdsxAMQAGoVB [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:20:07 plem.sapphic.site pleroma[18015]: 07:20:07.604 request_id=Fy8O7z3TVH43cPYAGoXR [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:20:07 plem.sapphic.site pleroma[18015]: 07:20:07.734 request_id=Fy8O7z3TVH43cPYAGoXR [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:20:13 plem.sapphic.site pleroma[18015]: 07:20:13.088 request_id=Fy8O8IUeTioVz1UAGoaB [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:20:13 plem.sapphic.site pleroma[18015]: 07:20:13.209 request_id=Fy8O8IUeTioVz1UAGoaB [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:20:33 plem.sapphic.site pleroma[18015]: 07:20:33.630 request_id=Fy8O9U2PT1Yl9pYAGodR [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:20:33 plem.sapphic.site pleroma[18015]: 07:20:33.751 request_id=Fy8O9U2PT1Yl9pYAGodR [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:20:35 plem.sapphic.site pleroma[18015]: 07:20:35.270 request_id=Fy8O9a7t-XjiohkAGodh [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:20:35 plem.sapphic.site pleroma[18015]: 07:20:35.401 request_id=Fy8O9a7t-XjiohkAGodh [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:20:40 plem.sapphic.site pleroma[18015]: 07:20:40.379 request_id=Fy8O9sDgehtIk04AGodx [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:20:40 plem.sapphic.site pleroma[18015]: 07:20:40.519 request_id=Fy8O9sDgehtIk04AGodx [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:20:54 plem.sapphic.site pleroma[18015]: 07:20:54.992 request_id=Fy8O-kbbq0nHHREAGoix [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:20:55 plem.sapphic.site pleroma[18015]: 07:20:55.114 request_id=Fy8O-kbbq0nHHREAGoix [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:20:56 plem.sapphic.site pleroma[18015]: 07:20:56.642 request_id=Fy8O-pbDNh9F0-YAGojB [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:20:56 plem.sapphic.site pleroma[18015]: 07:20:56.791 request_id=Fy8O-pbDNh9F0-YAGojB [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:21:13 plem.sapphic.site pleroma[18015]: 07:21:13.343 request_id=Fy8O_oypjAJa-mcAGorx [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:21:13 plem.sapphic.site pleroma[18015]: 07:21:13.483 request_id=Fy8O_oypjAJa-mcAGorx [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:21:16 plem.sapphic.site pleroma[18015]: 07:21:16.111 request_id=Fy8O_y8LMTQ-X0MAGosB [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:21:16 plem.sapphic.site pleroma[18015]: 07:21:16.258 request_id=Fy8O_y8LMTQ-X0MAGosB [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:21:19 plem.sapphic.site pleroma[18015]: 07:21:19.593 request_id=Fy8PAAE3i9eS8xkAGosh [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:21:19 plem.sapphic.site pleroma[18015]: 07:21:19.733 request_id=Fy8PAAE3i9eS8xkAGosh [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:21:20 plem.sapphic.site pleroma[18015]: 07:21:20.258 request_id=Fy8PACiC1oLFx9kAGosx [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:21:20 plem.sapphic.site pleroma[18015]: 07:21:20.380 request_id=Fy8PACiC1oLFx9kAGosx [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:21:22 plem.sapphic.site pleroma[18015]: 07:21:22.033 request_id=Fy8PAJIz6KgrzoMAGotB [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:21:22 plem.sapphic.site pleroma[18015]: 07:21:22.164 request_id=Fy8PAJIz6KgrzoMAGotB [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:21:23 plem.sapphic.site pleroma[18015]: 07:21:23.143 request_id=Fy8PANQqwTixmokAGouB [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:21:23 plem.sapphic.site pleroma[18015]: 07:21:23.281 request_id=Fy8PANQqwTixmokAGouB [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:21:23 plem.sapphic.site pleroma[18015]: 07:21:23.834 request_id=Fy8PAONeAPMNLd4AGouR [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} Dec 09 07:21:24 plem.sapphic.site pleroma[18015]: 07:21:24.014 request_id=Fy8PAONeAPMNLd4AGouR [error] Could not fetch user, {nil, {:error, "Object has been deleted"}} ``` ### Severity I can manage ### Have you searched for this issue? - [x] I have double-checked and have not found this issue mentioned anywhere.
Ghost commented 2022-12-10 06:16:49 +00:00
Author
Copy link

Following the commits that provide more logging in this case, it seems that all of these deleted objects are deleted (?) accounts?

Paste because too much text wrapping: https://paste.rs/YAj

All of the said accounts in every log of this error leads to a 404.

But I have no idea why this is happening so much every few seconds?

Also side note: I switched to source install, still on latest develop branch.

Following the commits that provide more logging in this case, it seems that all of these deleted objects are deleted (?) accounts? Paste because too much text wrapping: https://paste.rs/YAj All of the said accounts in *every* log of this error leads to a 404. But I have no idea why this is happening so much every few seconds? Also side note: I switched to source install, still on latest develop branch.
floatingghost commented 2022-12-14 02:53:12 +00:00
Owner
Copy link

is that the right link? it goes to some datastudio stuff

is that the right link? it goes to some datastudio stuff
Ghost commented 2022-12-14 03:20:55 +00:00
Author
Copy link

@floatingghost Looks like the paste expired and the paste ID got taken by something else.

Here's a new paste from a more reliable site: 0ea80966b1/gistfile1.txt

@floatingghost Looks like the paste expired and the paste ID got taken by something else. Here's a new paste from a more reliable site: https://gist.githubusercontent.com/r3g-5z/bc21259150ce5de9fb37f290044435f9/raw/0ea80966b126f02285ff35b990dc8d1607645e3d/gistfile1.txt
floatingghost commented 2022-12-14 10:06:42 +00:00
Owner
Copy link

that looks normal

mastodon[.]technology has gone offline, so after a few days your instance should notice that and stop bothering them

otherwise nothing seems too amiss there

that looks normal mastodon[.]technology has gone offline, so after a few days your instance should notice that and stop bothering them otherwise nothing seems too amiss there
Ghost commented 2022-12-14 13:34:42 +00:00
Author
Copy link

Is it really normal for "Object has been deleted" to be nearly spammed in my server logs? I get this message almost every 3 seconds.

Is it really normal for "Object has been deleted" to be nearly spammed in my server logs? I get this message almost every 3 seconds.
floatingghost commented 2022-12-14 13:35:37 +00:00
Owner
Copy link

yes

yes
Ghost commented 2022-12-14 13:37:58 +00:00
Author
Copy link

Guess I'll just have to ignore this and filter out these messages when I need to get logs.

Guess I'll just have to ignore this and filter out these messages when I need to get logs.
Ghost closed this issue 2022-12-14 13:37:58 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
develop
translations
customizable-docker-db
stable
static-adminfe
bad-uploader-config-warning
docker
user-expiry-backfill
config-db-keys
ensure-scrubbers-loaded
otp26
code-shaking-does-not-go-brr
elixir1.15
circleci
backup-fixes
mod-task
policy
mrf-coolness
frontend-switcher-9000
contentMap
language-on-posts
rabbit
credo-on-pr
unify-http
normalise-markup-by-default
buildx
v3.13.2
v2.6.3
v3.13.1
v3.13.0
snac-3.12.1.2
v3.12.2
snac-3.12.1.1
v3.12.1
v3.12.0
v3.11.0
v2.6.2
v2.6.1
2.6.1
v2.6.0
v2.5.5
v3.10.4
v3.10.3
v3.10.2
v3.10.1
v3.10.0
v2.5.4
v2.5.3
v3.9.3
v3.9.2
v3.9.1
V3.9.0
2023.05
v3.8.0
v3.7.1
v3.7.0
v3.6.0
v3.5.0
v2.4.5
v3.4.0
v3.3.1
v2.4.4
v3.3.0
v3.2.3
v3.2.2
v3.2.1
stable-202209
v3.2.0
v3.1.0
stable-2022.07
v3.0.0
v0.0.1
v2.5.2-6
v2.5.2-5
v2.5.2-4
v2.5.2-3
v2.5.2-2
v2.5.2-1
v2.5.2
v2.5.1
v2.5.0-8
v2.5.0-7
v2.5.0-6
v2.5.0-5
v2.5.0-4
v2.5.0-3
v2.5.0-2
v2.5.0-1
v2.5.0
v2.4.3
pre-rebase
v2.4.2
v2.4.1
v2.4.0
soapbox-v1.1.1
soapbox-v1.1.0
soapbox-v1.0.0
v2.3.0
v2.2.2
v2.2.1
v2.2.0
v2.1.2
v2.1.1
v2.1.0
v2.0.7
v2.0.6
v2.0.5
v2.0.4
v2.0.3
v2.0.2
v2.0.1
v2.0.0
v1.1.9
v1.1.8
v1.1.7
v1.1.6
v1.1.5
v1.1.4
v1.1.3
v1.1.2
v1.1.1
v1.1.0
v1.0.91
v1.0.90
v1.0.7
v1.0.6
v1.0.5
v1.0.4
v1.0.3
v1.0.2
v1.0.1
v1.0.0
v0.9.0
Labels
Clear labels   
approved, awaiting change

   
bug

Something is not working

  
configuration

This requires config changes

  
documentation

This is about human-readable docs

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
extremely low priority

   
feature request

Something new?

  
Fix it yourself

Support has been attempted

  
help wanted

Need some help

  
invalid

Something is wrong

  
mastodon_api

   
needs docs

   
needs tests

   
not a bug

   
planned

   
pleroma_api

   
privacy

   
question

More information is needed

  
static_fe

   
triage

   
wontfix

This won't be fixed

No labels
approved, awaiting change
bug
configuration
documentation
duplicate
enhancement
extremely low priority
feature request
Fix it yourself
help wanted
invalid
mastodon_api
needs docs
needs tests
not a bug
planned
pleroma_api
privacy
question
static_fe
triage
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: AkkomaGang/akkoma#355
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?