AkkomaGang/akkoma
AkkomaGang/akkoma
16
67
Fork 105
Code Issues 167 Pull requests 13 Projects 2 Releases 19 Packages Wiki Activity

Expose stats about finally failed AP deliveries in prometheus #882

Merged
Oneric merged 2 commits from Oneric/akkoma:telemetry-failed-deliveries into develop 2025-05-09 20:12:02 +00:00
Conversation 1 Commits 2 Files changed 6 +97 -17
Oneric commented 2025-03-14 19:43:05 +00:00
Owner
Copy link

And log about it. Makes it easier to tell when and what's going wrong.

If anyone knows how to make Grafana show this as a bar chart with one bar per target with bars segmented by reason pls let me know.
I only managed to get a bar chart over target loosing reason info

The oban event types and states per failure/return type are:

fail/return event final state retry state
:ok :stop :success
{:cancel, _} :stop :cancelled
{:discard, _} :stop :discard
{:error, _} :exception :discard :failure
exception :exception :discard :failure
And log about it. Makes it easier to tell when and what's going wrong. If anyone knows how to make Grafana show this as a bar chart with one bar per `target` with bars segmented by `reason` pls let me know. I only managed to get a bar chart over `target` loosing `reason` info The oban event types and states per failure/return type are: | **fail/return** | event | final state | retry state | | ------------------- | --------- | ------------- | --------------- | | `:ok` | `:stop` | `:success` | | | `{:cancel, _}` | `:stop` | `:cancelled` | | | `{:discard, _}` | `:stop` | `:discard` | | | `{:error, _}` | `:exception` | `:discard` | `:failure` | | exception | `:exception` | `:discard` | `:failure` |
Oneric added 2 commits 2025-03-14 19:43:05 +00:00 
:discard marks jobs as "discarded", i.e. jobs which permanently failed
due to e.g. exhausting all retries or explicitly being discared due to a
fatal error.
:cancel marks jobs as "cancelled" which does not imply failure.

While neither method counts as a job "exception" in the set of
telemetries we currently export via Prometheus, the different state
is visible in the (not-exported) metadata of oban job telemetry.
We can use handlers of those events to build bespoke statistics.

Ideally we'd like to distinguish in the receiver worker between
"invalid" and "already present or delete of unknown" documents,
but this is cumbersome to get get right with a list of
free-form, human-readable descriptions oof the violated constraints.
For now, just count both as an fatal error.
        # but that is cumbersome to get right with a list of string error descriptions
telemetry: expose stats about failed deliveries
Some checks are pending
ci/woodpecker/pr/build-amd64 Pipeline is pending approval
Details
ci/woodpecker/pr/build-arm64 Pipeline is pending approval
Details
ci/woodpecker/pr/docs Pipeline is pending approval
Details
ci/woodpecker/pr/lint Pipeline is pending approval
Details
ci/woodpecker/pr/test/1 Pipeline is pending approval
Details
ci/woodpecker/pr/test/2 Pipeline is pending approval
Details
249876d1f0 
And also log about it which we so far didn't do
Oneric force-pushed telemetry-failed-deliveries from 249876d1f0 to 1f6f5edf85 2025-04-15 17:41:33 +00:00 Compare
Oneric commented 2025-04-15 17:42:13 +00:00
Author
Owner
Copy link

rebased and cosmetic changes for elixir-1.18’s mix format

rebased and cosmetic changes for elixir-1.18’s `mix format`
Oneric merged commit 13940a558a into develop 2025-05-09 20:12:02 +00:00
Oneric deleted branch telemetry-failed-deliveries 2025-05-09 20:12:02 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
approved, awaiting change

   
bug

Something is not working

  
configuration

This requires config changes

  
documentation

This is about human-readable docs

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
extremely low priority

   
feature request

Something new?

  
Fix it yourself

Support has been attempted

  
help wanted

Need some help

  
invalid

Something is wrong

  
mastodon_api

   
needs change/feedback

Issue or PR is stalled on response from the author or a third party

  
needs docs

   
needs tests

   
not a bug

   
planned

   
pleroma_api

   
privacy

   
question

More information is needed

  
static_fe

   
triage

   
wontfix

This won't be fixed

No labels
approved, awaiting change
bug
configuration
documentation
duplicate
enhancement
extremely low priority
feature request
Fix it yourself
help wanted
invalid
mastodon_api
needs change/feedback
needs docs
needs tests
not a bug
planned
pleroma_api
privacy
question
static_fe
triage
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: AkkomaGang/akkoma#882
No description provided.
Delete branch "Oneric/akkoma:telemetry-failed-deliveries"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?