[feat] Use Accept-Encoding to allow data compression #755

New issue

Closed

opened 2024-04-22 11:06:17 +00:00 by mboelen · 2 comments

mboelen commented

2024-04-22 11:06:17 +00:00

The idea

As a webmaster, I see Akkoma in my logs and see it does not use data compression (accept-encoding header). All modern HTTP clients support this feature, so it is common to also support this on the server side.

In our case we recently decided to block all requests with HTTP/1.0 and those that are not offering to use data encoding. Due to that, we saw a few RSS feed readers showing up, and Akkoma.

Example from log (ip and instance masked), the 426 error means it is blocked:

2024-04-21T20:24:21+00:00 426 1.2.3.4 "GET /cheat-sheets/curl/ HTTP/1.1" 16 "-" "Akkoma 3.12.2; https://masked <masked>" TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 .

I did a quick look in the code, but as I'm not familiar with it, I couldn't find the HTTP handler.

My suggestion would be to enable at least Gzip, but preferably Brotli (better savings) and possibly zstd (not as common, yet).

Thanks for your consideration!

The reasoning

Save data traffic, which is beneficial for everyone involved (data centers, publishers, and clients). Especially as this involves every single HTTP request the software performs, it will add up over time considering that on a HTML/CSS/SVG/JS often the savings are great.

Have you searched for this feature request?

I have double-checked and have not found this feature request mentioned anywhere.
This feature is related to the Akkoma backend specifically, and not pleroma-fe.

### The idea As a webmaster, I see Akkoma in my logs and see it does not use data compression (accept-encoding header). All modern HTTP clients support this feature, so it is common to also support this on the server side. In our case we recently decided to block all requests with HTTP/1.0 and those that are not offering to use data encoding. Due to that, we saw a few RSS feed readers showing up, and Akkoma. Example from log (ip and instance masked), the 426 error means it is blocked: `2024-04-21T20:24:21+00:00 426 1.2.3.4 "GET /cheat-sheets/curl/ HTTP/1.1" 16 "-" "Akkoma 3.12.2; https://masked <masked>" TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 .` I did a quick look in the code, but as I'm not familiar with it, I couldn't find the HTTP handler. My suggestion would be to enable at least Gzip, but preferably Brotli (better savings) and possibly zstd (not as common, yet). Thanks for your consideration! ### The reasoning Save data traffic, which is beneficial for everyone involved (data centers, publishers, and clients). Especially as this involves every single HTTP request the software performs, it will add up over time considering that on a HTML/CSS/SVG/JS often the savings are great. ### Have you searched for this feature request? - [x] I have double-checked and have not found this feature request mentioned anywhere. - [ ] This feature is related to the Akkoma backend specifically, and not pleroma-fe.

mboelen added the

feature request

label

2024-04-22 11:06:17 +00:00

Oneric commented

2025-10-10 15:41:01 +00:00

Owner

In theory this should be as easy as adding Tesla.Middleware.DecompressResponse near the end of our Tesla middleware list exepct for HEAD requests (only followed by the telemetry middleware). However, the middleware is buggy in the Tesla version we currently use and crashes on responses with an empty body like below. A fix was already merged upstream and is included in the latest Tesla release though.

2025-10-09 23:49:01.991 [error] Failed to fetch https://www.litepc.com/98lite.html: ** (ErlangError) Erlang error: :data_error
    :zlib.inflateEnd_nif(#Reference<0.113196428.1107689474.136005>)
    :zlib.gunzip/1
    (tesla 1.14.1) lib/tesla/middleware/compression.ex:88: Tesla.Middleware.Compression.decompress_body/3
    (tesla 1.14.1) lib/tesla/middleware/compression.ex:72: Tesla.Middleware.Compression.decompress/1
    (tesla 1.14.1) lib/tesla/middleware/compression.ex:67: Tesla.Middleware.Compression.decompress/1
    (tesla 1.14.1) lib/tesla/middleware/follow_redirects.ex:49: Tesla.Middleware.FollowRedirects.redirect/3
    (pleroma 3.15.2-wf-121-gcf81c6e4-u24d828d9a-trunk) lib/pleroma/http/logger_middleware.ex:18: Pleroma.HTTP.LoggerMiddleware.call/3
    (pleroma 3.15.2-wf-121-gcf81c6e4-u24d828d9a-trunk) lib/pleroma/http.ex:88: Pleroma.HTTP.request/5

2025-10-10 02:03:04.622 [error] Failed to fetch https://www.theverge.com/news/797051/discord-government-ids-leaked-data-breach: ** (ErlangError) Erlang error: :data_error
    :zlib.inflateEnd_nif(#Reference<0.113196428.1119223809.170056>)
    :zlib.gunzip/1
    (tesla 1.14.1) lib/tesla/middleware/compression.ex:88: Tesla.Middleware.Compression.decompress_body/3
    (tesla 1.14.1) lib/tesla/middleware/compression.ex:72: Tesla.Middleware.Compression.decompress/1
    (tesla 1.14.1) lib/tesla/middleware/compression.ex:67: Tesla.Middleware.Compression.decompress/1
    (tesla 1.14.1) lib/tesla/middleware/follow_redirects.ex:49: Tesla.Middleware.FollowRedirects.redirect/3
    (pleroma 3.15.2-wf-121-gcf81c6e4-u24d828d9a-trunk) lib/pleroma/http/logger_middleware.ex:18: Pleroma.HTTP.LoggerMiddleware.call/3
    (pleroma 3.15.2-wf-121-gcf81c6e4-u24d828d9a-trunk) lib/pleroma/http.ex:88: Pleroma.HTTP.request/5

Note: in practice, the vast majority of data we receive or send will be ActivityPub documents to/from inboxes which cannot/will not be compressed either way. It certainly would be nice to use transport compression for rich media (link preview) requests though

EDIT: but even with the fix, the middleware currently always deletes the Content-encoding header, not sure if that might cause problems (other than for HEAD requests where we want to omit the middleware for other related reasons anyway)

In theory this should be as easy as adding `Tesla.Middleware.DecompressResponse` near the end of our Tesla middleware list *exepct for `HEAD` requests* (only followed by the telemetry middleware). However, the middleware is buggy in the Tesla version we currently use and crashes on responses with an empty body like below. A [fix](https://github.com/elixir-tesla/tesla/commit/5bc9b82823b3238257619ea3d67f0985a3707d2b) was already merged upstream and is included in the latest Tesla release though. ``` 2025-10-09 23:49:01.991 [error] Failed to fetch https://www.litepc.com/98lite.html: ** (ErlangError) Erlang error: :data_error :zlib.inflateEnd_nif(#Reference<0.113196428.1107689474.136005>) :zlib.gunzip/1 (tesla 1.14.1) lib/tesla/middleware/compression.ex:88: Tesla.Middleware.Compression.decompress_body/3 (tesla 1.14.1) lib/tesla/middleware/compression.ex:72: Tesla.Middleware.Compression.decompress/1 (tesla 1.14.1) lib/tesla/middleware/compression.ex:67: Tesla.Middleware.Compression.decompress/1 (tesla 1.14.1) lib/tesla/middleware/follow_redirects.ex:49: Tesla.Middleware.FollowRedirects.redirect/3 (pleroma 3.15.2-wf-121-gcf81c6e4-u24d828d9a-trunk) lib/pleroma/http/logger_middleware.ex:18: Pleroma.HTTP.LoggerMiddleware.call/3 (pleroma 3.15.2-wf-121-gcf81c6e4-u24d828d9a-trunk) lib/pleroma/http.ex:88: Pleroma.HTTP.request/5 2025-10-10 02:03:04.622 [error] Failed to fetch https://www.theverge.com/news/797051/discord-government-ids-leaked-data-breach: ** (ErlangError) Erlang error: :data_error :zlib.inflateEnd_nif(#Reference<0.113196428.1119223809.170056>) :zlib.gunzip/1 (tesla 1.14.1) lib/tesla/middleware/compression.ex:88: Tesla.Middleware.Compression.decompress_body/3 (tesla 1.14.1) lib/tesla/middleware/compression.ex:72: Tesla.Middleware.Compression.decompress/1 (tesla 1.14.1) lib/tesla/middleware/compression.ex:67: Tesla.Middleware.Compression.decompress/1 (tesla 1.14.1) lib/tesla/middleware/follow_redirects.ex:49: Tesla.Middleware.FollowRedirects.redirect/3 (pleroma 3.15.2-wf-121-gcf81c6e4-u24d828d9a-trunk) lib/pleroma/http/logger_middleware.ex:18: Pleroma.HTTP.LoggerMiddleware.call/3 (pleroma 3.15.2-wf-121-gcf81c6e4-u24d828d9a-trunk) lib/pleroma/http.ex:88: Pleroma.HTTP.request/5 ``` Note: in practice, the vast majority of data we receive or send will be ActivityPub documents to/from inboxes which cannot/will not be compressed either way. It certainly would be nice to use transport compression for rich media (link preview) requests though **EDIT**: but even with the fix, the middleware currently always deletes the `Content-encoding` header, not sure if that might cause problems (other than for HEAD requests where we want to omit the middleware for other related reasons anyway)

Oneric commented

2025-10-12 00:21:08 +00:00

Owner

The decompression middleware got some more fixes merged; after the next Tesla release (> 1.15.3) I think we should be able to make use of it