Cannot fetch from IPv6-only instance with CNAME entry #947

Open
opened 2025-07-11 16:28:12 +00:00 by Oneric · 3 comments
Owner

Your setup

From source

Extra details

Alpine 3.22

Version

3.15.2-wf-100-gd3da49f2-u16d7d612f-trunk (i.e. basically 16d7d612f)

PostgreSQL version

17

What’s wrong?

Fetches from an IPv6-only instance whose domain is a CNAME entry fail with :nxdomain (frequently showing up in logs), fetches from IPv6-only instances without a CNAME entry succeed since #708.

The odd thing is, manually Finch and Mint there’s no issue with connecting to the instance providing inet6: true is set in transport_opts (which we set by default). Not even manual Tesla.get calls fail when providing a simple client using the Finch adapter and our existing Finch pool.
However, using Tesla.get without providing this simple test client; this should whatever default was specified in the config — but our config provides an identical client setup(!):

# config/config.exs
config :tesla, :adapter, {Tesla.Adapter.Finch, name: MyFinch}

# remote shell of running instance
iex(1)> client = Tesla.client([], {Tesla.Adapter.Finch, name: MyFinch})
iex(2)> Tesla.get(client, "https://gts.wezen.famfo.xyz/")
# works
iex(3)> Tesla.get("https://gts.wezen.famfo.xyz/")                                                      
{:error, :nxdomain}

Not sure if somethings wrong with our Tesla setup or if it’s a Tesla bug

Test instance (just grabbing what showed up in logs):

  • ipv6.camp: IPv6-only, no CNAME
  • gts.wezen.famfo.xyz: CNAME domain; IPv6-only

Severity

I can manage

Have you searched for this issue?

  • I have double-checked and have not found this issue mentioned anywhere.
### Your setup From source ### Extra details Alpine 3.22 ### Version 3.15.2-wf-100-gd3da49f2-u16d7d612f-trunk (i.e. basically 16d7d612f) ### PostgreSQL version 17 ### What’s wrong? Fetches from an IPv6-only instance whose domain is a CNAME entry fail with `:nxdomain` (frequently showing up in logs), fetches from IPv6-only instances without a CNAME entry succeed since #708. The odd thing is, manually Finch and Mint there’s no issue with connecting to the instance providing `inet6: true` is set in `transport_opts` *(which we set by default)*. Not even manual `Tesla.get` calls fail when providing a simple client using the Finch adapter and our existing Finch pool. However, using `Tesla.get` _without_ providing this simple test client; this should whatever default was specified in the config — but our config provides an identical client setup(!): ```elixir # config/config.exs config :tesla, :adapter, {Tesla.Adapter.Finch, name: MyFinch} # remote shell of running instance iex(1)> client = Tesla.client([], {Tesla.Adapter.Finch, name: MyFinch}) iex(2)> Tesla.get(client, "https://gts.wezen.famfo.xyz/") # works iex(3)> Tesla.get("https://gts.wezen.famfo.xyz/") {:error, :nxdomain} ``` Not sure if somethings wrong with our Tesla setup or if it’s a Tesla bug Test instance *(just grabbing what showed up in logs)*: - `ipv6.camp`: IPv6-only, no CNAME - `gts.wezen.famfo.xyz`: CNAME domain; IPv6-only ### Severity I can manage ### Have you searched for this issue? - [x] I have double-checked and have not found this issue mentioned anywhere.

we're a bit out of date with tesla/finch deps - you could potentially try bumping those?

i'm trying to test it but my ISP doesn't provide ipv6

we're a bit out of date with tesla/finch deps - you could potentially try bumping those? i'm trying to test it but my ISP doesn't provide ipv6
Author
Owner

It gets weirder: it actually works on my local machine in both dev and prod env, but not on the actual prod instance from which the above report was generated. Erlang, Elixir, etc versions are equivalent (both use Aloine 3.22 packages). DNS responses also look similar.
The only difference is the local machine being x86_64 and the prod instance aarch64, but this shouldn’t matter for DNS resolution.

DNS responses on prod instance
$ dig gts.wezen.famfo.xyz A
; <<>> DiG 9.20.10 <<>> gts.wezen.famfo.xyz A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53430
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;gts.wezen.famfo.xyz.           IN      A

;; ANSWER SECTION:
gts.wezen.famfo.xyz.    3597    IN      CNAME   wezen.canis-major.famfo.xyz.

;; AUTHORITY SECTION:
famfo.xyz.              3597    IN      SOA     ns1.famfo.xyz. dns.famfo.xyz. 2025108060 10800 3600 604800 3600

;; Query time: 0 msec
;; MSG SIZE  rcvd: 124



$ dig gts.wezen.famfo.xyz AAAA
; <<>> DiG 9.20.10 <<>> gts.wezen.famfo.xyz AAAA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39796
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;gts.wezen.famfo.xyz.           IN      AAAA

;; ANSWER SECTION:
gts.wezen.famfo.xyz.    3250    IN      CNAME   wezen.canis-major.famfo.xyz.
wezen.canis-major.famfo.xyz. 3600 IN    AAAA    2a0a:4cc0:c0:313f:e300::3

;; Query time: 352 msec
;; MSG SIZE  rcvd: 108



$ dig wezen.canis-major.famfo.xyz A
; <<>> DiG 9.20.10 <<>> wezen.canis-major.famfo.xyz A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4745
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;wezen.canis-major.famfo.xyz.   IN      A

;; AUTHORITY SECTION:
famfo.xyz.              3600    IN      SOA     ns1.famfo.xyz. dns.famfo.xyz. 2025108064 10800 3600 604800 3600

;; Query time: 404 msec
;; MSG SIZE  rcvd: 100

$ dig wezen.canis-major.famfo.xyz AAAA

; <<>> DiG 9.20.10 <<>> wezen.canis-major.famfo.xyz AAAA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48387
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;wezen.canis-major.famfo.xyz.   IN      AAAA

;; ANSWER SECTION:
wezen.canis-major.famfo.xyz. 1886 IN    AAAA    2a0a:4cc0:c0:313f:e300::3

;; Query time: 184 msec
;; MSG SIZE  rcvd: 84
DNS responses on local machine
$ dig gts.wezen.famfo.xyz A
; <<>> DiG 9.20.10 <<>> gts.wezen.famfo.xyz
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33710
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;gts.wezen.famfo.xyz.		IN	A

;; ANSWER SECTION:
gts.wezen.famfo.xyz.	1760	IN	CNAME	wezen.canis-major.famfo.xyz.

;; AUTHORITY SECTION:
famfo.xyz.		15	IN	SOA	ns1.famfo.xyz. dns.famfo.xyz. 2025108052 10800 3600 604800 3600

;; Query time: 7 msec
;; MSG SIZE  rcvd: 124



$dig gts.wezen.famfo.xyz AAAA
; <<>> DiG 9.20.10 <<>> gts.wezen.famfo.xyz AAAA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33636
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;gts.wezen.famfo.xyz.		IN	AAAA

;; ANSWER SECTION:
gts.wezen.famfo.xyz.	1721	IN	CNAME	wezen.canis-major.famfo.xyz.
wezen.canis-major.famfo.xyz. 1722 IN	AAAA	2a0a:4cc0:c0:313f:e300::3

;; Query time: 0 msec
;; MSG SIZE  rcvd: 108



$dig wezen.canis-major.famfo.xyz A

; <<>> DiG 9.20.10 <<>> wezen.canis-major.famfo.xyz A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51356
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;wezen.canis-major.famfo.xyz.	IN	A

;; AUTHORITY SECTION:
famfo.xyz.		15	IN	SOA	ns1.famfo.xyz. dns.famfo.xyz. 2025108052 10800 3600 604800 3600

;; Query time: 3 msec
;; MSG SIZE  rcvd: 100



$ dig wezen.canis-major.famfo.xyz AAAA

; <<>> DiG 9.20.10 <<>> wezen.canis-major.famfo.xyz AAAA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33302
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;wezen.canis-major.famfo.xyz.	IN	AAAA

;; ANSWER SECTION:
wezen.canis-major.famfo.xyz. 1671 IN	AAAA	2a0a:4cc0:c0:313f:e300::3

;; Query time: 0 msec
;; MSG SIZE  rcvd: 84

(note: the Akkoma version listed in OP is patched to use a newer Finch, but the result does not change when rolling the instance back to the default Finch version nor does it change when moving Finch to 0.19 on the local machine. At least on the local machine prod vs dev env did not make a difference)

It gets weirder: it actually works on my local machine in both `dev` and `prod` env, but not on the actual prod instance from which the above report was generated. Erlang, Elixir, etc versions are equivalent (both use Aloine 3.22 packages). DNS responses also look similar. The only difference is the local machine being x86_64 and the prod instance aarch64, but this _shouldn’t_ matter for DNS resolution. <details> <summary>DNS responses on prod instance</summary> ``` $ dig gts.wezen.famfo.xyz A ; <<>> DiG 9.20.10 <<>> gts.wezen.famfo.xyz A ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53430 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;gts.wezen.famfo.xyz. IN A ;; ANSWER SECTION: gts.wezen.famfo.xyz. 3597 IN CNAME wezen.canis-major.famfo.xyz. ;; AUTHORITY SECTION: famfo.xyz. 3597 IN SOA ns1.famfo.xyz. dns.famfo.xyz. 2025108060 10800 3600 604800 3600 ;; Query time: 0 msec ;; MSG SIZE rcvd: 124 $ dig gts.wezen.famfo.xyz AAAA ; <<>> DiG 9.20.10 <<>> gts.wezen.famfo.xyz AAAA ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39796 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;gts.wezen.famfo.xyz. IN AAAA ;; ANSWER SECTION: gts.wezen.famfo.xyz. 3250 IN CNAME wezen.canis-major.famfo.xyz. wezen.canis-major.famfo.xyz. 3600 IN AAAA 2a0a:4cc0:c0:313f:e300::3 ;; Query time: 352 msec ;; MSG SIZE rcvd: 108 $ dig wezen.canis-major.famfo.xyz A ; <<>> DiG 9.20.10 <<>> wezen.canis-major.famfo.xyz A ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4745 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;wezen.canis-major.famfo.xyz. IN A ;; AUTHORITY SECTION: famfo.xyz. 3600 IN SOA ns1.famfo.xyz. dns.famfo.xyz. 2025108064 10800 3600 604800 3600 ;; Query time: 404 msec ;; MSG SIZE rcvd: 100 $ dig wezen.canis-major.famfo.xyz AAAA ; <<>> DiG 9.20.10 <<>> wezen.canis-major.famfo.xyz AAAA ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48387 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;wezen.canis-major.famfo.xyz. IN AAAA ;; ANSWER SECTION: wezen.canis-major.famfo.xyz. 1886 IN AAAA 2a0a:4cc0:c0:313f:e300::3 ;; Query time: 184 msec ;; MSG SIZE rcvd: 84 ``` </details> <details> <summary>DNS responses on local machine</summary> ``` $ dig gts.wezen.famfo.xyz A ; <<>> DiG 9.20.10 <<>> gts.wezen.famfo.xyz ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33710 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;gts.wezen.famfo.xyz. IN A ;; ANSWER SECTION: gts.wezen.famfo.xyz. 1760 IN CNAME wezen.canis-major.famfo.xyz. ;; AUTHORITY SECTION: famfo.xyz. 15 IN SOA ns1.famfo.xyz. dns.famfo.xyz. 2025108052 10800 3600 604800 3600 ;; Query time: 7 msec ;; MSG SIZE rcvd: 124 $dig gts.wezen.famfo.xyz AAAA ; <<>> DiG 9.20.10 <<>> gts.wezen.famfo.xyz AAAA ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33636 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;gts.wezen.famfo.xyz. IN AAAA ;; ANSWER SECTION: gts.wezen.famfo.xyz. 1721 IN CNAME wezen.canis-major.famfo.xyz. wezen.canis-major.famfo.xyz. 1722 IN AAAA 2a0a:4cc0:c0:313f:e300::3 ;; Query time: 0 msec ;; MSG SIZE rcvd: 108 $dig wezen.canis-major.famfo.xyz A ; <<>> DiG 9.20.10 <<>> wezen.canis-major.famfo.xyz A ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51356 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;wezen.canis-major.famfo.xyz. IN A ;; AUTHORITY SECTION: famfo.xyz. 15 IN SOA ns1.famfo.xyz. dns.famfo.xyz. 2025108052 10800 3600 604800 3600 ;; Query time: 3 msec ;; MSG SIZE rcvd: 100 $ dig wezen.canis-major.famfo.xyz AAAA ; <<>> DiG 9.20.10 <<>> wezen.canis-major.famfo.xyz AAAA ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33302 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;wezen.canis-major.famfo.xyz. IN AAAA ;; ANSWER SECTION: wezen.canis-major.famfo.xyz. 1671 IN AAAA 2a0a:4cc0:c0:313f:e300::3 ;; Query time: 0 msec ;; MSG SIZE rcvd: 84 ``` </details> *(note: the Akkoma version listed in OP is patched to use a newer Finch, but the result does not change when rolling the instance back to the default Finch version nor does it change when moving Finch to 0.19 on the local machine. At least on the local machine `prod` vs `dev` env did not make a difference)*
Author
Owner

After upgrading Tesla (and its deps) to the newest version:
Both the version with explicit and implicit client, sometimes fail and other times succeed. If it succeeded once, it usually keeps succeeding for a while suggesting the successful resolution gets cached for a while?

I cannot say whether the same occured before upgrading versions.
Either way, during normal operation, basically all my instance’s attempts to fetch something from the CNAME-IPv6 instance would run into the cold-cache case, explaining the prevalence of logs about it.

Still can’t reproduce this on the same machine with Tesla+Finch directly though

unrelated side note: we probably should set protocols: [:http2, :http1] in our Finch pool opts to not force HTTP1. Though unless we go HTTP2-only (which might cause interop issues) we won’t get multiplexing

After upgrading Tesla (and its deps) to the newest version: Both the version with explicit and implicit `client`, _sometimes_ fail and other times succeed. If it succeeded once, it usually keeps succeeding for a while suggesting the successful resolution gets cached for a while? I cannot say whether the same occured before upgrading versions. Either way, during normal operation, basically all my instance’s attempts to fetch something from the CNAME-IPv6 instance would run into the cold-cache case, explaining the prevalence of logs about it. Still can’t reproduce this on the same machine with `Tesla+Finch` directly though unrelated side note: we probably should set `protocols: [:http2, :http1]` in our Finch pool opts to not _force_ HTTP1. Though unless we go HTTP2-only *(which might cause interop issues)* we won’t get multiplexing
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
AkkomaGang/akkoma#947
No description provided.