atenet: bind the Envoy listeners, admin sockets, and Services dual-stack - #911
Open
Yuan Gao (ygao-g) wants to merge 3 commits into
Open
atenet: bind the Envoy listeners, admin sockets, and Services dual-stack#911Yuan Gao (ygao-g) wants to merge 3 commits into
Yuan Gao (ygao-g) wants to merge 3 commits into
Conversation
The HTTP and HTTPS ingress listeners bound 0.0.0.0 only, so on a dual-stack cluster Envoy answered on the router Service's IPv4 ClusterIP and nothing at all on its IPv6 one. Pair each primary socket with an additional "::" address on the same port. Ipv4Compat is false on the additional address. Setting it would clear IPV6_V6ONLY and collide with the primary IPv4 wildcard already bound to that port, and Envoy rejects the whole listener when an additional address fails to bind -- that would take down all ingress, not just the IPv6 half. Hoisting the literal into a helper keeps the two listeners from drifting. No behaviour change on an IPv4-only cluster: the primary address is untouched, and a host without IPv6 simply has no second socket to bind.
The Envoy admin socket bound 0.0.0.0, leaving it reachable over IPv4 only. It binds "::" with ipv4_compat now -- one socket for both families. ipv4_compat is required rather than incidental here: Envoy sets IPV6_V6ONLY without it, and dataplane.go health-checks the listener over http://127.0.0.1:9901/ready, so dropping it would take the dataplane component of /statusz unhealthy. The atenet-router Service carried no ipFamilyPolicy, which the API server defaults to SingleStack -- an IPv4 ClusterIP and nothing else, which leaves the listeners above with no IPv6 address to answer on. Prefer, not Require, so this stays valid on a single-stack cluster, where it is a no-op. spec.ipFamilies is deliberately left alone: the primary family is immutable and the API server appends the secondary one itself.
…families The gateway's two Envoy sockets bound 0.0.0.0, so on an IPv6-primary cluster the kubelet's startup probe against the admin port was refused and atenet-egress crashlooped while Envoy itself started fine and logged "admin address: 0.0.0.0:15000". The :443 listener had the same gap, leaving no v6 path for an actor's CONNECT. Both are single sockets, so they bind "::" with ipv4_compat rather than taking the additional-address pairing the ingress listeners use. On the admin socket ipv4_compat is load-bearing: the ext-proc sidecar's drainer reaches it at 127.0.0.1:15000 and envoydrain.go reads a refusal as "Envoy already exited", so a bare "::" would silently skip the drain. The Service gets PreferDualStack for the same reason the router's does -- without it a dual-stack cluster hands out one ClusterIP and the new v6 bind is unreachable.
Yuan Gao (ygao-g)
marked this pull request as ready for review
August 13, 2026 15:19
Contributor
Author
|
Bowei Du (@bowei) mind taking a look? |
This was referenced Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #910
Sub-task of #246, step 2 of bowei's concrete steps: "Fixing the Envoy bind to include IPv6 addresses."
Every Envoy socket in atenet is bound to the IPv4 wildcard, and neither gateway's Service asks for a second IP family. On an IPv6-primary cluster the kubelet probes a pod on its only address, so
atenet-egresscrashlooped for 14h on a v6-only kind cluster — Envoy started fine, loggedadmin address: 0.0.0.0:15000, and then:On a dual-stack cluster the same gap lands on the data path instead: the router answers on its Service's IPv4 ClusterIP and on nothing at all for IPv6.
Two socket shapes, two mechanisms. The ingress listeners keep their
0.0.0.0primary and gain an additional::address on the same port,ipv4_compat: false— setting it would clearIPV6_V6ONLYand collide with the primary already bound to that port, and Envoy rejects the whole listener when an additional address fails to bind, taking down all ingress rather than just the IPv6 half. The two admin sockets and the egress:443listener are single sockets, so they bind::withipv4_compat: trueinstead. That flag is load-bearing there, not incidental:dataplane.gohealth-checks the router admin overhttp://127.0.0.1:9901/ready, and the egress ext-proc sidecar's drainer dials127.0.0.1:15000, whereenvoydrain.goreads a refusal as "Envoy already exited" and skips the drain silently.Services. Neither
atenet-routernoratenet-egresscarried anipFamilyPolicy, which defaults to SingleStack — one ClusterIP and nothing else, leaving the listeners above with no IPv6 address to answer on. Both becomePreferDualStack: Prefer, not Require, which fails Service creation outright on a single-stack cluster.spec.ipFamiliesis left alone; the primary family is immutable and the API server appends the secondary itself.No behaviour change on IPv4-only: the ingress primary is untouched,
PreferDualStackis a no-op there, and a host without IPv6 simply has no second socket to bind.Not in scope, so this makes both gateways accept IPv6 without yet reaching IPv6 destinations:
dns_lookup_family: V4_ONLYon the egress dynamic forward proxy (already named in #246), Envoy's V4-only default on the OTLP cluster inxds.go,cmd/atenet/internal/dns/corefile.goemittingtemplate IN Awith no AAAA path, and atunnel's original-destination lookup (#686).Testing
go test ./cmd/atenet/internal/router/...passes.TestXdsServer_UpdateSnapshotandTestXdsServer_UpdateSnapshot_WithHttpsassert the0.0.0.0primary, the::additional address,ipv4_compatfalse, and the port on both ingress listeners. The manifest changes have no Go test hook.atenet-egresswent from1/2 CrashLoopBackOff(64 restarts, startup probe refused) to2/2 Runningwith 0 restarts, Envoy loggingadmin address: [::]:15000.hack/install-ate-kind.sh --deploy-ate-systemcompletes, and Envoy's/listenersthen reports all four ingress sockets —ingress_http_listener::0.0.0.0:8080,ingress_http_listener::[::]:8080,ingress_https_listener::0.0.0.0:8443,ingress_https_listener::[::]:8443.http://127.0.0.1:15000/readyandhttp://[::1]:15000/readyboth return 200, so the singleipv4_compatsocket still serves the drainer's IPv4 loopback dial.hack/create-kind-cluster.shonmainwrites nonetworking:block, so there is no dual-stack cluster to testPreferDualStackagainst — that is hack: create-kind-cluster.sh cannot make a dual-stack or IPv6-only cluster #845, step 1 of the same plan. The v6-only run above used unmerged tooling from that issue, and it cannot distinguishPreferDualStackfromSingleStackanyway, because the single family there is v6 either way. I will confirm both Services come up with two ClusterIPs once hack: create-kind-cluster.sh cannot make a dual-stack or IPv6-only cluster #845 lands.