Skip to content

atenet: bind the Envoy listeners, admin sockets, and Services dual-stack - #911

Open
Yuan Gao (ygao-g) wants to merge 3 commits into
agent-substrate:mainfrom
ygao-g:atenet-envoy-dualstack
Open

atenet: bind the Envoy listeners, admin sockets, and Services dual-stack#911
Yuan Gao (ygao-g) wants to merge 3 commits into
agent-substrate:mainfrom
ygao-g:atenet-envoy-dualstack

Conversation

@ygao-g

Copy link
Copy Markdown
Contributor

Fixes #910

It's a good idea to open an issue first for discussion.

  • Tests pass
  • Appropriate changes to documentation are included in the PR (none needed — no flags, APIs, or user-facing behaviour change)

Sub-task of #246, step 2 of bowei's concrete steps: "Fixing the Envoy bind to include IPv6 addresses."


Every Envoy socket in atenet is bound to the IPv4 wildcard, and neither gateway's Service asks for a second IP family. On an IPv6-primary cluster the kubelet probes a pod on its only address, so atenet-egress crashlooped for 14h on a v6-only kind cluster — Envoy started fine, logged admin address: 0.0.0.0:15000, and then:

Startup probe failed: Get "http://[fd00:10:244::18]:15000/ready": connect: connection refused

On a dual-stack cluster the same gap lands on the data path instead: the router answers on its Service's IPv4 ClusterIP and on nothing at all for IPv6.

Two socket shapes, two mechanisms. The ingress listeners keep their 0.0.0.0 primary and gain an additional :: address on the same port, ipv4_compat: false — setting it would clear IPV6_V6ONLY and collide with the primary already bound to that port, and Envoy rejects the whole listener when an additional address fails to bind, taking down all ingress rather than just the IPv6 half. The two admin sockets and the egress :443 listener are single sockets, so they bind :: with ipv4_compat: true instead. That flag is load-bearing there, not incidental: dataplane.go health-checks the router admin over http://127.0.0.1:9901/ready, and the egress ext-proc sidecar's drainer dials 127.0.0.1:15000, where envoydrain.go reads a refusal as "Envoy already exited" and skips the drain silently.

Services. Neither atenet-router nor atenet-egress carried an ipFamilyPolicy, which defaults to SingleStack — one ClusterIP and nothing else, leaving the listeners above with no IPv6 address to answer on. Both become PreferDualStack: Prefer, not Require, which fails Service creation outright on a single-stack cluster. spec.ipFamilies is left alone; the primary family is immutable and the API server appends the secondary itself.

No behaviour change on IPv4-only: the ingress primary is untouched, PreferDualStack is a no-op there, and a host without IPv6 simply has no second socket to bind.

Not in scope, so this makes both gateways accept IPv6 without yet reaching IPv6 destinations: dns_lookup_family: V4_ONLY on the egress dynamic forward proxy (already named in #246), Envoy's V4-only default on the OTLP cluster in xds.go, cmd/atenet/internal/dns/corefile.go emitting template IN A with no AAAA path, and atunnel's original-destination lookup (#686).

Testing

  • go test ./cmd/atenet/internal/router/... passes. TestXdsServer_UpdateSnapshot and TestXdsServer_UpdateSnapshot_WithHttps assert the 0.0.0.0 primary, the :: additional address, ipv4_compat false, and the port on both ingress listeners. The manifest changes have no Go test hook.
  • On an IPv6-only kind cluster: atenet-egress went from 1/2 CrashLoopBackOff (64 restarts, startup probe refused) to 2/2 Running with 0 restarts, Envoy logging admin address: [::]:15000. hack/install-ate-kind.sh --deploy-ate-system completes, and Envoy's /listeners then reports all four ingress sockets — ingress_http_listener::0.0.0.0:8080, ingress_http_listener::[::]:8080, ingress_https_listener::0.0.0.0:8443, ingress_https_listener::[::]:8443.
  • From inside the egress pod's netns, http://127.0.0.1:15000/ready and http://[::1]:15000/ready both return 200, so the single ipv4_compat socket still serves the drainer's IPv4 loopback dial.
  • Dual-stack is not verified, which is why this is a draft. hack/create-kind-cluster.sh on main writes no networking: block, so there is no dual-stack cluster to test PreferDualStack against — that is hack: create-kind-cluster.sh cannot make a dual-stack or IPv6-only cluster #845, step 1 of the same plan. The v6-only run above used unmerged tooling from that issue, and it cannot distinguish PreferDualStack from SingleStack anyway, because the single family there is v6 either way. I will confirm both Services come up with two ClusterIPs once hack: create-kind-cluster.sh cannot make a dual-stack or IPv6-only cluster #845 lands.
  • Not run: the networking e2e suites. They need actors, and actors cannot boot on a v6-only kind cluster until the CoreDNS resolver fix (also hack: create-kind-cluster.sh cannot make a dual-stack or IPv6-only cluster #845 territory — CoreDNS inherits the node's IPv4 resolver) is in place.

The HTTP and HTTPS ingress listeners bound 0.0.0.0 only, so on a
dual-stack cluster Envoy answered on the router Service's IPv4 ClusterIP
and nothing at all on its IPv6 one. Pair each primary socket with an
additional "::" address on the same port.

Ipv4Compat is false on the additional address. Setting it would clear
IPV6_V6ONLY and collide with the primary IPv4 wildcard already bound to
that port, and Envoy rejects the whole listener when an additional
address fails to bind -- that would take down all ingress, not just the
IPv6 half. Hoisting the literal into a helper keeps the two listeners
from drifting.

No behaviour change on an IPv4-only cluster: the primary address is
untouched, and a host without IPv6 simply has no second socket to bind.
The Envoy admin socket bound 0.0.0.0, leaving it reachable over IPv4 only.
It binds "::" with ipv4_compat now -- one socket for both families.
ipv4_compat is required rather than incidental here: Envoy sets
IPV6_V6ONLY without it, and dataplane.go health-checks the listener over
http://127.0.0.1:9901/ready, so dropping it would take the dataplane
component of /statusz unhealthy.

The atenet-router Service carried no ipFamilyPolicy, which the API server
defaults to SingleStack -- an IPv4 ClusterIP and nothing else, which
leaves the listeners above with no IPv6 address to answer on. Prefer, not
Require, so this stays valid on a single-stack cluster, where it is a
no-op. spec.ipFamilies is deliberately left alone: the primary family is
immutable and the API server appends the secondary one itself.
…families

The gateway's two Envoy sockets bound 0.0.0.0, so on an IPv6-primary
cluster the kubelet's startup probe against the admin port was refused and
atenet-egress crashlooped while Envoy itself started fine and logged "admin
address: 0.0.0.0:15000". The :443 listener had the same gap, leaving no v6
path for an actor's CONNECT.

Both are single sockets, so they bind "::" with ipv4_compat rather than
taking the additional-address pairing the ingress listeners use. On the
admin socket ipv4_compat is load-bearing: the ext-proc sidecar's drainer
reaches it at 127.0.0.1:15000 and envoydrain.go reads a refusal as "Envoy
already exited", so a bare "::" would silently skip the drain. The Service
gets PreferDualStack for the same reason the router's does -- without it a
dual-stack cluster hands out one ClusterIP and the new v6 bind is
unreachable.
@ygao-g

Copy link
Copy Markdown
Contributor Author

Bowei Du (@bowei) mind taking a look?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

atenet: every Envoy socket binds 0.0.0.0 and neither gateway Service is dual-stack

2 participants