ci: add a label-gated IPv6-only kind e2e job - #939
Draft
Yuan Gao (ygao-g) wants to merge 4 commits into
Draft
Conversation
IP_FAMILY selects ipv4, ipv6 or dual and becomes networking.ipFamily, leaving kind's per-family subnet defaults alone. The script also recreates a pre-IPv6 "kind" Docker network, fails fast if the daemon has IPv6 off, sets proxy_ndp alongside proxy_arp for gVisor pod-to-pod traffic, and repoints an ipv6 kubeconfig from [::1] at localhost so a client outside the Docker host can still reach the apiserver. Tested on kind with all three families: node InternalIPs, Service ClusterIPs and pod IPs land in the requested families, pod-to-pod and CoreDNS work on them, and pods still pull through the local registry.
CoreDNS runs dnsPolicy: Default and inherits the node's Docker-generated /etc/resolv.conf, which always names an IPv4 resolver -- unreachable from a v6-only pod, so every external lookup dies at "connect: network is unreachable". Behind that sits a second failure: this script wires the registry into containerd on the node, but atelet pulls actor images from its own pod netns, where kind-registry does not resolve at all. Rewrite the Corefile on IPv6-only clusters: a hosts block mapping kind-registry to the registry's GlobalIPv6Address, with fallthrough so non-registry names still reach the new IPv6 forwarder. Hard-fail if the substitution was a no-op, and probe both names from a pod rather than the node, which is dual-stack and would resolve them either way.
…families The gateway's two Envoy sockets bound 0.0.0.0, so on an IPv6-primary cluster the kubelet's startup probe against the admin port was refused and atenet-egress crashlooped while Envoy itself started fine and logged "admin address: 0.0.0.0:15000". The :443 listener had the same gap, leaving no v6 path for an actor's CONNECT. Both are single sockets, so they bind "::" with ipv4_compat rather than taking the additional-address pairing the ingress listeners use. On the admin socket ipv4_compat is load-bearing: the ext-proc sidecar's drainer reaches it at 127.0.0.1:15000 and envoydrain.go reads a refusal as "Envoy already exited", so a bare "::" would silently skip the drain. The Service gets PreferDualStack for the same reason the router's does -- without it a dual-stack cluster hands out one ClusterIP and the new v6 bind is unreachable.
Runs the full install plus the demo and networking e2e suites against an IPv6-only kind cluster, on its own non-default cluster name. The job is gated on the ci/ipv6 label and stays out of the e2e-test merge gate, so it costs nothing on a normal PR and cannot block one. ubuntu-latest has no IPv6 egress, so the job stands up tayga for NAT64 and points CoreDNS at an upstream resolver through the well-known prefix. DNS64 is scoped to a catch-all server block: synthesizing AAAA over the cluster zones destroys the v6-only ClusterIP answers and the control plane never comes up.
Yuan Gao (ygao-g)
force-pushed
the
ci-kind-ipv6
branch
from
August 13, 2026 21:37
d1f319d to
ea307ee
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an
e2e-test-ipv6job that stands up a single-stack IPv6-only kind cluster,installs the full control plane, and runs the
demoandnetworkinge2e suitesagainst it.
The job only runs when a PR carries the
ci/ipv6label, and it isdeliberately not part of the
e2e-testmerge gate. A normal PR costs nothingand cannot be blocked by it.
Stacked on #877
This branch is built on #877's
kind-ip-family-core, which provides theIP_FAMILYknob the job depends on. GitHub will not accept a base branch thatlives only in a fork, so this PR targets
mainand the diff currently shows#877's commit alongside its own. Once #877 merges I will rebase, and the diff
drops to the three commits below. Do not merge before #877.
Commits
hack/create-kind-cluster.sh: fix DNS on IPv6-only clusters— CoreDNS on afresh
IP_FAMILY=ipv6cluster inherits the node's IPv4 resolver, so nothingresolves from a pod and no actor boots.
atenet/egress: serve the admin socket, listener, and Service on both families— duplicates atenet: bind the Envoy listeners, admin sockets, and Services dual-stack #911. Without it--deploy-ate-systemnevercompletes:
install-ate.shwaits on theatenet-egressrollout, whosereadiness probe is an httpGet the kubelet dials over IPv6 while Envoy's admin
socket binds
0.0.0.0. This commit drops out when atenet: bind the Envoy listeners, admin sockets, and Services dual-stack #911 merges.ci: add a label-gated IPv6-only kind e2e job.Why the job carries NAT64
ubuntu-latesthas no IPv6 egress — measured, not assumed. A v6-only clusterstill has to reach
storage.googleapis.com, since atelet fetches the gVisorrelease named in the
SandboxConfigat runtime. So the job installstayga,routes the well-known
64:ff9b::/96prefix through it, and points CoreDNS at anupstream resolver via that prefix. Both the runner-side translation and
in-cluster resolution are asserted before anything is built, so a NAT64
regression fails as itself rather than as a mysterious install timeout.
DNS64 is scoped to a catch-all
.:53server block.translate_allwraps theentire plugin chain below it, so leaving it over the cluster zones synthesizes
AAAA for every v6-only ClusterIP answer and the control plane never comes up.
Current results
demonetworkingThe five skips are family-independent: the Golden resume source is micro-VM
only, and no CSI StorageClass is installed.
TestActorEgressfails, and it is a genuine gap rather than a CI artifact.internal/ateomnet/net.go:40-47gives the sandbox netns a hardcoded IPv4link-local
/30with v4-only masquerade, so an actor has no IPv6 stack andcannot reach a v6-only ClusterIP resolver:
Dual-stacking the sandbox veth is out of scope here; filing separately.
These numbers come from a base that also carries
45b7db75andbf9d6d69, bothalready in
mainand both predated by #877's branch point. Without them the jobadditionally shows three
ResumeGoldenActortimeouts indemoand a secondnetworkingfailure, so the rebase described above is what makes the job greenapart from
TestActorEgress.Part of #246.