dstack-ingress: follow-ups after the delegation (#104) and tls-alpn-01 (#105) merges

[kvinwang]

Follow-ups from reviewing #104 and #105 after both landed. Ordered by impact.
Nothing here needs a revert; all are incremental.

P1 — A failed HAProxy reload is never retried

dns01.sh / tlsalpn.sh:

if [ "$changed" -eq 1 ]; then
    collect_evidence     || echo "Evidence generation failed" >&2
    build_combined_pems  || echo "Combined PEM build failed" >&2
    haproxy_reload       || true
fi
[ "$failed" -eq 0 ]

The reload failure is swallowed and does not set failed, so the pass reports
success. Twelve hours later the ACME client says "nothing to renew", changed
is 0, and the reload is never attempted again. HAProxy keeps serving the
previous certificate — which was renewed precisely because it was near expiry
— until it expires. Same for a failed build_combined_pems.

#102 solved this with a renewal-applied.stamp and a certs_pending_apply()
check: certificate material newer than the stamp means the renewal has not
reached HAProxy yet, so the apply step retries on the next cycle. That PR
predates the restructure and targets renewal-daemon.sh / renew-certificate.sh,
which #105 removed, so the mechanism needs porting rather than merging — and it
can be simpler here: #102 needed a file because bootstrap and the daemon were
two processes, whereas one process now owns the whole lifecycle, so an in-memory
"apply pending" flag is enough. Credit to @pacoyang for finding it.

P1 — Delegation CAA verification is single-resolver and fails closed

verify_delegation_caa() queries only dns.google and refuses to start when the
record is not found. Public resolvers cache negative answers for the zone's SOA
minimum — up to an hour — so a resolver queried before the operator created the
record keeps reporting it absent well after it exists. That is a false-positive
refusal to start, observed during #105's testing on exactly this kind of check.

The failure asymmetry also runs the wrong way:

Situation	Current behaviour
DoH unreachable / non-zero DNS status	warn and continue — no CAA protection
Record genuinely present but cached negative	hard fail, container will not start

grep -F "accounturi=…" on .data compounds it: that matches presentation-form
rdata, which is what dns.google returns. Adding a second resolver naively would
break it, because Cloudflare returns CAA in RFC 3597 generic hex
(\# 47 00 05 69 73 73 75 65 …) and the literal match silently fails — which,
being fail-closed, is an outage.

dnsguide.py (added in #105) already does this properly: two resolvers with
union quorum for CAA, both rdata encodings, unit-tested against strings real
resolvers returned. Delegation should call it instead of rolling its own check.

That also gets the delegation path DNS_SETUP_MODE for free — wait blocks until
the operator's records appear instead of failing on the first look, and webhook
POSTs them to an automation endpoint with an HMAC and a TDX quote. One record
type has to be added to build_records: the _acme-challenge CNAME. The only
awkward part is ALLOW_MISSING_CAA, which has no dnsguide equivalent; mapping it
to --mode print would also skip CNAME/TXT verification, so it likely needs its
own flag.

P2 — unset_txt_record reports success when the zone lookup failed

get_dns_records() returns [] both when a name has no records and when the
zone could not be resolved, so unset_txt_record iterates nothing and returns
True. Observed:

acme-dns-alias-hook: removing challenge TXT _acme-challenge.svc.example.com.deleg.example.net
Error: Could not find zone for domain _acme-challenge.svc.example.com.deleg.example.net
Successfully unset TXT record for _acme-challenge.svc.example.com.deleg.example.net

Exit code 0. The hook discards cleanup failures deliberately, so the impact is a
stale TXT in the delegation zone (the next auth overwrites it) — but the log
states the opposite of what happened, which will cost someone an afternoon.

P2 — The delegated issuance path has never been executed

#104 states its testing as py_compile plus bash -n. Reviewing after the fact,
the mechanism does hold up:

• _get_zone_info does longest-suffix matching, so
  _acme-challenge.svc.example.com.deleg.example.net resolves to
  deleg.example.net and not to the served zone whose name it contains;
• the hook computes the right record name, finds dnsman.py on PATH, and uses
  the venv interpreter;
• a failed auth exits non-zero, so certbot does see the challenge fail.

But no one has run an actual delegated issuance — hook writes the TXT, Let's
Encrypt follows the CNAME, validation passes. TESTING.md describes how to set
this up; it needs a delegation zone.

P3 — set_txt_record deletes same-name records, which SAN will break

It removes every existing TXT at the name before creating the new one. Today
process_domain passes a single -d per certbot invocation, so only one
challenge value ever exists at _acme-challenge.<name> and this is fine.

It stops being fine as soon as one certificate covers several identifiers that
validate at the same name — example.com plus *.example.com, or #86's SAN
support. certbot then calls the auth hook once per identifier with the same
CERTBOT_DOMAIN and different CERTBOT_VALIDATION, both records have to exist
at once, and the second call deletes the first. Worth fixing before #86 lands:
the challenge hook wants add-without-delete semantics, distinct from the
set-and-replace the routing records want.

P3 — _ensure_zone_id falls back to a stale cached zone on lookup failure

zone_info = self._get_zone_info(domain)
if zone_info:
    self.zone_id, self.zone_domain = zone_info
return self.zone_id      # previous zone's id when the lookup failed

A failed lookup returns whichever zone was cached last, so a subsequent write
could land in the wrong zone. Pre-existing, and not currently reachable —
dnsman.py runs as a fresh process per call, so the cache is empty at the point
that matters. Worth closing anyway, since delegation mode is precisely the
feature that makes "which zone are we writing to" a security property.

[kvinwang]

One more, found while testing the delegation path for #107.

check_alias accepts a CNAME whose target does not resolve

It matches on the CNAME record rather than following it:

cnames = [_fqdn(v) for v in query_union(record.name, RR_CNAME, resolvers)]
if target in cnames:
    return True, "ok (CNAME)"

So a hostname pointed at a gateway that has since been removed reports
ok (CNAME). Observed during #107's testing: svc2.kvin.wang was CNAME'd to a
gateway hostname that no longer existed, and the check passed. Issuance was
unaffected — dns-01 never connects to the server — but the operator is told the
record is fine when nothing can reach the app.

The early return is deliberate: it exists so the address comparison below, which
also handles a flattened apex ALIAS, is not needed in the common case. The
cheapest improvement is probably to keep returning true but say so — "ok (CNAME,
target does not resolve)" — rather than to start failing, since a gateway that is
temporarily unresolvable should not block certificate renewal.

Worth doing together with whatever handles the tls-alpn-01 side, where an
unresolvable gateway does break issuance, so the two modes may want different
verdicts on the same observation.

[kvinwang]

Status after #107.

Done:

• P1 delegated-record verification → now goes through dnsguide.py (two resolvers, both CAA wire formats, DNS_SETUP_MODE)
• P2 unset_txt_record reporting success on a failed zone lookup
• P3 _ensure_zone_id falling back to a stale cached zone
• P2 "delegated issuance never executed" — run twice now, once locally and once against a real gateway and CVM with traffic verified end to end

Still open:

• set_txt_record deletes every TXT at a name before writing. Invisible today because one -d goes to certbot per invocation; breaks as soon as one certificate covers several identifiers validating at the same name — example.com plus *.example.com, or dstack-ingress: new optional SAN field on certs #86's SAN support. The challenge hook wants add-without-replace, which is the opposite of what the routing records want.
• check_alias accepts a CNAME whose target does not resolve (comment above).
• A delegation token scoped to only the delegation zone. Testing so far used a name under a zone the token already owned, which exercises the mechanism but not the scoping property.
• Renewal through the delegated path — certbot reuses the authenticator and hooks from the renewal config, and that reuse has not been exercised.

Separately, the vmm-cli friction hit while setting up the CVM test is Dstack-TEE/dstack#832.

[kvinwang]

Correction to the previous comment: I overstated the #86 interaction.

#86 adds ALIAS_DOMAIN, a different hostname as a SAN. Two different hostnames
validate at two different _acme-challenge names, so the hook writes to separate
records and nothing is overwritten. #86 does not trigger this.

The collision needs two identifiers that validate at the same name, which in
practice means a base domain and its wildcard on one certificate —
-d example.com -d '*.example.com', where certbot calls the auth hook twice with
the same CERTBOT_DOMAIN and different CERTBOT_VALIDATION. Nothing in this
repo requests that today: process_domain passes a single -d per invocation.

So it stays latent rather than imminent. Still worth fixing before anything
starts issuing multi-identifier certificates, since the failure mode — the
second challenge silently deleting the first — is not obvious from the symptom.