feat(dstack-ingress): TLS-ALPN-01 mode that needs no DNS credentials - #105
Conversation
dstack-ingress full functional test reportBranch 32/32 passed. Four defects found and fixed. 1. What was exercised, and against whatNothing here is mocked. Every certificate was issued by Let's Encrypt staging
The A-group path in full: 2. ResultsA. tls-alpn-01 on real TDX hardware — 14/14
A6 is worth singling out because it is the mechanism the whole mode rests on, A9/A10/A12 together close the attestation chain: Intel root → quote B. dns-01 regression, real Cloudflare — 10/10
B7 is the regression for the C. Negative paths and configuration — 8/8
C5 turned out to be the strongest anti-impersonation property in the feature {"payload": "<canonical JSON>", "hmac_sha256": "…", "attestation": {"quote": "…", …}}Verified independently: the HMAC matches the payload under the shared token, C7 was incidental. A container advertising an instance ID the gateway has 3. Defects found and fixedFour, all found by this run, all now committed and pushed. 1. dns-01 ran its entire first pass twice ( dns-01 issues before it serves, so Inherited from upstream, where bootstrap and the renewal daemon both ran at 2. CAA rejection reasons printed twice ( The union across resolvers deduplicates raw rdata, but Cloudflare returns CAA in 3. The variable was wired into This is also the answer to why the dns-01 renewal branch was initially untested. Fixed by writing the setting before 4. The placeholder path reported itself as a failure ( A first tls-alpn-01 run opened with: Skipping is correct — it is what leaves the placeholder in place so haproxy can 4. Observations not fixedJudgement calls, listed so they are decisions rather than oversights.
5. Not covered
6. CleanupTest CVM removed, lab iptables rules 0, all test DNS records deleted from |
Follow-up: wildcard rejection was correct but too lateChecked on the question "does tls-alpn-01 actually reject wildcards". It did —
Mixed configurations did contain the damage correctly — with Now checked once at the top of Three log lines and exit 1 — before the venv, the placeholder, or the evidence It lives in Deliberately fatal for the whole container rather than skipping the one domain — The per-domain check went with it: the domain list comes from env vars fixed for Verified: pure wildcard, mixed config, dns-01 wildcard (unaffected), and a normal |
b7d144e to
098741c
Compare
…let the gateway's DNS cache expire
dns-01 issues before it serves, so run_pass is called explicitly at startup and cert_loop then ran it again immediately. For a single domain that is four certbot invocations and two rounds of DNS writes on every container start, all to reach a state the first pass had already reached. The duplicate is inherited from upstream, where bootstrap and the renewal daemon both ran at startup with no initial sleep; the refactor kept it faithfully. tls-alpn-01 never had it because there is no bootstrap pass to duplicate. Also deduplicate CAA records after parsing rather than before. Resolvers disagree on the wire format -- Cloudflare returns RFC 3597 generic hex, Google the presentation form -- so one record arrives as two distinct strings and its rejection reason was printed twice.
…failure
On a first tls-alpn-01 run the log read:
Generating placeholder certificate for app.example.com
Warning: Cert files missing for app.example.com, skipping
The warning is what build_combined_pems prints whenever lego has no
certificate yet, which on the first pass is exactly the situation the
placeholder exists to cover. Nothing is wrong -- skipping is what keeps
the placeholder in place so haproxy can bind and forward acme-tls/1 --
but back to back with the line above it reads like the placeholder was
discarded, and it is the first thing a new deployment shows.
Distinguish the two cases: keeping a placeholder is normal, having
neither a certificate nor a placeholder is not. dns-01 has no
placeholder, so its copy keeps the original wording.
…t testing RENEW_DAYS_BEFORE was wired into legoman.py only, so it silently did nothing on the dns-01 path -- while the README listed it in the shared table, with every other tls-alpn-01-only setting explicitly marked as such and this one not. The asymmetry also cost test coverage. lego takes the renewal window as a flag, which makes "certificate is due" reachable on demand and lets both renewal branches be exercised. certbot has no equivalent flag: it reads `renew_before_expiry` from the lineage's renewal config. With nothing writing that, a fresh certificate could not be made due, and the dns-01 renewal path could only ever be observed in its "nothing to do" branch. Write the setting before `certbot renew` so the variable means the same thing in both modes. Unset keeps certbot's own 30-day default. TESTING.md documents how to exercise this component against real infrastructure -- three tiers by cost, what each can and cannot observe, how to reach both renewal branches, and the traps that make a test pass for the wrong reason (leftover records changing which branch runs, single-resolver negative caching, gateway TXT caching).
… pass A wildcard under tls-alpn-01 can never succeed -- RFC 8737 forbids the challenge for wildcard identifiers -- so it is a configuration error, not a runtime failure. process_domain did reject it, but only once the certificate loop reached that domain, by which point the container had generated a placeholder certificate for a name that will never get a real one, started haproxy serving it, and settled into retrying an unsatisfiable config every 60s to 1800s forever. Check it once at the top of tlsalpn.sh, before the startup sequence that does all of the above. It belongs to tls-alpn-01, not to entrypoint.sh: that file validates what both modes share and dispatches, and putting a CHALLENGE_TYPE branch back in it is the mode-specific logic the split was meant to remove -- HTTP-01 would then add a third. Deliberately fatal for the container rather than skipping the one domain. A mixed config did already contain the damage -- the other domains were issued normally -- but it left a self-signed certificate served for the wildcard indefinitely and a permanently failing loop, which reads as a TLS bug rather than a typo. The per-domain check is gone with it: the domain list comes from env vars fixed for the container's lifetime, so it saw exactly what the startup check saw and could not fire. legoman.py keeps its own check, which is a real boundary -- it guards the ACME call and can be invoked standalone. dns-01 wildcards are unaffected.
…ls-alpn-01 ACME_STAGING was read only on the tls-alpn-01 path. Under dns-01 nothing looked at it -- certman.py and functions.sh both read CERTBOT_STAGING -- so setting it there silently issued *production* certificates. The README did not mark it mode-specific, so there was nothing to warn you. Same shape as the RENEW_DAYS_BEFORE bug, and worse in effect: instead of a knob that does nothing, this one does the opposite of what it says on a path where the mistake costs a real certificate and real rate limit. Normalise both names once in entrypoint.sh and export them. This does belong there, unlike the wildcard rule: an ACME account and a staging toggle are genuinely shared by both modes, which is exactly what that file validates. The per-mode copies in dns01.sh and tlsalpn.sh are gone. Name them after ACME rather than certbot throughout. What you configure is an ACME account; which client implements a mode is an implementation detail, and this image already changed it once. CERTBOT_EMAIL and CERTBOT_STAGING keep working -- they are the historical names, not the preferred ones, and the README no longer implies the preference depends on the mode. Also fixes the webhook example, which showed the payload's keys in an order the code does not emit (it sorts them), and drops a stale duplicate comment on ensure_placeholder_certs that credited certbot with forwarding the tls-alpn-01 challenge.
…restructure #104 added ACME DNS-01 challenge delegation to entrypoint.sh. This branch moves that file's logic into dns01.sh, so a plain rebase drops the shell half of it: set_alias_record / set_txt_record / set_caa_record kept this branch's versions and the delegation branches went with them. Port them into dns01.sh, which is where the delegation belongs anyway -- it is a dns-01 capability, and entrypoint.sh no longer holds mode-specific logic. Everything else from #104 survived the rebase untouched: the hook script, unset_txt_record, the dnsman.py action, the certman.py branch and the README section. Two of the ports are deliberate substitutions, not copies: - The CAA helpers reuse this branch's caa_tag_for() and "${domain#\*.}" instead of re-introducing caa_domain_and_tag(). Behaviour is identical for both wildcard and plain names; the second helper would only be a second place to keep them in sync. - The delegated TXT instruction prints $(txt_record_value) rather than "$APP_ID:$PORT". APP_ID is an optional override, so the original printed ":443" whenever it was unset -- which is the common case. Printing the same function that writes the record in non-delegation mode also stops the instruction drifting from what the gateway actually expects. Two more changes are integration fixes: bugs neither PR has alone, only the combination. - The delegation branch built its certbot command with CERTBOT_STAGING, which this branch renamed to ACME_STAGING. ACME_STAGING=true plus delegation would have issued *production* certificates. - It also passed --email unconditionally, which this branch made optional; with no contact address configured that is `--email ""`. It now uses the same optional-contact handling as the plugin path. Verified against a delegation run: the served zone is untouched, all four static records print with correct values, the certbot command carries --manual with both hooks plus --staging and --register-unsafely-without-email, CAA verification fails closed with exit 1, and ALLOW_MISSING_CAA=true overrides it.
2511933 to
42a7bbd
Compare
Problem
dstack-ingresscan only obtain certificates through DNS-01, which means the tenant hands a DNS API token to the CVM. That token is zone-wide (Cloudflare needs Zone:Read + DNS:Edit), it lives inside the enclave for the whole life of the deployment, and it is the largest thing a customer has to trust us with before putting their own domain in front of a dstack app.TLS-ALPN-01 answers the challenge on the TLS port the ingress already owns, so certificate issuance and renewal never touch DNS.
Fix
New
CHALLENGE_TYPE=tls-alpn-01. Thedns-01path keeps the same certbot, the same providers and the same ordering.No DNS credentials, so the container guides instead of writing.
dnsguide.pycomputes the exact CNAME / TXT / CAA records, prints them, and by default polls public DNS until they are visible before letting issuance start.DNS_SETUP_MODEselectswait/print/webhook; the webhook lets an operator service create the records automatically.The TXT record names an instance, not an app. Under
dns-01it carries the app id and the gateway load-balances across every instance. That cannot work here: the challenge is answered by whichever instance holds the ACME order, while the gateway races connections across the app's instances (select_top_n_hosts→connect_multiple_hosts) and Let's Encrypt validates from several vantage points at once — measured on this branch, 5 distinct source IPs within ~2 seconds on every issuance. With more than one instance the challenge lands on the wrong replica almost every time. So this mode publishes the instance id, with the two consequences documented: effectively single-instance, and the record changes on every instance replacement (which is what the webhook mode is for).haproxy keeps the public port. tls-alpn-01 needs the ACME responder to complete the TLS handshake itself, so the public port becomes a plain TCP frontend that peeks at the ClientHello and forwards only
acme-tls/1to lego on loopback; everything else goes to the real TLS frontend, moved to loopback behind PROXY protocol so the backend still sees the real client. Ordering is load-bearing: haproxy has to be listening before the first certificate can exist, so it starts on a self-signed placeholder and reloads onto the real certificate when it arrives.Structure: one top-level script per challenge type
The two modes have little in common — different ACME client, different on-disk layout, different DNS story, different proxy topology, different startup order. An earlier revision expressed that with
CHALLENGE_TYPEbranches spread across four shared files (22 of them). This version gives each mode its own program:The libraries contain no mode branch at all; each mode composes them:
dns01.shis close to a move of the original top-level script, so the dns-01 path reads as "unchanged, relocated". Adding the HTTP-01 relay later is a new file rather than a third branch in four files.Why lego and not certbot
certbot cannot do tls-alpn-01, and this is not a version to wait for:
[challenges.HTTP01]fromget_chall_pref;acmelibrary removed the challenge in 4.2.0. The last release carryingacme.standalone.TLSALPN01Serveris 4.1.1, where the class is already marked.. deprecated:: 4.1.0.A certbot authenticator plugin against 4.1.1 was built first and does load and register — but only by pinning both
certbotandacmeto 4.1.1 (pinning certbot alone resolvesacme5.x and the import fails). Running a component whose entire job is TLS on a deleted, unmaintained API is not a good trade, so this uses lego, pinned by checksum in the Dockerfile.The ACME contact address is optional
ACME_EMAIL(falling back toCERTBOT_EMAIL) may be left unset in either mode. RFC 8555 makescontactoptional and Let's Encrypt stopped sending expiry mail in 2025. It also does not stay private: the account document is published as evidence at/evidences/acme-account.json, so an address set here is readable by anyone who fetches it. lego simply omits the flag; certbot needs--register-unsafely-without-email, which the container passes for you.Bugs found while testing
Most of these are in code this PR adds. Two are pre-existing and were fixed because the new design depends on them.
Bootstrap raced the renewal daemon. Certificates had two drivers — a one-shot
bootstrapand a 12-hourly daemon — that both calledrenew-certificate.sh. That was only safe because bootstrap finished before the daemon started, an ordering that stops holding once tls-alpn-01 forces bootstrap to run behind a live proxy. Both ACME clients then bound the same challenge port, and the loser still spent one of Let's Encrypt's five failed validations per hostname per hour. Collapsed into one idempotent pass used for both jobs.renew-certificate.shalways exited 0 (pre-existing), so every pass reported "something changed" and regenerated evidence plus reloaded haproxy every 12 hours regardless. Now propagates 0 changed / 2 nothing to do / 1 failed.Public DNS being correct is not the same as the gateway acting on it. The gateway resolves the app-address TXT itself and caches it for the record's TTL, so right after the value changes — every instance replacement, since the record names the instance — it can still route the challenge to the previous target. Observed exactly that: the DNS check passed, then the CA reported
Error getting validation data. AddedDNS_SETTLE_SECONDS(default 30, first pass only).One DoH resolver is not enough to trust either answer. The CAA check was silently passing on a name that did restrict issuance. Two independent causes: one resolver kept serving a stale negative answer for a CAA record added after it was first queried, and the other returns CAA in RFC 3597 generic form (
\# 47 00 05 69 73 73 75 65 ...) rather than presentation form, which the parser did not handle. Now two resolvers with deliberately opposite quorums — propagation checks (CNAME, TXT) require every resolver to agree, the CAA permission check takes the union so any resolver seeing a restriction stops us — and both rdata encodings parse identically (unit-tested).Evidence files were served 403 in this mode. lego writes its account document and certificates 0600 and
cpkeeps that mode; certbot's equivalents are 0644. Neither carries a private key. The evidence server is also now bound to loopback: its only consumer is haproxy'sbe_evidencebackend, and listening on every interface additionally exposed it to other containers on the compose network, bypassing the proxy. That also silences a long-standingbind: Address already in use— mini_httpd binds[::]:portfirst and withnet.ipv6.bindv6only=0that socket already covers IPv4, so its second bind fails. The same line appears on the publisheddstacktee/dstack-ingress:2.2image; it was never a port conflict.set -ekilled the loop on a quiet renewal.( process_domain ); status=$?looks like it captures the status, but a bare non-zero command underset -eterminates the script — so the first renewal pass that returned 2 ("nothing to do") would exit the container. Only reachable 12 hours after deployment, which is why every earlier test missed it. Now evaluated inifcontext.Renewal detection was matching lego's log text.
"no renewal"is what lego 4.x printed; 5.3.1 printsSkip renewal: ..., so every renewal pass was reported as a change and would have reloaded haproxy every 12 hours forever — reintroducing bug 2 through the back door. Replaced with lego's own signal:--deploy-hookruns only "in cases where a certificate is successfully created/renewed". The decision itself stays with lego on purpose, because it consults the CA's ARI endpoint (RFC 9773) and the CA can ask for early renewal during an incident; deciding here fromnotAfterwould override that.Only the second of the two dns-01 issuance attempts was reported. dns-01 issues, writes the CAA record naming the now-existing account, then issues again. Reporting the second attempt's status means a clean first deployment ends on "nothing to renew" (2), so the pass looks quiet and evidence generation is skipped entirely —
/evidencesempty on first deploy. Earlier regression runs missed it because a leftover CAA record from the previous run made attempt one fail and attempt two succeed; cleaning the zone exposed the normal path. Now either attempt producing a certificate counts as a change.Verification
End-to-end against Let's Encrypt staging on a real TDX CVM (qemu
confidential-guest-support=tdx, QGS over vsock) through a realdstack-gateway:tls_peek be_tls_terminate/local→tls_in~ be_upstream/app1, source10.44.0.1= the gateway's WireGuard address). Evidence endpoint and all three evidence files return 200.Skip renewal, hook does not fire, no reload; forced due viaRENEW_DAYS_BEFORE=365→Renewing→Server responded with a certificate→ hook fires →HAProxy reloaded with the new certificates.dns-01regression: full pass with real Cloudflare credentials — CNAME → TXT (app id) → issuance → CAA (correctlyvalidationmethods=dns-01) → second attempt reports nothing to renew → evidence, combined PEM and the 12-hour sleep all reached. Repeated with no contact address, confirming--register-unsafely-without-emailand an account document with nocontact.ROUTING_MAP), 42–64 lines each. The restructure changes no proxy behaviour.scripts/tests/test_dnsguide.pyadds 16 tests covering both CAA rdata encodings, the CAA permission matrix, TXT normalisation and record building — the first Python tests in this directory.scripts/tests/test_sanitizers.shstill passes.Limitations, documented in the README
dns-01.dns-01deployment saysvalidationmethods=dns-01and will make the CA refuse. The container checks this before requesting a certificate so the failure is legible instead of an opaque validation error.cert-datavolume changes the account URI, and unlikedns-01the container cannot rewrite a pinnedaccounturiCAA record itself.Testing
custom-domain/dstack-ingress/TESTING.mddocuments how to exercise thiscomponent against real infrastructure: three tiers by cost, what each can and
cannot observe, how to reach both renewal branches, and the traps that make a
test pass for the wrong reason.
A full functional run of this branch — 32 cases against Let's Encrypt staging on
real TDX hardware — is in a comment below.
It found four more defects, all fixed here:
same record deduplicated as two)
RENEW_DAYS_BEFOREsilently did nothing on dns-01, which is also why thatmode's renewal branch had no way to be tested on demand