fix(msb): TLS-intercept upstream CA trust behind intercepting proxies - #306
Draft
mogul wants to merge 9 commits into
Draft
fix(msb): TLS-intercept upstream CA trust behind intercepting proxies#306mogul wants to merge 9 commits into
mogul wants to merge 9 commits into
Conversation
Add a read-only host-side diagnostic for the msb TLS-interception failure mode behind an intercepting corporate proxy (Zscaler, Netskope, etc.). msb's --tls-intercept (which acq enables to substitute secrets on the wire) runs a host-side MITM proxy whose upstream leg verifies the real server's certificate against the host trust store via rustls_native_certs load_native_certs() — not the guest bundle, and not --trust-host-cas. When that enumeration does not surface the corporate root, the upstream handshake fails after the guest-facing TLS already completed, so the guest client sees curl (56) unexpected eof for every intercepted host (USAi and GitHub at once). That simultaneous double failure is the fingerprint of this problem rather than a bad or expired key. The script prints where and how the corporate root is trusted on the host (keychains, admin vs user trust-settings domains, native-certs-style counts), whether egress is actually being intercepted (served-leaf issuer), and host reachability — so a working host and a failing host can be compared and the right remedy chosen (e.g. passing the root explicitly via --tls-upstream-ca-cert). Read-only: cert metadata and status codes only, no private keys or secrets. macOS-focused with graceful degradation elsewhere. Refs: #305
…loop On GSA-managed Macs the sudo calls triggered a PIV/SSH askpass that looped on the smart-card PIN prompt and never completed. The privilege was unnecessary: the default keychain search list already includes the System keychain for reads, and root CAs are world-readable — only private keys need privilege, which this probe never accesses. Remove both System.keychain-explicit and admin-trust-settings-domain steps (they required sudo); the unprivileged search-list scan and the user-domain trust-settings dump already establish whether the corporate root is present and how load_native_certs() would see it. Steps renumbered accordingly. Refs: #305
Behind a TLS-intercepting corporate proxy (Zscaler, Netskope) an msb sandbox
failed every outbound HTTPS call — USAi AND GitHub at once — with
curl (56) unexpected eof, which acq previously could only diagnose as a
network problem outside its control.
Root cause (verified against microsandbox source): acq enables
--tls-intercept so msb can substitute injected secrets on the wire. That runs
a host-side MITM proxy which re-originates an upstream TLS connection to the
real server. Behind a corporate proxy that upstream peer presents a
corporate-signed leaf, and msb verifies it against the host native trust store
(rustls_native_certs load_native_certs). --trust-host-cas does NOT feed that
verifier — it only ships CAs into the guest. When load_native_certs does not
surface the corporate root (a known macOS keychain/trust-settings unevenness),
the upstream handshake fails after the guest TLS already completed, so the
guest sees unexpected eof for every intercepted host — the reporter's exact
double failure, and distinct from a 401/expired key.
Fix: when interception is on (the default), pass the corporate/host root CA to
msb's upstream verifier via --tls-upstream-ca-cert. Precedence:
- ACQ_MSB_UPSTREAM_CA_CERT — explicit PEM path(s) (colon/space separated),
passed through verbatim; use on non-macOS or when the root is on disk.
- macOS auto-detect (on; ACQ_MSB_UPSTREAM_CA_AUTODETECT=0 to disable) —
export the host search-list roots to a PEM and trust that upstream,
capturing the corporate root exactly as the host trusts it regardless of
load_native_certs gaps. Unprivileged (no sudo/PIV prompt).
Nothing is emitted when interception is off (ACQ_MSB_NO_TLS_INTERCEPT).
Reproduction toggle: ACQ_MSB_NO_UPSTREAM_CA=1 withholds the upstream CA even
when available, so a host whose proxy does not intercept the acq endpoints can
still reproduce the failing condition and confirm the fix.
Also generalize scripts/diagnose-tls-intercept (from the earlier probe) to
point at this fix, add nine test-acq assertions (explicit CA, repro toggle,
interception-off, and the helper unit), and update BACKEND_GUIDE,
KNOWN_FAILURE_MODES §30, and ADR-0011.
Refs: #305
…rcept Replace em dashes, the section sign, and other decorative UTF-8 with plain ASCII equivalents so the script is byte-clean ASCII. No behavior change; the braille spinner characters used as real spinner output/assertions elsewhere are intentionally left untouched (and are not present in this file). Refs: #305
…nator Field testing disproved the branch's cert-trust theory for quickstart#305: egress still fails with msb interception fully disabled (ACQ_MSB_NO_TLS_INTERCEPT=1), so no --tls-upstream-ca-cert change can fix it. The failure is below msb's TLS layer, in the microVM's egress path. - diagnose-tls-intercept: add step 6 (decisive discriminator: reproduce with interception disabled) and step 7 (localize via a same-host podman-machine VM comparison -> network block vs msb VM-networking bug). Rewrite the header to distinguish Cause A (interception cert trust) from Cause B (VM egress path). - KNOWN_FAILURE_MODES.md section 30: retitle, split into Cause A vs Cause B, mark the Zscaler policy/enrollment explanation as an unverified hypothesis, and add a 'Confirming the mechanism' section listing the host-vs-host checks needed to prove it. - msb.sh / ADR-0011 / BACKEND_GUIDE.md: relabel the --tls-upstream-ca-cert logic as defense-in-depth for the interception case, explicitly NOT the #305 fix. No behavior change. Keeps established facts separate from hypothesis; asserts no cause not yet measured.
…e-tls-intercept Probe each host both default (dual-stack) and IPv4-forced (curl -4) in the step 6 (msb) and step 7 (podman) in-guest checks, and add step 8 which reads the default-vs--4 pair to identify the microsandbox IPv6-first egress stall: on a host with no usable IPv6 route (common under Zscaler force-tunneling) the guest gets a v6 address + AAAA answers, the microVM gateway answers the v6 SYN locally, and Happy Eyeballs never falls back to v4 -> every dual-stack host stalls with unexpected eof. If forcing IPv4 succeeds where default fails, the failure is likely a duplicate of that upstream issue (workaround: force IPv4 / disable guest IPv6). Update KNOWN_FAILURE_MODES section 30 Cause B2 with the IPv6-first variant and the curl -4 discriminator.
The ACQ_MSB_UPSTREAM_CA_CERT path splitter set IFS via a multi-line literal whose first line ended in a literal space+tab. The trailing-whitespace pre-commit hook strips those bytes, which both fails CI and would silently break the split. Rebuild the separator set (colon, space, tab, newline) with printf into a variable so no literal trailing whitespace sits on a source line; a guard char preserves the newline that command substitution would otherwise drop. Verified the resulting IFS is byte-identical (3a 20 09 0a) and splits colon/space/tab/newline-separated paths correctly.
The ACQ_DIAGNOSE_RUN_STEP6=1 auto-run invoked `acq exec <name> sh -c …`, which forwards to `msb exec [OPTS] <NAME> sh -c …`. msb requires a `--` separator before the command, so it rejected the first word with 'error: unexpected argument sh found' and the in-guest interception-disabled probe never ran. Add the `--` so the command reaches the guest.
…ndbox#1344) Field measurement with interception disabled (ACQ_MSB_NO_TLS_INTERCEPT=1) confirmed the failure is Cause B2: the msb guest fails curl (56) unexpected eof on every host over both IPv4 and -4, while the host process and a same-host podman machine VM (v4 and v6) reach everything. This rules out Cause A (cert trust), microsandbox#1226 (IPv6-first: -4 fails identically), and B1 (blanket network block: podman works). Filed upstream as microsandbox#1344. Update section 30 with the measured evidence table, the ruled-out list, and the upstream issue link; keep the Zscaler policy mechanism marked unverified.
mogul
force-pushed
the
fix/msb-tls-intercept-upstream-ca
branch
from
August 13, 2026 23:34
d3b8b7e to
3b0dab2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Diagnostic-first work for #305 (
msbbackend: every outbound HTTPS call — USAi, GitHub, npm — fails at once withcurl: (56) ... unexpected eof).This PR does not claim to fix #305. Field evidence disproved the original cert-trust theory for the reporter, so this PR instead ships the tooling and documentation to classify the failure, plus the cert-trust handling relabeled as defense-in-depth for a different (genuinely-intercepted) case.
What changed
scripts/diagnose-tls-intercept— host-side, read-only, unprivileged probe (cert metadata + HTTP status codes only, no secrets):ACQ_MSB_NO_TLS_INTERCEPT=1). If egress still fails while the host reaches everything, it is Cause B (network/egress path, not certs) — no cert change can help.podman machineVM. Podman reaching what msb cannot → B2 (msb-specific libkrun/vmnet path, report upstream); nothing reaching it → B1 (blanket network block, security-admin matter).curl -4); if forcing IPv4 succeeds where default fails, it is likely a duplicate of #1226 (workaround: force IPv4 / disable guest IPv6).docs/KNOWN_FAILURE_MODES.md§30 — rewritten to separate Cause A (interception cert trust) from Cause B (network egress block, B1/B2 sub-cases, incl. the #1226 IPv6-first variant), keeping established facts distinct from the still-unverified Zscaler-policy hypothesis.acq.backends/msb.sh/ ADR-0011 /docs/BACKEND_GUIDE.md— the--tls-upstream-ca-certhandling is retained as defense-in-depth for genuinely-intercepted endpoints (Cause A) and explicitly relabeled as NOT the [Bug] Initial setup fails on gfe #305 fix. No behavior change: it still emits--tls-upstream-ca-certwhen interception is on.Why the original cert theory is not the #305 fix
ACQ_MSB_NO_TLS_INTERCEPT=1run fails identically — with interception off there is no--tls-interceptand no--tls-upstream-ca-cert; evennpm install(no secrets, no CA) still fails. If the whole interception mechanism is gone and it still fails, no cert change can be the fix.--tls-upstream-ca-certto validate.Cause: confirmed Cause B / B2 (upstream msb bug), filed as microsandbox#1344
Field measurement (interception fully disabled,
ACQ_MSB_NO_TLS_INTERCEPT=1) settled it:podman machineVM (v4 and v6)-4)curl -4fails identically to dual-stack.Confirmed as an msb-specific libkrun/vmnet guest-egress bug (B2) and filed upstream as superradcompany/microsandbox#1344. There is no acq-side fix; this PR's role is to diagnose and classify (probe + docs), and the
--tls-upstream-ca-certcode remains defense-in-depth for the genuinely-intercepted Cause A only.Verification
bash -nclean onacq,acq.backends/*.sh,scripts/test-acq,scripts/diagnose-tls-intercept.shellcheck --severity=warning, one file per invocation (per repo convention): all clean, no findings../scripts/test-acq: 857 passed, 0 failed (the [Bug] Initial setup fails on gfe #305 assertions unchanged).npm run lint:md: 0 errors.Rollback
Revert the branch commits; the only behavioral surface is the
--tls-upstream-ca-certflag (already gated behind interception-on and overridable viaACQ_MSB_*env vars). The probe script and docs are additive.Security impact
No change to authn/authz or data handling. The probe is read-only and emits only cert metadata + HTTP status codes (no secrets, no private keys, no env dump). The retained
--tls-upstream-ca-certpath only adds the host's existing corporate root to msb's upstream verifier when interception is already enabled.AI-assisted (OpenCode).