fix: trust the OS certificate store so login works behind TLS proxies#205
Merged
Conversation
Users behind a corporate proxy hit this at the Login step:
Login failed: fetch failed (self-signed certificate in certificate chain)
Node verifies TLS against its own bundled Mozilla CA snapshot and never
consults the OS trust store (tls.rootCertificates: "fixed at release
time... identical on all supported platforms"). A TLS-intercepting proxy
(Zscaler/Netskope/Fortinet) or HTTPS-scanning AV re-signs every chain
with a corporate root that MDM/GPO installed into the *OS* store — so
the user's browser works and we fail. That per-machine variation is why
only some users are affected.
The error code pins it: SELF_SIGNED_CERT_IN_CHAIN is an untrusted
self-signed ROOT in the presented chain (interception), not a
self-signed leaf, which reports DEPTH_ZERO_SELF_SIGNED_CERT instead.
Merge the OS store into Node's default CA set via
tls.getCACertificates("system"), which reads that store even without the
--use-system-ca flag — so affected users need no configuration.
Load-bearing details:
- Merge "default" + "system", never "bundled" + "system". "default"
already folds in NODE_EXTRA_CA_CERTS; narrowing it would silently
drop the certs of users who fixed this the documented way.
- Call it from loggedFetch, not at startup: it costs ~25ms (a native
OS-store read) and commands that never open a socket shouldn't pay.
Every network path goes through loggedFetch (16 call sites, zero raw
fetch), so first-request is the cheapest and completest hook.
- Every failure mode is a no-op, including an empty system store.
Trust config isn't ours to have opinions about, and a bad merge would
break users whose certs already work.
- The APIs need Node >=22.19/24.5 (our floor is 22.5), hence
tlsApi.supported(). Accessed off the default import, never
destructured — a named ESM import of a missing builtin export is a
link-time error on older Node.
Also add describeNetworkError: unwrap Node's bare `fetch failed` (the
real reason hides on err.cause) and append a remedy for cert codes. Node
only volunteers its own --use-system-ca hint for DEPTH_ZERO /
UNABLE_TO_VERIFY_LEAF / UNABLE_TO_GET_ISSUER, which pointedly excludes
SELF_SIGNED_CERT_IN_CHAIN — the corporate-proxy case it most exists for
prints nothing. When a cert error survives the merge the root isn't in
the OS store either, so the hint names NODE_EXTRA_CA_CERTS rather than
--use-system-ca, which would be a dead end.
Verified by driving the real loggedFetch against a proxy-shaped chain
(leaf + untrusted self-signed root): with the root in the OS store login
now succeeds (HTTP 200) where it previously failed with the exact
reported error; without it, the error carries an actionable remedy.
Note the blast radius: this covers our own process only. `npm i -g`
during install and the agents themselves are separate processes behind
the same proxy and need NODE_EXTRA_CA_CERTS / npm's cafile of their own.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The eager merge regressed Windows CI. tls.getCACertificates("system") is
synchronous and costs ~20ms on macOS but ~300ms+ on Windows, where it
blocks the event loop — and loggedFetch ran it before the first request
of every command, including in tests.
Windows per-file durations vs the same runner on #204:
backend.test.ts 172ms -> 511ms (+197%)
TaskList.test.tsx 7,644ms -> 19,541ms (+156%)
ConfigApp.test.tsx 10,091ms -> 17,240ms (+71%)
InstallApp.test.tsx 57,903ms -> 87,657ms (+51%)
Everything slowed, including TaskList, which never fetches — the stalls
starved Ink's render timers, and the three tests with the tightest budgets
failed. Not a test-only problem: every networked command on Windows would
have eaten the same stall.
Invert it. Run the request first; only when it fails with a cert error do
we merge and retry once (fetchTrustingSystemCa). A cert error is a precise
signal that the user is one of the affected minority, so the happy path
now costs exactly zero and only affected users pay. applySystemCaCertsOnce
returns null once it has run, bounding this at one retry per process — a
chain that stays untrusted surfaces its error instead of looping. Replay
is safe: a TLS handshake fails before any body is sent, and every call site
passes a replayable body (string/URLSearchParams/FormData/Buffer).
certHint is now pure for the same reason: reading the store to word a
sentence would put the stall back on the error path, and the retry already
read it.
Tests pin both halves — a successful request never touches the OS store,
and a cert failure merges, retries once, then gives up. Re-verified
end-to-end: with the root in the OS store login still succeeds (HTTP 200);
without it the error still carries the remedy; the store is read only on
the failure path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The report
Users hit this at the Login step of
codevhub install:Root cause
Node verifies TLS against its own bundled Mozilla CA snapshot and never consults the OS trust store (tls.rootCertificates: "fixed at release time… identical on all supported platforms"). A TLS-intercepting proxy (Zscaler/Netskope/Fortinet) or HTTPS-scanning AV re-signs every chain with a corporate root that MDM/GPO installed into the OS store — so the user's browser works and we fail. That per-machine variation is exactly why only some users are affected.
The error code pins it. Fetching two local servers through the same path as
loggedFetch:fetch failed (self-signed certificate in certificate chain)—SELF_SIGNED_CERT_IN_CHAINfetch failed (self-signed certificate; …try --use-system-ca)—DEPTH_ZERO_SELF_SIGNED_CERTThe first matches the report exactly: an untrusted self-signed root in the chain — interception, not a misconfigured server.
The fix
src/lib/tls.tsmerges the OS store into Node's default CA set viatls.getCACertificates("system"), which reads that store even without the--use-system-caflag. Affected users need no configuration.The merge happens only after a certificate failure, never up front.
loggedFetchruns the request; if it fails with a cert error,fetchTrustingSystemCamerges and retries once. This matters — the first revision of this PR merged eagerly and regressed Windows CI badly (see below).Other load-bearing details:
applySystemCaCertsOncereturns null once it has run, so a chain that stays untrusted surfaces its error instead of looping. Replay is safe: a TLS handshake fails before any body is sent, and every call site passes a replayable body (string/URLSearchParams/FormData/Buffer), never a stream.default+system, neverbundled+system."default"already folds inNODE_EXTRA_CA_CERTS(verified: 146 = bundled 145 + the extra one), so users who fixed this the documented way keep working. This was the one way the fix could itself cause a regression.setDefaultCACertificatesthen never runs, keeping Node byte-identical). Trust config isn't ours to have opinions about, and a bad merge would break users whose certs already work.tlsApi.supported(). Accessed off the default import, never destructured — a named ESM import of a missing builtin export is a link-time error on older Node.Why merge-on-failure, not merge-up-front
The first revision merged before the first request.
tls.getCACertificates("system")is synchronous and costs ~20ms on macOS but ~300ms+ on Windows, where it blocks the event loop. Windows per-file test durations against the same runner on #204:backend.test.tsTaskList.test.tsxConfigApp.test.tsxInstallApp.test.tsxEverything slowed, including
TaskList, which never fetches — the stalls starved Ink's render timers, and the three tests with the tightest budgets failed. That was never just a test problem: every networked command on Windows would have eaten the same stall.Inverting it makes a cert error the trigger, which is a precise signal that the user is one of the affected minority. The happy path now costs exactly zero, and only affected users pay. A test pins that a successful request never touches the OS store.
The hint
describeNetworkErrorunwraps Node's barefetch failed(the real reason hides onerr.cause) and appends a remedy for cert codes. Now used at all three login surfaces (components/Login.tsx— the one in the report,components/AdminLogin.tsx,LoginApp.tsx), replacing hand-rolled unwrapping.It exists because Node's own
--use-system-cahint pointedly excludesSELF_SIGNED_CERT_IN_CHAIN—crypto_common.ccgates it onDEPTH_ZERO_SELF_SIGNED_CERT/UNABLE_TO_VERIFY_LEAF_SIGNATURE/UNABLE_TO_GET_ISSUER_CERTonly (visible in the table above: only the self-signed-leaf case got a hint). The corporate-proxy case, the one the hint most exists for, prints nothing. When a cert error survives the merge the root isn't in the OS store either, so our hint namesNODE_EXTRA_CA_CERTSrather than--use-system-ca, which would be a dead end.Verification
Drove the real
loggedFetchagainst a proxy-shaped chain (leaf + untrusted self-signed root):Unit tests cover the merge, the one-retry bound, the old-Node path, and that every failure mode leaves TLS untouched.
pnpm fix/typecheck/test(944 passing) /build && node dist/index.js --versionall green, Windows CI included.Note for reviewers
Blast radius is our process only.
npm i -gduring install and the agents themselves are separate processes behind the same proxy, and will still fail on their own TLS — they needNODE_EXTRA_CA_CERTSor npm'scafile. Worth confirming with an affected user whether they now get past login and stall at agent launch; if so, that's a follow-up.🤖 Generated with Claude Code