[Bug]: A timed-out provider status probe is cached as an error, so the provider stays unavailable for new sessions

[tsouth89]

Before submitting

• I searched existing issues and did not find a duplicate.
• I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

1. Configure several provider instances (I run five: two Claude, one Codex, one OpenCode, one Cursor).
2. Put the host under sustained load. Running a fleet of subagents does it; so does any workload that keeps the CPU and disk busy for tens of seconds.
3. Wait for a provider health refresh, or open Settings → Providers and refresh.
4. Try to start a new thread.

The trigger is CPU/disk contention on the host, not anything about a particular provider. It is easiest to hit on Windows, where CLI cold starts are slowest.

Expected behavior

A status check that runs out of time should leave the provider as it was and try again on the next refresh. "We could not check in time" is not the same fact as "this provider is broken".

Actual behavior

The provider flips to an error state and cannot be used to start a new session, and it stays that way well after the host is idle again.

Each status check runs a CLI subprocess under a short fixed deadline:

• apps/server/src/provider/providerSnapshot.ts:21 — DEFAULT_TIMEOUT_MS = 4_000 (version probes)
• apps/server/src/provider/providerSnapshot.ts:23 — AUTH_PROBE_TIMEOUT_MS = 10_000 (Codex auth)
• apps/server/src/provider/Layers/CursorProvider.ts:581 — ABOUT_TIMEOUT_MS = 8_000
• apps/server/src/provider/Layers/GrokProvider.ts:44 — VERSION_PROBE_TIMEOUT_MS = 4_000

When the deadline passes, each provider builds a snapshot that asserts failure rather than reporting that the check did not finish:

• CodexProvider.ts → status: "error", version: null, and models: emptyModels, message "Timed out while checking Codex app-server provider status."
• ClaudeProvider.ts → status: "error", message "Claude Agent CLI is installed but failed to run. Timed out while running command."
• CursorProvider.ts and GrokProvider.ts → the same shape.

That snapshot is then treated as authoritative:

1. makeManagedServerProvider.applySnapshotBase publishes it, replacing the last good snapshot.
2. ProviderRegistry.ts:324 writes it to the per-instance status cache on disk.
3. ProviderRegistry.ts:279 hydrates from that cache on the next boot.

So a single busy moment is persisted and survives a restart. The provider keeps showing an error until some later check happens to win the race. For Codex the model list is emptied too, which is why the new-session path has nothing to offer.

Two things make this easy to hit rather than rare:

• Every configured instance runs its own refresh loop, so a multi-instance setup spawns several CLI probes at roughly the same time. Those probes compete with each other and with whatever else is loading the machine.
• The deadlines are fixed constants with no retry and no distinction between "no answer yet" and "answered, and the answer is bad".

Impact

Major degradation or frequent failure

Version or commit

main @ 4c1d99d

Environment

Windows 11, T3 Code Nightly, five configured provider instances (Claude x2, Codex, OpenCode, Cursor)

Logs or stack traces

Provider snapshot after a timed-out check (shape, from the source path above):

  status:  "error"
  version: null
  models:  []            # Codex only; other providers keep their fallback list
  message: "Timed out while checking Codex app-server provider status."

The same object is what gets written to <providerStatusCacheDir>/<instanceId>.json.

Screenshots, recordings, or supporting files

No response

Workaround

Open Settings → Providers and hit refresh once the machine is quiet. It usually recovers on the first successful check. Restarting the app does not help by itself, because the error snapshot is read back from the on-disk cache.

Related

#5794 raises the Windows probe deadlines. That reduces how often the race is lost, and it is worth having, but it does not change what happens when a probe does time out: the error snapshot is still published and still cached. The two are complementary.

Timeouts in OpenCodeProvider are a separate gap; #6577 adds them.