Skip to content

fix(core): contain thrown prewarm errors in the cachekeep tick - #155

Open
iceteaSA wants to merge 1 commit into
cortexkit:mainfrom
iceteaSA:fix/cachekeep-tick-throw
Open

fix(core): contain thrown prewarm errors in the cachekeep tick#155
iceteaSA wants to merge 1 commit into
cortexkit:mainfrom
iceteaSA:fix/cachekeep-tick-throw

Conversation

@iceteaSA

@iceteaSA iceteaSA commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Fixes #150.

sendPrewarm() calls fetch with signal: AbortSignal.timeout(...), which throws on timeout or a network error instead of returning a result. Nothing between that throw and start()'s .catch() handled it, so a single throwing target aborted the entire tick():

prewarm attempts: ["A"]        <- B and C never attempted
tick +1m attempts: ["A"]
tick +2m attempts: ["A"]
tick +3m attempts: ["A"]

And it never recovered: the backoff assignment lives in the !result.ok branch that a throw skips, so the failing target's cacheExpiresAt was never advanced and it stayed first-due, re-aborting every subsequent tick. Later targets were starved until they aged out as stale ~2h later, and publishTrackedSessions() never ran, so the cross-process lease stopped refreshing.

The change

  • sendPrewarm() catches the fetch throw and returns { ok: false, reason, transient: true }.
  • prewarm() routes transient failures to the existing backoff branch instead of the delete branch.
  • tick() wraps each prewarm() call defensively, applying the same backoff, so a future unanticipated throw cannot reintroduce head-of-line blocking.

The delete path was the trap here. prewarm() used result.status == null to mean "unbuildable body, drop the tracked session", so mapping a thrown fetch onto that shape would have made a transient network blip permanently delete a live session — worse than the bug being fixed. Hence the separate transient discriminator: transient: true is set at exactly one site, and the delete predicate is now result.status == null && !result.transient. Unbuildable bodies still delete; HTTP-status failures still back off, unchanged.

Two things worth flagging

  • prewarmNow() is a behavior change for consumers. It previously rejected on a fetch throw; it now resolves { ok: false, transient: true }. The only in-repo consumer (warmFableAfterOpus) already handles both, but this is a public surface of @cortexkit/anthropic-auth-core, so it's a contract change worth knowing about.
  • loadStorage() at the top of tick() is still unguarded and can abort a tick. I left it alone: it's pre-existing, start()'s .catch() handles it, it retries in 60s, and skipping a pass when the schedule can't be read is arguably right. Happy to wrap it if you'd rather.

Verification

New tests assert that a throwing target does not prevent later targets from being prewarmed, that the throwing target is retained rather than deleted, that it backs off instead of retrying on the next tick, and that an unbuildable body still deletes its target.

Proven red before green — reverting the source change makes the new tests fail (2 fail, with the unbuildable-delete test correctly still passing as a regression guard). Gates on the branch: bun run typecheck clean, bun run test 1025 pass / 0 fail, bun run lint clean.

Two other defects in this file are filed separately — #149 (failing prewarms retried at a fixed cadence forever) and #151 (ticks overlapping into duplicate paid prewarms). Kept independent so they can be triaged on their own; happy to send PRs for either if you want them.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Contain thrown prewarm errors during cachekeep ticks to prevent head-of-line blocking and session starvation. Previously, a timeout/network error from fetch in sendPrewarm() threw and aborted the entire tick(); now these errors are caught, treated as transient failures, and the tick proceeds to later targets.

Behavior changes and migration

  • sendPrewarm() catches fetch throws and returns { ok: false, reason, transient: true }; prewarm() treats transient failures as backoff, not delete; tick() wraps per-target calls to apply the same backoff on unexpected throws.
  • prewarmNow() no longer rejects on network/timeout errors. It resolves a failure result with transient: true. Migration: update callers to check result.ok (and optionally result.transient) instead of catching a rejection.
  • Delete-on-prewarm now only applies to unbuildable bodies (status == null and not transient). Transient/network failures never delete tracked sessions.

Written for commit e9afa64. Summary will update on new commits.

Review in cubic

Greptile Summary

The PR contains fetch failures within cachekeep prewarming so one failed target no longer aborts the tick or starves later targets.

  • Adds an explicit transient-failure result for fetch exceptions.
  • Preserves transient targets and advances them using the existing retry backoff rather than deleting them.
  • Defensively isolates each target during a tick and adds regression coverage for continuation, retention, backoff, publication, and unbuildable-body deletion.

Confidence Score: 5/5

The PR appears safe to merge, with the changed failure paths preserving target lifecycle semantics while preventing one prewarm failure from aborting the entire tick.

The implementation distinguishes transient fetch failures from unbuildable requests, applies retry backoff without deleting live targets, continues processing later targets, and retains the existing deletion behavior for invalid prewarm bodies.

Important Files Changed

Filename Overview
packages/core/src/cachekeep.ts Converts thrown fetch failures into transient results, preserves affected targets with backoff, and isolates unexpected per-target failures so each tick can continue and publish state.
packages/opencode/src/tests/cachekeep.test.ts Updates timeout expectations for the resolved transient result and adds coverage for target continuation, retention, backoff, publication, and unbuildable-body deletion.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  T[Cachekeep tick] --> D{Target is due?}
  D -- No --> N[Check next target]
  D -- Yes --> P[Build and send prewarm]
  P --> S{Result}
  S -- Success --> U[Advance cache expiry]
  S -- HTTP or transient failure --> B[Apply retry backoff]
  S -- Unbuildable body --> X[Delete tracked target]
  P -- Unexpected throw --> C[Log and apply defensive backoff]
  U --> N
  B --> N
  X --> N
  C --> N
  N --> F[Publish tracked sessions]
Loading

Reviews (1): Last reviewed commit: "fix(core): contain thrown prewarm errors..." | Re-trigger Greptile

Context used:

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.
Architecture diagram
sequenceDiagram
    participant Start as start()
    participant Tick as tick()
    participant Prewarm as prewarm()
    participant Send as sendPrewarm()
    participant Fetch as fetchImpl()
    participant Cache as cacheExpiresAt
    participant Publish as publishTrackedSessions()

    Note over Start,Publish: CacheKeep Tick Failure-Containment Flow

    Start->>Tick: schedule tick (60s)
    Tick->>Tick: loadStorage()
    loop each tracked target
        Tick->>Tick: check target.cacheExpiresAt <= dueAt
        alt target is due
            Tick->>Prewarm: prewarm(target, now)
            Prewarm->>Send: sendPrewarm(target)
            Send->>Fetch: fetch(url, { signal: AbortSignal.timeout(...) })
            alt fetch resolves successfully
                Fetch-->>Send: Response
                Send-->>Prewarm: { ok: true, ... }
                Prewarm-->>Tick: ok result
                Tick->>Cache: advance cacheExpiresAt
            else fetch throws (timeout/network)
                Fetch-->>Send: throws TimeoutError/TypeError
                Send-->>Prewarm: { ok: false, transient: true }
                Prewarm->>Cache: apply backoff (transient branch, not delete)
                Prewarm-->>Tick: transient failure result
                Note over Tick,Cache: Target retained, backed off, not deleted
            end
        else unexpected throw from prewarm
            Tick->>Tick: catch error, log warning
            Tick->>Cache: apply defensive backoff
        end
    end
    Tick->>Publish: publishTrackedSessions()

    Note over Send,Prewarm: Discriminator: status == null && !transient
    alt unbuildable body (status == null, not transient)
        Prewarm->>Prewarm: delete tracked target
    end
Loading

Re-trigger cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] A thrown prewarm aborts the whole cachekeep tick and blocks every later target permanently

1 participant