feat: add stale-on-error Redis recovery

[lan17]

Summary

Add opt-in stale-on-error recovery from the existing Redis value, rebased onto the native client-clock Redis design from #140.

• F = ttlSec[CacheLayer.REMOTE] remains the logical fresh age.
• M = staleOnErrorMaxAgeSec is the absolute recovery-age ceiling.
• Ordinary serving reads require 0 <= age < F.
• After a definitive ordinary Redis miss and source rejection, one bounded reread may serve 0 <= age < M.
• Recovered data is never republished, stored in process-local cache, or sent through shadow validation.

Closes #117

Configuration

new DialCacheKeyConfig({
  ttlSec: { [CacheLayer.REMOTE]: 300 },
  staleOnErrorMaxAgeSec: 3_600,
});

F = ttlSec[CacheLayer.REMOTE] is the logical fresh lifetime. M = staleOnErrorMaxAgeSec is the absolute recovery-age ceiling, not an additional duration after F. Ordinary serving requires 0 <= age < F; recovery requires 0 <= age < M. A positive policy must satisfy 0 < F < M <= 31,536,000 seconds.

In a non-null runtime provider overlay, omitting staleOnErrorMaxAgeSec inherits the value from defaultConfig; if neither supplies a positive value, recovery is off. 0 explicitly disables an inherited policy. DialCacheKeyConfig.disabled() supplies that explicit zero as part of its complete kill-switch overlay.

Like TTL and ramp leaves, the DialCacheKeyConfig constructor preserves this value for later policy resolution rather than range-validating it immediately. Static defaultConfig validation occurs when cached() is called, before the use case is registered, and on each getOrLoad() invocation. Invalid static combinations throw. Runtime overlays are validated per invocation; an invalid runtime M records config_resolution, disables only stale recovery for that invocation, and preserves an otherwise-valid fresh Redis policy.

When M is positive, subsequent ordinary and shadow-fill Redis writes request physical retention through M; otherwise they request F. Untracked values receive that requested TTL. Tracked values retain the existing one-hour physical cap from #140 and emit tracked_ttl_clamped when the request exceeds it. M remains an eligibility ceiling rather than a retention guarantee: invalidation, eviction, expiry, and the tracked cap can remove or fence a value sooner.

Changing policy does not mutate TTLs already stored in Redis. Enabling M lengthens only later successful writes. Disabling it stops recovery for later invocations and makes later writes request F, but previously retained keys are not proactively shortened; current readers still hide them after logical age F.

Public API changes

• DialCacheKeyConfig adds the optional readonly staleOnErrorMaxAgeSec constructor field.
• DialCacheKeyConfig.disabled() now sets staleOnErrorMaxAgeSec: 0 so a runtime kill-switch overlay cannot inherit stale recovery.
• The package root exports StaleRecoveryOutcome and StaleRecoveryMetricLabels.
• DialCacheMetricsAdapter adds the optional staleRecovery(labels) callback. Its bounded outcomes are served, miss, read_error, read_timeout, and deserialization_error; its labels are cacheNamespace, useCase, keyType, and outcome.
• The built-in adapters expose dialcache_stale_recovery_counter in Prometheus (subject to the configured prefix) and dialcache.stale_recovery.count in Datadog (subject to the configured namespace).
• No DialCache method, Redis SPI method/type shape, Redis key, frame version, or wire encoding is added. The deliberate runtime contract change for createdAtMs is described under Compatibility and rollout.

Execution

flowchart TD
  A["Initial native GET or MGET; core validates age against F"] -->|fresh hit| B["Return fresh value"]
  A -->|semantic miss| C["Call source; recovery eligible if it rejects"]
  A -->|deserialization or decompression miss| D["Call source; recovery forbidden"]
  A -->|transport, protocol, or read timeout| E["Call source; recovery forbidden"]

  C -->|fulfills| F["Attempt Redis fill using the resolved retention policy; return source value"]
  D -->|fulfills| F
  E -->|fulfills| G["Return source value; do not write Redis"]

  C -->|rejects| H["One bounded native reread; core validates age against M"]
  D -->|rejects| I["Throw the identical source rejection"]
  E -->|rejects| I

  H -->|eligible and payload loads| J["Return retained value without publication"]
  H -->|semantic miss, load miss, transport or protocol error, or timeout| I

Loading

A semantic miss is a null result from the semantic Redis client—such as a missing, short, unsupported-version, malformed-watermark, or watermark-fenced frame—or a frame rejected by core because its timestamp is invalid, logically expired, or future-dated. A valid-age frame whose payload cannot be loaded after decompression handling is a distinct deserialization miss. Invalid native reply types, unsupported frame payload encodings, client rejections, and read deadlines are read failures.

Only an initial semantic miss can lead to stale recovery. A successful source call after either kind of miss attempts a normal Redis fill; a successful source call after a read failure does not, because the initial read did not establish a safe Redis state. Every failed recovery path preserves the exact original source rejection, including arbitrary rejection values and FallbackTimeoutError.

The invocation's once-resolved F, M, and remote-read deadline govern the complete chain. Recovery receives a new independent remoteReadTimeoutMs budget. Existing request and process coalescing share the complete initial-read/source/recovery chain.

Redis and clock behavior

The command and adapter shapes remain native and unchanged:

• untracked reads use one native GET;
• tracked reads use one primary-routed atomic MGET(value, watermark);
• writes use one native SET of a complete frame;
• invalidation remains the only Lua path;
• frame v1, Redis keys, watermarks, DialCacheRedisClient, and RedisReadRequest are unchanged.

The bundled adapters stamp each write with the writer process's Date.now(). After a read settles, core samples the reader process's Date.now() once and applies the strict logical-age boundary to every serving frame, tracked or untracked. Ordinary and initial-shadow reads use F; recovery uses M. The non-serving shadow C1 confirmation intentionally bypasses logical age solely to determine whether the original payload bytes were superseded.

Future-dated serving frames fail closed before deserialization. Because logical-age enforcement now covers untracked frames, the existing observeFutureTimestampOffset hook and built-in future-offset metrics may newly receive untracked layer="remote" observations, including from recovery reads. A shadow confirmation may retain a future-dated frame only for payload comparison and can never serve it.

DialCache does not call Redis TIME or estimate clock offsets. Cross-node clock offset can move logical expiry early or late, so synchronized and monitored application-node clocks remain an operational requirement. Durations and deadlines continue to use the monotonic clock.

Failure behavior and observability

• Initial Redis read errors, protocol failures, timeouts, and payload-load/deserialization misses never trigger recovery. Missing, short, unsupported-version, logically expired, future-dated, and watermark-fenced frames are semantic misses and may qualify.
• A recovery failure never replaces the original source rejection.
• Tracked recovery rereads the value and watermark atomically, so an invalidation during the source attempt can fence the retained frame.
• Recovery does not refresh Redis, populate process-local cache, or schedule shadow validation. Request-local memoization may retain a recovered result only within the active request scope.
• Each attempted recovery emits one optional bounded staleRecovery outcome: served, miss, read_error, read_timeout, or deserialization_error.
• The reread also traverses ordinary layer="remote" telemetry, adding a second request/get observation. A semantic or deserialization miss adds a second miss; a transport/protocol error or timeout adds the corresponding cache-read error. Request-derived hit-rate and Redis-QPS panels should separate this recovery population using staleRecovery outcomes.
• Prometheus and Datadog expose the bounded recovery counter. Existing fallback error and duration telemetry continues to record the source rejection even when recovery succeeds.

Compatibility and rollout

The configuration and metrics additions are optional and additive at the TypeScript surface; existing custom metrics adapters continue to compile. The Redis SPI shape, Redis keys, frame version, and wire format are unchanged.

The runtime serving contract is deliberately breaking even when stale recovery is omitted or set to 0: ordinary reads now treat every decoded frame's createdAtMs as authoritative and enforce logical F, including untracked reads. Custom Redis clients that return a constant timestamp must migrate to the frame's real epoch-millisecond writer timestamp before upgrading. A lowered runtime F therefore affects existing frames immediately, and reader/writer clock skew can make logical expiry early or late.

The stale-retention-specific mixed-version hazard begins once positive M affects physical writes. An older reader does not enforce logical F, so it can serve a physically retained frame as fresh between F and M.

1. Deploy the new version with staleOnErrorMaxAgeSec omitted or 0.
2. Upgrade the complete reader fleet.
3. Enable positive M only for selected use cases.
4. Monitor source failures, recovery outcomes, Redis memory and evictions, and application-node clock health.

Disabling M on current readers is safe because they continue to enforce F, but it does not shorten keys already retained under the earlier policy. Do not reintroduce an older reader until the longest physical retention actually written for affected keys has elapsed, or those keys have been isolated or removed.

Cost model

• Fresh hit or ordinary miss: unchanged Redis command count (GET or MGET only) and only local age arithmetic. Native reads transfer a logically stale retained frame before core filters it by age.
• Successful source refresh: unchanged one native SET; only its requested physical TTL may be longer.
• Qualifying source rejection: at most one additional native GET or MGET, including a second transfer of the retained frame when it is still present.
• The main operational cost is longer resident payload lifetime and the resulting memory, expiration, and eviction pressure—not Lua or Redis clock calls.

Validation

• Node 22.22.0: typecheck, 585 unit tests with 98.11% statement coverage, build, and packed ESM/CJS/TypeScript consumers.
• 135 live Redis integration tests passed; 2 GLIDE Cluster cases skipped after the expected unavailable-cluster connection timeout.
• Strict F/M and equality boundaries, future frames, runtime overlays, source failures/timeouts, invalidation races, coalescing, request-local behavior, compression, shadow confirmation, metrics, and physical retention.
• The isolated live stale-on-error benchmark passed its semantic assertions and reported the expected one native read per fresh/retained operation and two reads per recovery operation.

BREAKING CHANGE: Ordinary Redis reads now enforce logical age from each frame's createdAtMs, including untracked reads. Custom Redis clients must return real epoch-millisecond writer timestamps, and deployments must roll out new readers before enabling physical M retention.

[lan17]

Deep review: stale-on-error recovery

Verdict: core semantics are sound; nothing here blocks the serving path. The F/M boundary math (strict 0 <= age < limit, shared by serving, recovery, and shadow C0 through the single validateFrameAge), recovery isolation (recovered values provably never reach Redis, the local LRU, or shadow — the writers at dialcache.ts:819/1065/1277 are all unreachable from the recovery return), original-rejection preservation, coalescing of the full read/source/recovery chain, exactly-one-staleRecovery-outcome accounting, and the tracked reread's watermark fencing all held up under adversarial verification. Typecheck and all 588 unit tests pass on the branch.

15 findings as inline comments — 4 medium, 11 low. The mediums cluster on observability and validation depth, not data correctness:

• silent age-gate rejections (redis-cache.ts:474) — the only frame-rejection class with zero diagnostic;
• README:452's "telemetry remains unchanged" vs the intentional double-emission into layer="remote";
• M never validated against the tracked 1h cap (runtime-config.ts:137) — per-write tracked_ttl_clamped error noise plus a silently truncated recovery window;
• the unreachable pre-metric throw in recoverWithResolvedConfig (redis-cache.ts:167) — the one zero-outcome recovery exit.

Verified non-issues (they look like bugs but aren't)

• Dropping finishRedisChain's resolvedRemoteConfig parameter removed dead code: on main, the remoteErrored ? resolvedRemoteConfig arm could never execute — the call sites that passed the parameter only ever received hit/miss/error results.
• Invalid-stamp frames classifying as recovery-eligible cache_miss is consistent with the documented taxonomy (README:612 groups timestamp-domain with watermark/future-time rejections); the excluded "invalid reply/encoding" class is thrown protocol errors, which already bypass recovery via the status: "error" arm.
• The recovery read's inFallback: false on errors is correct per README:950's cache-plumbing-versus-application definition.

Below-the-cut minors (verified, not inlined)

• The three benchmark scripts now share five copy-pasted helpers with no common module: deferred, readPositiveInteger, noOpMetrics (already drifted — the new copy has five members the old lacks), the cmdstat parse loop, and the connect boilerplate. A scripts/benchmark-lib.mjs stops the drift.
• measureScenario's redis parameter has exactly one possible argument (every call site passes the module-level client), and four of its five per-command counters are stored but never read.
• safeMetrics wraps staleRecovery in the 6-line presence-preserving spread copied from shadowValidation, but nothing gates on its presence (sole consumer is metrics.staleRecovery?.(...)); the one-line callObserver(() => metrics.staleRecovery?.(labels)) already used for compression in the same literal is behavior-identical.

_{Methodology: 9 independent finder angles → 20 adversarial verifiers (one per deduped candidate; 3 candidates refuted and withheld) → gap sweep. Two findings were validated by implementation: the reason-enum alternative to skipStaleRecovery compiles clean and passes all stale-on-error tests, flipping that failure mode from fail-open to fail-closed.}

[lan17]

Follow-up on the below-the-cut scripts/benchmark-lib.mjs suggestion: I am not extracting a shared benchmark module in this PR. The overlap is pairwise rather than coherent across all three scripts (deferred and environment parsing in one pair; Redis connection and command-stat parsing in another), and their options and metrics semantics differ. A module now would create an abstraction around coincidences.

I will take the local simplifications, including the unused measureScenario parameter and counters. Cross-script extraction can wait until there is a stable common surface.