feat(observability): record Redis connection state on failed slot operations by waleedlatif1 · Pull Request #7194 · simstudioai/sim

waleedlatif1 · 2026-08-28T02:45:55Z

Summary

A Redis command that never gets a reply fails identically in three very different situations, and nothing we record distinguishes them:

Situation	What we'd want to see
Connection still being established	`status: connecting`, non-zero queue depth
Reconnecting, command parked in the offline queue	`status: reconnecting`, non-zero queue depth
Socket silently dead	`status: ready`, queue depth 0

ioredis reports all three as Error: Command timed out with only its own timer frames in the stack — no app frame naming the call, no lifecycle event saying which happened. Today the cause can only be inferred from timing.

Adds describeRedisConnection() — client status, connection and ready ages, offline-queue depth, lifecycle counters (connects/reconnects/errors + last error), and whether the configured host is an IP literal or a DNS name.
Attaches it to the usage-reservation slot operations. Those are the first Redis calls a queued workflow makes, so an unusable connection surfaces there first.
connect and ready now log elapsed-since-construction, making the wait before a connection becomes usable directly measurable. Today that wait is spent inside a command's deadline, where it reads as a command timeout rather than as connection latency.
Extends the shared redisConfigMock, which documents that every export of the real module is present.

Why these fields

status is usually decisive on its own.
queuedCommands confirms it: a command parked in the offline queue was waiting on connection setup, whereas one written to a ready socket that never answered means the socket died without anything reporting it. ioredis has no public accessor for the offline queue, so this reads it through a narrow interface with a double-cast-allowed annotation.
hostKind rules DNS resolution in or out. No server-side telemetry can see a slow or retrying resolve — not VPC flow logs, not NLB counters, not cache metrics — so without this it stays a permanent unknown.

Security

REDIS_URL carries the AUTH token and is never logged. Only derived facts (IP vs DNS, TLS on/off, SNI override in play) are emitted. There's a test asserting the snapshot cannot contain the host, and I verified it fails if the URL is leaked into the payload.

Type of Change

Observability / diagnostics (no behavior change)

Testing

No behavior change: this adds a read-only snapshot and log fields. No timeouts, retry policy, or control flow were touched.

New tests cover no-client, connecting vs ready, offline-queue depth, lifecycle counting, IP vs DNS classification, and the secret-leak guard. Verified each fails for the right reason — forcing status to a constant breaks the discriminator tests, and leaking the URL into the snapshot breaks the secret test.

bun run type-check clean in apps/sim and packages/testing. 3149 tests pass across lib/core, lib/billing, lib/execution, background, plus 40 in packages/testing. bun run lint and all 37 audits pass.

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

vercel · 2026-08-28T02:46:05Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Aug 28, 2026 6:25pm

greptile-apps · 2026-08-28T02:48:04Z

Greptile Summary

Adds Redis connection-state diagnostics to usage-reservation failures without changing reservation control flow.

Records client age, readiness and ping timing, lifecycle counters, the last Redis error, and non-sensitive host/TLS classifications.
Logs diagnostics when reservation and slot-refresh Redis operations fail while preserving the original thrown error.
Extends Redis tests and the shared testing mock for the new diagnostic export.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
apps/sim/lib/core/config/redis.ts	Adds read-only Redis connection diagnostics, lifecycle timestamps and counters, and sanitized URL-derived classifications.
apps/sim/lib/billing/calculations/usage-reservation.ts	Attaches connection diagnostics to failed reservation and refresh operations while retaining existing error propagation.
apps/sim/lib/core/config/redis.test.ts	Covers diagnostic states, lifecycle counters, URL classification, discarded clients, and non-throwing behavior.
apps/sim/lib/billing/calculations/usage-reservation.test.ts	Verifies that diagnostic logging does not replace the original Redis error object.
packages/testing/src/mocks/redis-config.mock.ts	Adds a resettable shared mock for the new diagnostic export.

Sequence Diagram

sequenceDiagram
  participant Workflow
  participant Reservation as Usage Reservation
  participant Redis
  participant Diagnostics as Redis Diagnostics
  participant Logger
  Workflow->>Reservation: Reserve or refresh slot
  Reservation->>Redis: Execute Redis operation
  alt Redis operation succeeds
    Redis-->>Reservation: Result
    Reservation-->>Workflow: Continue
  else Redis operation fails
    Redis--xReservation: Original error
    Reservation->>Diagnostics: describeRedisConnection()
    Diagnostics-->>Reservation: Connection snapshot
    Reservation->>Logger: Log operation, error, and snapshot
    Reservation--xWorkflow: Rethrow original error
  end

_{Reviews (9): Last reviewed commit: "feat(observability): record Redis connec..." | Re-trigger Greptile}

cubic-dev-ai

All reported issues were addressed across 4 files

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

waleedlatif1 · 2026-08-28T02:54:34Z

@greptile

waleedlatif1 · 2026-08-28T03:03:35Z

@greptile

waleedlatif1 · 2026-08-28T03:12:55Z

@greptile

waleedlatif1 · 2026-08-28T03:16:24Z

@greptile

waleedlatif1 · 2026-08-28T03:47:23Z

@greptile

waleedlatif1 · 2026-08-28T03:49:50Z

@greptile

waleedlatif1 · 2026-08-28T04:54:44Z

@cubic review

waleedlatif1 · 2026-08-28T04:54:49Z

@greptile review

cubic-dev-ai · 2026-08-28T04:54:50Z

@cubic review

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 5 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You’re at about 95% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.}

_{Re-trigger cubic}

…rations A Redis command that never gets a reply fails identically whether the connection was still being established, was reconnecting with the command parked in the offline queue, or was a socket that had silently died. ioredis reports all three the same way — `Error: Command timed out` with only its own timer frames in the stack, no app frame naming the call, and no lifecycle event saying which happened. Nothing recorded anywhere distinguishes them, so the cause can only be inferred from timing. Adds `describeRedisConnection()`: client status, connection and ready ages, offline-queue depth, lifecycle counters, and whether the configured host is an IP or a DNS name. `status` alone usually decides it; queue depth confirms, since a parked command was waiting on connection setup while one written to a `ready` socket that never answered means the socket died unreported. Host kind rules DNS resolution in or out, which no server-side telemetry can see. Attaches it to the usage-reservation slot operations — the first Redis calls a queued workflow makes, so an unusable connection surfaces there first. Connect and ready now log elapsed-since-construction, making the wait before a connection becomes usable directly measurable; today it is spent inside a command's deadline, where it reads as a command timeout rather than as connection latency. Purely additive. Redis call arguments are unchanged, the wrapper rethrows the original error object, and `describeRedisConnection` never throws — it runs inside catch blocks where a throw would replace the real failure. All three have tests that fail if the behavior is removed. Only non-sensitive facts are derived from REDIS_URL, which carries the AUTH token and is never logged.

waleedlatif1 · 2026-08-28T18:25:44Z

@greptile

cubic-dev-ai Bot reviewed Aug 28, 2026

View reviewed changes

Comment thread apps/sim/lib/core/config/redis.ts Outdated

Comment thread apps/sim/lib/core/config/redis.ts Outdated

vercel Bot temporarily deployed to Preview August 28, 2026 02:54 Inactive

vercel Bot temporarily deployed to Preview August 28, 2026 03:03 Inactive

vercel Bot temporarily deployed to Preview August 28, 2026 03:07 Inactive

vercel Bot temporarily deployed to Preview August 28, 2026 03:16 Inactive

vercel Bot temporarily deployed to Preview August 28, 2026 03:19 Inactive

waleedlatif1 force-pushed the observability/redis-connection-diagnostics branch from 36d47da to 9334a4d Compare August 28, 2026 03:47

vercel Bot temporarily deployed to Preview August 28, 2026 03:47 Inactive

waleedlatif1 force-pushed the observability/redis-connection-diagnostics branch from 9334a4d to d387743 Compare August 28, 2026 03:49

vercel Bot temporarily deployed to Preview August 28, 2026 03:49 Inactive

cubic-dev-ai Bot reviewed Aug 28, 2026

View reviewed changes

waleedlatif1 force-pushed the observability/redis-connection-diagnostics branch from d387743 to 2e58e0d Compare August 28, 2026 18:25

vercel Bot temporarily deployed to Preview August 28, 2026 18:25 Inactive

waleedlatif1 merged commit 60cb6f5 into staging Aug 28, 2026
21 checks passed

waleedlatif1 deleted the observability/redis-connection-diagnostics branch August 28, 2026 18:29

TheodoreSpeaks mentioned this pull request Aug 28, 2026

v0.8.16: aws lambda, dynamics 365, actorless run fixes, usage chart #7224

Merged

Uh oh!

Conversation

waleedlatif1 commented Aug 28, 2026

Summary

Why these fields

Security

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Sequence Diagram

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Aug 28, 2026

Uh oh!

waleedlatif1 commented Aug 28, 2026

Uh oh!

waleedlatif1 commented Aug 28, 2026

Uh oh!

waleedlatif1 commented Aug 28, 2026

Uh oh!

waleedlatif1 commented Aug 28, 2026

Uh oh!

waleedlatif1 commented Aug 28, 2026

Uh oh!

waleedlatif1 commented Aug 28, 2026

Uh oh!

waleedlatif1 commented Aug 28, 2026

Uh oh!

cubic-dev-ai Bot commented Aug 28, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Aug 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented Aug 28, 2026 •

edited

Loading

greptile-apps Bot commented Aug 28, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading