Skip to content

fix(telemetry): dedupe retries by uuid and bound the send queue (v1.55.3) - #190

Merged
paulkr merged 1 commit into
mainfrom
fix/telemetry-dedupe-retry-queue
Aug 25, 2026
Merged

fix(telemetry): dedupe retries by uuid and bound the send queue (v1.55.3)#190
paulkr merged 1 commit into
mainfrom
fix/telemetry-dedupe-retry-queue

Conversation

@paulkr

@paulkr paulkr commented Aug 25, 2026

Copy link
Copy Markdown
Member

Summary

Part of the PostHog optimization plan (Cleanup → volume reduction). CLI Usage Rollup is 86% of all PostHog events (~1M/week), and 93% of that comes from two users' Macs — not from heavy usage, but from the CLI re-sending the same rollups over and over:

  • PostHog dedupes on the event uuid, not on $insert_id, which the CLI relied on.
  • A CLI process aborts in-flight sends at exit; the request body has usually already left, so PostHog ingests the event while the CLI keeps it queued for "retry next run". Every quick command then re-sends the whole backlog. One user's queue re-sent each rollup ~233 times (45,215 events from 194 real rollups in 2 days).
  • Side effect: the internal metrics dashboard multiplies command_count by those duplicates.

Changes (src/lib/analytics.ts, src/cli.ts)

  • Idempotent retries. Every queued event carries a uuid derived deterministically from its $insert_id (uuidFromInsertId, v5-shaped) and sends it in the POST body. Queue lines written by an older CLI get one when drained, so an inherited backlog dedupes on upgrade.
  • Bounded queue. drainQueue() counts dispatches per event (persisted before the send so a crash can't reset it), drops an event after SEND_MAX_ATTEMPTS = 3 or QUEUE_MAX_AGE_MS = 24h, and never dispatches the same event twice in one run.
  • Exit grace. flush() (now async) waits up to EXIT_GRACE_MS = 300 for in-flight sends only when there are any, then aborts the rest. postAction also drains the queue so a rollup emitted at exit usually lands in the same run instead of on the next one. Worst case added exit latency: 300 ms, only on runs that had something to send.
  • Anonymous-rate rollups. Rollups carry $process_person_profile: false and no $set (sheet row: "rollups do not need person profiles"). Note: person-property filters (e.g. the project's internal-user filter by email) won't apply to rollup events; they still key on distinct_id.
  • Version bump 1.55.2 → 1.55.3.

Expected effect once users update: CLI events drop from ~1M/week to roughly 70K/week.

Test plan

  • src/lib/analytics.test.ts: 27 tests — the 17 existing rollup tests plus 10 new delivery tests using a stubbed fetch (uuid derivation + legacy backfill, uuid in the wire body, delivered events pruned, grace period lets a slow send land, attempts counted and capped, age cap, no double dispatch per run, failed send retried).
  • npm run typecheck clean; npm run build (tsup) succeeds.
  • Full suite: 468 pass; the 6 resolveConfig failures in src/lib/config.test.ts fail identically on a clean main (verified by stashing this change) and are unrelated.
  • After release: CLI Usage Rollup volume per user in PostHog (HogQL count() / uniq($insert_id) per distinct_id) drops to ~1.

🤖 Generated with Claude Code

…5.3)

PostHog dedupes on the event uuid, not on $insert_id, and a CLI process
aborts in-flight sends at exit after the body has usually already left.
So every quick command re-sent the whole backlog, PostHog ingested each
copy as a new event, and nothing ever pruned the queue: one user's queue
re-sent every rollup ~230 times, and two users alone were 93% of all
CLI Usage Rollup events (~1M/week).

- every queued event carries a uuid derived from its $insert_id and sends
  it, so a re-sent copy collapses on ingest; lines written by an older CLI
  get one when drained, so an inherited backlog dedupes too
- drainQueue counts dispatches per event (persisted before the send) and
  drops an event after 3 attempts or 24h; never dispatches the same event
  twice in one run
- flush() lets in-flight sends finish for up to 300ms before aborting, and
  postAction drains the queue so a rollup emitted at exit usually lands in
  the same run
- rollups are sent with $process_person_profile: false (anonymous rate)
  and without $set; the person already exists from the signup events

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@paulkr
paulkr merged commit 934a259 into main Aug 25, 2026
7 checks passed
@paulkr
paulkr deleted the fix/telemetry-dedupe-retry-queue branch August 25, 2026 14:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant