Bug-hunt round 32: unmergeable-once-wrapped capture records, dual-recorder misdiagnosis

.abcd/work/DECISIONS.md

@@ -803,3 +803,50 @@ Architecture-shaping decisions graduate to an ADR under
   diagnostic one, and the "three consistent siblings" premise doesn't
   hold — `timeline.go:514` prints neither form's first occurrence either,
   and `validate.go:155` prints raw like `analyze.go` does).
+- 2026-08-06 — Bug-hunt round 32: two confirmed defects. `POST
+  /api/interactions` (`demo.go`) and `transcribe` now check the *timeline
+  entry* a record becomes — via new `timeline.EventEntry`/`SpeechEntry` and
+  `session.EncodedLen` — against the JSONL line limit, not just the record
+  itself: `merge`'s `src`/`id`/`payload` wrapping (`timeline.BuildEntries`)
+  could push an entry over the limit that its source record, checked alone,
+  passed, so a record was durably persisted at 204/exit 0 and then
+  permanently unmergeable: `merge` refused to write its timeline entry on
+  every re-run, leaving `report` and `analyze` with no timeline to read, with
+  no CLI-level repair. `session.WriteJSONL`'s encoder also stopped HTML-escaping
+  `<`, `>`, `&` into six-byte `\uXXXX` sequences (`SetEscapeHTML(false)`,
+  matching `compactLine`'s existing non-escaping behaviour): the escaping
+  alone could inflate an accepted record's wrapped entry up to sixfold,
+  pulling the failure window down from a ~30-byte sliver at the very top of
+  the range to any record from roughly 700 KiB up. Second: `record -video`
+  excluded only the single recorder `anyExit`'s select happened to observe
+  (`if c != dead`) from the missing-output sweep; when a second recorder also
+  exited on its own at the same moment, it fell through to
+  `classifyMissingOutput`'s "stayed blocked on the permission prompt"
+  narrative — disproved by its own exit — with its real exit status never
+  surfaced. Now every child whose `done` channel is already closed when the
+  exit is observed (sampled before `stopAll`'s SIGINT reaches the others, so
+  a live recorder's clean shutdown is never mistaken for a self-exit) is
+  excluded and gets its own `classifyRecorderExit` diagnosis. Refuted:
+  `ci.yml`'s `check`-job comment omitting gofmt/ldflags/installer steps from
+  its own prose (both refuters: gofmt was already in the job when the
+  comment was first written, so its omission was never staleness, and the
+  comment was rewritten in round 15 — after the installer steps already
+  existed — without naming them either: a rationale note about
+  Ubuntu-only/single-job-name, not an enumeration, unlike the file header
+  ten lines above which round 26 fixed for exactly this reason); `release.yml`'s header omitting the install-e2e/version-assert/
+  no-branch-commit-tripwire steps (both refuters: the header states the
+  tripwire as an invariant ("nothing is pushed to any branch"), predates
+  none of the "omitted" steps — the post-publish attestation verify is
+  equally unnamed and equally day-one — and a full rewrite would duplicate
+  the fuller per-step comments 200 lines below). Not re-raised:
+  `session-directory.md`'s `words` row omission-cause gap, already
+  adjudicated and discarded on a split refuter verdict in round 31.
+  Post-hoc adversarial review of the round's own PR (correctness; docs
+  accuracy) independently caught a regression the second finding's fix
+  introduced: the sibling early-exited child's `atStartup` was sampled
+  after `stopAll`/`stopDemo`, reopening the exact slow-stop-poisons-
+  classification bug `dead`'s own pre-`stopAll` sampling exists to avoid.
+  Fixed by moving that sampling into the same pre-`stopAll` loop, with a
+  new regression test. Both reviewers' verdicts were BLOCK, so per the
+  loop's merge gate the PR (#48) stays open for the human rather than
+  auto-merging, even with the fix pushed and CI green.

CHANGELOG.md

@@ -523,6 +523,23 @@ Capture integrity:
   ordinary first-run whisper.cpp failure, since it never auto-downloads one —
   used to destroy the record-origin capture for a refusal `-model` alone
   already made knowable in advance.
+- **Behaviour:** `POST /api/interactions` and `transcribe` refuse a record
+  whose *timeline entry* — not the record itself — would exceed the JSONL
+  line limit, and `WriteJSONL`'s encoder no longer HTML-escapes `<`, `>`,
+  and `&` into six-byte `\uXXXX` sequences. `merge` wraps an accepted record
+  into a larger entry (a `src`/`id`/`payload` envelope, and — because of the
+  escaping — up to sixfold inflation for any of those three characters), so
+  a record within the raw line limit could still be durably persisted at
+  204/exit 0 and then permanently unmergeable: `merge` refused to write its
+  timeline entry on every re-run, leaving `report` and `analyze` with no
+  timeline to read, with no CLI-level repair.
+- `record -video` no longer misdiagnoses a second recorder that also exits
+  on its own before being asked to stop: only the recorder `anyExit`'s
+  select happened to observe was excluded from the missing-output sweep, so
+  any other recorder that self-exited at the same moment was still routed
+  through `classifyMissingOutput`'s "stayed blocked on the permission
+  prompt" narrative — disproved by its own exit — with its real exit status
+  never surfaced.
 
 ## [0.4.0] - 2026-07-24
 

docs/how-to/instrument-your-own-app.md

@@ -15,7 +15,7 @@ This creates a fresh session directory (with `manifest.json` anchoring the sessi
 | `POST /api/interactions` | one normalised interaction (a single JSON object) | `interactions.jsonl` (one line per request) |
 | `POST /api/events` | a batch of raw rrweb events (a JSON array) | `events.rrweb.jsonl` (one line per array element) |
 
-Both endpoints accept POST only (anything else returns 405) and return `204 No Content` on success. `/api/interactions` caps the body at 4 MiB — the readable JSONL line limit, since one request becomes one line — and rejects with 400 anything the merge step could not read back: invalid JSON, a body that is not a JSON object, or one with a missing/implausible `t` (it must be positive and on a plausible session clock) or a missing `kind`, per the table below; `/api/events` caps the batch body at 8 MiB and rejects anything that is not a JSON array with 400. A body over its cap gets 413, as does a batch element that would itself exceed the 4 MiB line limit. A body that passes every check but fails to append (a filesystem error) gets 500 rather than a false 204 — treat it as "not captured", the same as any other non-204 response.
+Both endpoints accept POST only (anything else returns 405) and return `204 No Content` on success. `/api/interactions` caps the body at 4 MiB — the readable JSONL line limit, since one request becomes one line — and rejects with 400 anything the merge step could not read back: invalid JSON, a body that is not a JSON object, or one with a missing/implausible `t` (it must be positive and on a plausible session clock) or a missing `kind`, per the table below; `/api/events` caps the batch body at 8 MiB and rejects anything that is not a JSON array with 400. A body over its cap gets 413, as does a batch element that would itself exceed the 4 MiB line limit, or an interaction that fits the line limit alone but whose timeline entry would not once `merge` wraps it (a `src`/`id`/`payload` envelope on the session-relative clock, see [`timeline.jsonl`](../reference/session-directory.md#timelinejsonl)). A body that passes every check but fails to append (a filesystem error) gets 500 rather than a false 204 — treat it as "not captured", the same as any other non-204 response.
 
 To defend the evidence against cross-origin forgery (CSRF), DNS-rebinding, and a non-browser client on the same network simply forging a loopback `Host`, the write endpoints require the request to come from a loopback remote address, a loopback `Host`, and — when present — a loopback `Origin` (any loopback host, whatever its port; anything else gets 403), plus `Content-Type: application/json` (415 otherwise). The remote-address check means these endpoints only ever accept writes from the machine running `testimony demo` itself — proxying `/api/*` to it from your dev server (step 5) still works, since the proxy's own request originates there, but a browser on another device never can, however it is served the page. Post from your app's own origin and always set the JSON content type, as the snippet below does. Each accepted body is re-encoded before it is appended: an `/api/interactions` body becomes exactly one JSONL line, while an `/api/events` batch is split into one line per array element, so a single request there can produce several records.
 

docs/reference/cli.md

@@ -32,7 +32,7 @@ Behaviour: creates a new session directory named after the current time (`YYYY-M
 - `POST /api/interactions` — one JSON object per request, appended as one line of `interactions.jsonl`.
 - `POST /api/events` — a JSON array per request, each element appended as one line of `events.rrweb.jsonl`.
 
-Both accept POST only (405 otherwise) and require a loopback remote peer, a loopback `Host`, and — when an `Origin` header is present — a loopback origin, and `Content-Type: application/json` (403 for any of the first three, 415 for the last). This guards the unauthenticated write endpoints against cross-site and DNS-rebinding forgery, and against a non-browser client on the same network simply forging a loopback `Host`. They return 204 on success and 400 on malformed bodies; `/api/interactions` also refuses with 400 any record `merge` would refuse — a body that is not a JSON object, or one missing the required `t` (a positive epoch-millisecond time on a plausible session clock) or `kind`. `POST /api/interactions` limits the body to 4 MiB — the readable JSONL line limit, since one request becomes one line — and `POST /api/events` limits the batch body to 8 MiB; a body over its limit is refused with 413, as is a batch element that would itself exceed the 4 MiB line limit. A body that passes every check but fails to append (a filesystem error) answers 500 rather than a false 204. Every refused capture write is logged to stderr, the operator's only signal, since the page posts via `sendBeacon`, which surfaces no status. The command blocks until interrupted (`Ctrl+C`).
+Both accept POST only (405 otherwise) and require a loopback remote peer, a loopback `Host`, and — when an `Origin` header is present — a loopback origin, and `Content-Type: application/json` (403 for any of the first three, 415 for the last). This guards the unauthenticated write endpoints against cross-site and DNS-rebinding forgery, and against a non-browser client on the same network simply forging a loopback `Host`. They return 204 on success and 400 on malformed bodies; `/api/interactions` also refuses with 400 any record `merge` would refuse — a body that is not a JSON object, or one missing the required `t` (a positive epoch-millisecond time on a plausible session clock) or `kind`. `POST /api/interactions` limits the body to 4 MiB — the readable JSONL line limit, since one request becomes one line — and also refuses with 413 a record that fits that limit alone but whose timeline entry would not once `merge` wraps it (a `src`/`id`/`payload` envelope on the session-relative clock, see [`timeline.jsonl`](session-directory.md#timelinejsonl)); `POST /api/events` limits the batch body to 8 MiB, refusing with 413 a batch over that cap or a batch element that would itself exceed the 4 MiB line limit. A body that passes every check but fails to append (a filesystem error) answers 500 rather than a false 204. Every refused capture write is logged to stderr, the operator's only signal, since the page posts via `sendBeacon`, which surfaces no status. The command blocks until interrupted (`Ctrl+C`).
 
 The loopback remote-peer requirement means an explicit non-loopback `-addr` host (e.g. `0.0.0.0:8737`) serves the page to other devices but refuses their capture posts — even one that sends a loopback `Host` — so only the machine running `record`/`demo` itself can ever write evidence; the command warns at startup that such a bind serves the page only.
 

internal/demo/demo.go

@@ -356,6 +356,26 @@ func (s *server) appendLines(w http.ResponseWriter, r *http.Request, f *os.File,
 			refuseWrite(w, r, msg, msg, http.StatusBadRequest)
 			return
 		}
+		// Fitting the JSONL line limit as received is not enough: merge re-frames
+		// this record into a timeline entry (rebased t, a src/id/payload envelope)
+		// that session.WriteJSONL checks against the same limit, and that entry can
+		// be larger than the record itself — so check the entry a record accepted
+		// here will become, not just the record, or a record this endpoint answers
+		// 204 to could still be one merge permanently refuses.
+		var rec timeline.Interaction
+		if err := json.Unmarshal(line, &rec); err != nil {
+			refuseWrite(w, r, "invalid JSON", "invalid JSON", http.StatusBadRequest)
+			return
+		}
+		entryLen, err := session.EncodedLen(timeline.EventEntry(rec, s.t0))
+		if err != nil {
+			refuseWrite(w, r, "invalid JSON", "invalid JSON", http.StatusBadRequest)
+			return
+		}
+		if tooLongOnceWrapped(entryLen) {
+			refuseWrite(w, r, "interaction's timeline entry over the JSONL line limit", "record exceeds the readable JSONL line limit", http.StatusRequestEntityTooLarge)
+			return
+		}
 		lines = append(lines, line)
 	}
 
@@ -428,6 +448,24 @@ func tooLongForJSONL(line []byte) bool {
 	return len(line)+1 > session.MaxJSONLLine
 }
 
+// eventIDGrowthMargin bounds tooLongOnceWrapped's blind spot: timeline.EventEntry
+// sizes an interaction's entry with the placeholder id "ev-001" (6 bytes), but
+// the id a merged session actually assigns grows with the interaction's
+// position among every interaction in the session, which this single record
+// does not know. 32 spare bytes cover an "ev-%03d" ordinal up to 32 digits —
+// past 10^32 interactions — headroom no real session comes remotely close to
+// needing.
+const eventIDGrowthMargin = 32
+
+// tooLongOnceWrapped reports whether an interaction's timeline entry, sized at
+// entryLen by session.EncodedLen(timeline.EventEntry(...)), could exceed the
+// JSONL line limit once merge assigns its real id — the entry this record
+// becomes, not the record itself, is what session.WriteJSONL checks when merge
+// writes timeline.jsonl.
+func tooLongOnceWrapped(entryLen int) bool {
+	return entryLen+eventIDGrowthMargin > session.MaxJSONLLine
+}
+
 // allowWrite guards the capture write endpoints against cross-origin forgery
 // (CSRF), DNS-rebinding of the loopback server, and — on a deliberately wide
 // bind — any non-browser client that simply forges the Host header, since

internal/demo/demo_test.go

@@ -582,6 +582,13 @@ func TestOversizedInteractionIsRefusedNotPersisted(t *testing.T) {
 		// Exactly the line limit: the terminating newline pushes the physical line
 		// one byte past what the readers can scan back, so it too must be refused.
 		"line limit plus newline": jsonRecordOfSize(t, session.MaxJSONLLine),
+		// Just inside the line limit as received, so this record alone would pass
+		// tooLongForJSONL — but merge's src/id/payload envelope pushes its timeline
+		// entry back over the limit, and that entry is what session.WriteJSONL
+		// checks when merge writes timeline.jsonl. Pre-fix this was accepted (204)
+		// and durably persisted, then permanently unreadable from the first merge
+		// onward with no CLI-level repair.
+		"record fits alone but not once wrapped": jsonRecordOfSize(t, session.MaxJSONLLine-1),
 	}
 	for name, body := range cases {
 		t.Run(name, func(t *testing.T) {
@@ -603,13 +610,15 @@ func TestOversizedInteractionIsRefusedNotPersisted(t *testing.T) {
 }
 
 // TestAcceptedInteractionStaysReadable pins the other side of the limit: a
-// record just inside it is still accepted and can be read straight back by the
-// same reader merge uses, so the refusal above is not simply refusing
-// everything large.
+// large record is still accepted and can be read straight back by the same
+// reader merge uses, so the refusal above is not simply refusing everything
+// large. It stays well clear of the line limit itself — a record that close
+// also has to leave room for its timeline entry's src/id/payload envelope,
+// which "record fits alone but not once wrapped" above exercises instead.
 func TestAcceptedInteractionStaysReadable(t *testing.T) {
 	s, dir := newTestServer(t)
 	w := httptest.NewRecorder()
-	s.handleInteraction(w, jsonPost("/api/interactions", jsonRecordOfSize(t, session.MaxJSONLLine-1), nil))
+	s.handleInteraction(w, jsonPost("/api/interactions", jsonRecordOfSize(t, session.MaxJSONLLine-4096), nil))
 	if w.Code != http.StatusNoContent {
 		t.Fatalf("status = %d, want 204", w.Code)
 	}