Skip to content

fix(supervisor-run): read Runtime file run contexts - #494

Open
drewstone wants to merge 2 commits into
mainfrom
fix/runtime-run-envelope-reader
Open

fix(supervisor-run): read Runtime file run contexts#494
drewstone wants to merge 2 commits into
mainfrom
fix/runtime-run-envelope-reader

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

Summary

  • read canonical agent-runtime FileRunContext directories directly in the existing supervisor-run analyzer
  • flatten nested spawn-journal envelopes by exact node identity without duplicating cost
  • retain exact profile digests, runtime tags, terminal status/reasons, and measured spend
  • leave roles, quality, completion time, provider-session identity, transcripts, verification, judge evidence, and missing spend unavailable
  • detect direct run paths as one report and parent directories as rollups

Regression

Before, traces 0.9.19 rejected a canonical Runtime run with: no supervisor run found.
After, the same CLI path reports one run and the exact Runtime root/profile/token facts.

Proof

  • focused supervisor-run tests: 88 passed
  • full suite: 4,585 passed, 3 skipped
  • lint, typecheck, example typecheck, build, and package verification pass
  • traces 0.9.19 packed integration passes on scientific-rk42-20260730 and technical-shopify-20260730 read-only paths

@drewstone
drewstone marked this pull request as ready for review July 30, 2026 17:56

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — 1e406155

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

tangletools · auto-approval · reason: drewstone_author · 2026-07-30T18:08:15Z

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Value Audit — sound

Verdict sound
Concerns 0 (none)
Heuristic 0.0s
Duplication 0.0s
Interrogation 175.3s (2 bridge agents)
Total 175.3s

💰 Value — sound

Adds a second on-disk reader for agent-runtime's FileRunContext journals that feeds the existing pure analyzer unchanged — a correct, in-grain extension through the codebase's documented source-agnostic contract.

  • What it does: Introduces readRuntimeSupervisorRun, a new SupervisorRunReader that parses agent-runtime's spawn-journal.jsonl envelope format (where each line is {kind:'begin'|'event', root, event}), strips the duplicate root marker that represents a nested driver twice (once as a child spawn in the parent tree, once as its own tree's root), and translates the result into the same SupervisorRunSources bytes the
  • Goals it achieves: Unblock the supervisor-run analyzer (and the traces CLI path that depends on it) on runs produced by agent-runtime's createFileRunContext, which previously failed with 'no supervisor run found.' After merge the same analyze/render/rollup/rollout pipeline works for both stores, with each source declaring exactly which facts it structurally cannot record (brain finish reasons, per-worker patches, pr
  • Assessment: This is the textbook use of the codebase's documented extension point. types.ts:130-132 states 'Any store that can produce these strings is a valid source; loopsSupervisorRunReader is ONE implementation,' and SupervisorRunReader (types.ts:208) is the strategy interface built for exactly this. The PR reuses the entire pure analyzer, the rollout-row minter, the renderer, the rollup, and the integrit
  • Better / existing approach: Searched for any existing spawn-journal / FileRunContext parser in src (grep spawn-journal|FileRunContext|createFileRunContext: none outside this PR). The two readers each define a local readMaybe, but they are deliberately different contracts — loops-reader swallows all errors (read-loops is best-effort across many artifacts), runtime-reader rethrows non-ENOENT errors (the journal is required, so
  • Model: opencode/zai-coding-plan/glm-5.2
  • Bridge attempts: 2
  • Bridge warning: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content

🎯 Usefulness — sound

Adds the agent-runtime file-backed layout as a second SupervisorRunReader that auto-routes from every existing entry point, unblocking the traces CLI on canonical Runtime runs — built exactly in the grain the source-agnostic reader interface was designed for.

  • Integration: Fully wired. The three new exports are reachable from every existing entry point: analyzeSupervisorRun() auto-detects Runtime dirs via isRuntimeSupervisorRunDir() and routes to readRuntimeSupervisorRun (loops-reader.ts:273-278); writeSupervisorRunReport() does the same (loops-reader.ts:311-313); findSupervisorRunDirs() walks both layouts (loops-reader.ts:405-431); all re-exported from src/index.ts
  • Fit with existing patterns: Perfect grain. The SupervisorRunReader interface (types.ts:208-212) was explicitly designed source-agnostic — its own doc states 'loopsSupervisorRunReader is ONE implementation'. This is exactly the second implementation that contract was built for. It feeds the SAME analyzer (analyzeSupervisorRunSources) with zero analyzer-core changes, and declares its structural limits via SourceLimits (the una
  • Real-world viability: Holds up across error and edge paths, not just happy path. Fail-closed on every malformed input: refuses mismatched profile digests (test:481), mismatched result roots (test:523), two top-level roots (test:555). Keeps usdKnown:false as unavailable, not zero (test:309). Unsettled children do not fabricate worker spend (test:345). Interruption is recorded as execution status, never as quality succes
  • Model: opencode/zai-coding-plan/glm-5.2
  • Bridge attempts: 1

No concerns — sound change, no better or existing approach found. ✅


What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass What it asks
Heuristic Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication Do added function/class names already exist elsewhere in the repo?
Value Audit What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

value-audit · 20260730T181135Z

@tangletools

Copy link
Copy Markdown
Contributor

✅ No Blockers — 1e406155

Review health 100/100 · Reviewer score 83/100 · Confidence 90/100 · 4 findings (4 low)

glm: Correctness 83 · Security 83 · Testing 83 · Architecture 83

Reviewer score is advisory once the run is complete and the verdict has no blockers.

Full multi-shot audit completed 6/6 planned shots over 16 changed files. Global verifier still owns final merge decision.

🟡 LOW Envelope shape notation conflates begin and event records — docs/trace-analysis.md

The doc says the reader unwraps { kind: 'begin' | 'event', root, event } envelopes. In source (runtime-reader.ts:89-116), begin records carry { kind, root, at } with NO event field, while only event records carry { kind, root, event }. A reader inspecting actual journal lines would see begin rows with at and no event, contradicting the notation. Fix: describe as two shapes, e.g. begin envelopes { kind: 'begin', root, at } and event envelopes { kind: 'event', root, event }. Cosmetic; no behavioral impact.

🟡 LOW Economics totalUsd assumes Runtime settled.spent is a subtree rollup — src/supervisor-run/analyze.ts

rootChildCloses (direct children of root only) are summed for journalWorkerUsd/journalWorkerIn/journalWorkerOut. This is correct ONLY if Runtime's settled.spent includes the entire subtree's spend (rollup). The test validates this (child 25 includes grandchild 5; result.spentTotal = 35 = brain 10 + child 25). But there is no assertion or guard that catches a non-rolled-up spend — if Runtime changes to per-node-only spend, totalUsd would silently undercount with no error. Consider cross-checking against result.spentTotal.usd when available and flagging a gap on mismatch.

🟡 LOW Nested root-marker deduplication uses strict parent===undefined check — src/supervisor-run/runtime-reader.ts

The root marker detection (entry.root === id && entry.event.parent === undefined) and the dedup filter both use strict === undefined. If Runtime ever emits parent: null or parent: "" in a nested-tree root marker, the marker would NOT be recognized or deduplicated, leaving a duplicate spawned event in the normalized journal. The child-spawn indexing (nonEmptyString(parent) !== null) is correct. This is robust for the current Runtime format but brittle if the contract changes. Consider using nonEmptyString(entry.event.parent) === null for both checks to be format-resilient.

🟡 LOW completedAt fallback changed from max-timestamp heuristic to root-close time — src/supervisor-run/source-facts.ts

Old code: completedAt = state.completedAt ?? max(all spawn.at, close.at). New code: completedAt = state.completedAt ?? rootClose?.at ?? null. For loops journals the root (supervisor) never emits a settled/cancelled event, so rootClose is null. If a loops run lacks state.completedAt (e.g., interrupted before state.json was finalized), supervisorWallMs now becomes unavailable instead of getting the old best-effort value from the last worker event. Low impact because loops state.json is written at completion and typically includes completedAt. Consider documenting this behavioral change or adding a fallback for journals with no root close.


tangletools · 2026-07-30T18:19:02Z · trace

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Approved — 4 non-blocking findings — 1e406155

Full multi-shot audit completed 6/6 planned shots over 16 changed files. Global verifier still owns final merge decision.

Full immutable report for this review: trace

Summary comment for this run: full summary


tangletools · 2026-07-30T18:19:02Z · immutable trace

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants