feat(analyst): measure and validate trace analysis

[drewstone]

What

• Add reusable trace analyst benchmark cases, paired runner comparison, and Markdown reports.
• Add public AgentRx and CodeTraceBench label/output adapters that preserve each upstream task.
• Add Phoenix and Autoevals scoring adapters without reimplementing their evaluators.
• Require exact finding digests plus an independent completeness assessment before analyst output becomes optimizer data.
• Track precision, recall, F1, clean-case failures, critical-step accuracy, latency, tokens, and cost without inventing missing values.

Proof

• 323 test files: 4,360 passed, 3 skipped, 0 failed.
• 603 source files linted with no errors.
• TypeScript source and examples passed.
• Build, package exports, publint, package type checks, and production dependency audit passed.
• Mutation checks proved that editing reviewed evidence fails and omitting issues lowers the optimizer score.
• The branch merges cleanly into current main.

Limits

No paid model-backed analyst quality run is claimed here. This change makes that comparison reproducible and prevents unreviewed or selectively reported findings from being optimization data.

[tangletools]

✅ Auto-approved drewstone PR — aebce176

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: drewstone_author · 2026-07-30T01:08:07Z}

[tangletools]

🟢 Value Audit — sound

Verdict	sound
Concerns	1 (1 weak-concern)
Heuristic	0.0s
Duplication	0.0s
Interrogation	148.0s (2 bridge agents)
Total	148.0s

💰 Value — sound

Adds a principled, evidence-aware benchmark + paired-comparison harness for the trace analyst, plus digest-bound review integrity that blocks unreviewed findings from becoming optimizer data — built in the grain of the codebase and reusing the right existing primitives.

• What it does: Adds (1) runAnalystBenchmark/scoreAnalystFindings (src/analyst/benchmark.ts:312, :215) which scores finding-sets against labeled issues via optimal one-to-one assignment (linear-sum-assignment), tracking issue recall, finding precision, F1, critical-step accuracy, citation coverage, citation-label agreement, citation resolution against a real trace store, clean-case false positives, repeat
• Goals it achieves: Make analyst quality measurable against ground-truth labels instead of self-referential LLM judgment or gameable substring cues; make runner comparison reproducible with honest decomposition (quality / reliability / cost separated, failed runs counted as zero quality, unmeasured dimensions labeled rather than invented); and block unreviewed or selectively-reported findings from flowing into optimi
• Assessment: Coherent and well-grained. It reuses the correct existing primitives rather than reinventing: pairedBootstrap for CIs, makeFinding/AnalystFinding/EvidenceRef types, FeedbackTrajectory/feedbackTrajectoryToOptimizerRow, campaign JudgeConfig/Scenario, TraceAnalysisStore, CaptureIntegrityError, and canonicalString/hashCanonical from ledger-core. The 'fail loud, no silent zeros'
• Better / existing approach: none — this is the right approach. I checked the existing benchmark surfaces for overlap: src/benchmark.ts BenchmarkRunner (multi-turn agent scenarios, single string artifact, [0,1] LLM-judge score) and src/benchmarks/runner.ts + BenchmarkAdapter (retrieval/RAG, single artifact + [0,1] adapter score) are both the wrong shape — neither has any concept of matching a set of structured finding
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 2
• Bridge warning: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content

🎯 Usefulness — sound

A coherent, correctly-wired measurement+validation layer for trace analysts that reuses the campaign judge contract, the existing paired-bootstrap statistics, and the feedback-trajectory optimizer-row pipeline rather than forking them.

• Integration: Reachable now via two in-repo consumers, not just library exports. scoreAnalystFindings is consumed by src/campaign/analyst-surface.ts:76 inside traceAnalystQualityJudge(), which returns a JudgeConfig that runCampaign already scores scenarios with — so the campaign loop measures analyst quality through this code today. The integrity gate (analystFindingDigest, readAnalystReview, `com
• Fit with existing patterns: Fits established patterns; no competing primitive introduced. The judge adapters return the existing JudgeConfig shape (src/campaign/upstream-evaluators.ts:106-124) instead of defining a parallel scoring contract. compareAnalystRunners reuses pairedBootstrap from src/statistics.ts:1195 rather than reimplementing bootstrap logic. The behavioral-analyst rewrite follows the repo's fail-loud
• Real-world viability: Holds up past the happy path. The new pagination in behavioral-analyst.ts (listTraceIds) detects concurrent store mutation via total-drift, empty-page-with-has_more, and unique/total mismatch guards, and threads context.signal (which AnalystContext already exposes at src/analyst/types.ts:183). The benchmark runner counts failures as zero-quality rather than survivor-biased (`benchmark.ts:4
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 1

💰 Value Audit

🟡 Breaking rename of campaign analyst-surface exports + digest-bound optimizer rows requires consumer coordination [maintenance] ``

The PR renames public exports from @tangle-network/agent-eval/campaign: buildAnalystSurfaceDispatch→buildTraceAnalystSurfaceDispatch, failureModeRecallJudge→traceAnalystQualityJudge, AnalystScenario→TraceAnalystScenario, AnalystArtifact→TraceAnalystArtifact (src/campaign/index.ts:24-31). It also removes TraceAnalystGolden from kind-factory.ts and makes feedbackTrajectoryToOptimizerRow require a complete digest-bound review before exporting findings as optimizer data (src/

---

What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass	What it asks
Heuristic	Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication	Do added function/class names already exist elsewhere in the repo?
Value Audit	What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit	Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

_{value-audit · 20260730T011052Z}