fix(analyst): enforce exact reviews, cost, and scoring

[drewstone]

Summary

• bind analyst review requests and decisions to the canonical full-run digest
• route Phoenix and Autoevals model calls through campaign cancellation and cost accounting
• score AgentRx category and causal-step location independently
• reject invalid comparison confidence and resample controls

Proof

• pnpm test: 321 files passed, 2 skipped; 4,383 tests passed, 3 skipped
• pnpm typecheck: passed
• pnpm typecheck:examples: passed
• pnpm lint: 603 files passed
• pnpm build: passed
• pnpm verify:package: passed
• pnpm audit --prod --audit-level high: no known vulnerabilities
• five original adversarial cases: 73/73 focused tests passed

[tangletools]

✅ Auto-approved drewstone PR — 34ac2b25

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: drewstone_author · 2026-07-30T01:39:58Z}

[tangletools]

🟢 Value Audit — sound

Verdict	sound
Concerns	0 (none)
Heuristic	0.0s
Duplication	0.0s
Interrogation	120.2s (2 bridge agents)
Total	120.2s

💰 Value — sound

Four integrity hardening changes — comparison validation, critical-step scoring independence, run-digest review binding, and cost-ledger routing for third-party evaluators — each extending existing patterns; no duplication or better architecture found.

• What it does: Four distinct integrity tightenings: (1) compareAnalystRunners now validates confidence in (0,1), resamples as a positive safe int <=1e6, and rejects non-finite output or invalid intervals (benchmark-comparison.ts:54-56,232-267); (2) scoreAnalystFindings decouples critical-step-location accuracy from issue-identity matching — pooled all-evidence replaces the old 'only count evidence on a finding a
• Goals it achieves: Close four silent-signal-corruption holes: (a) NaN/Infinity leaking through bootstrap statistics into comparison output; (b) a finding naming the right evidence location but the wrong category being double-penalized (lost issue credit AND lost causal-step credit); (c) review decisions replayable against a different run sharing a finding_id; (d) third-party LLM evaluator calls evading spend account
• Assessment: Coherent and grain-aligned. Every change reuses existing infrastructure rather than reinventing: comparison validation mirrors the assert* guards already in benchmark.ts (validateCase, validateBenchmarkOptions); run-digest extends the established analystFindingDigest->hashCanonical->snapshotAnalystFindings pattern one level up (finding->run) with a parallel snapshotAnalystRun; upstream-evaluator a
• Better / existing approach: none — this is the right approach. I checked for duplication in two places: (1) runUpstreamEvaluation partially overlaps llmJudge's paid-call wrapping (llm-judge.ts:186-203), but they carry genuinely different policy — llmJudge keeps a private-ledger fallback and trusts the receipt, while upstream-evaluators mandate the campaign ledger, branch CODE vs LLM evaluators, and reject post-call accountin
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 2
• Bridge warning: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content

🎯 Usefulness — sound

A coherent integrity-hardening across analyst review binding, upstream-evaluator cost accounting, and benchmark scoring/comparison validation, all wired into existing reachability paths with nothing dead.

• Integration: All four changes plug into already-wired paths. phoenixEvaluatorJudge/autoevalsScorerJudge are public judge adapters (campaign/index.ts:405) and the campaign runner already threads costLedger/costPhase/costTags into every judge.score (run-campaign.ts:608-614; contract at campaign/types.ts:122-130), so the new runPaidCall routing (upstream-evaluators.ts:220) is reached the moment a consumer registe
• Fit with existing patterns: Matches established patterns precisely. runUpstreamEvaluation (upstream-evaluators.ts:220-247) mirrors run-campaign.ts:456-568's runPaidCall + succeeded inspection + receipt settlement, and is shared by both phoenix and autoevals adapters (good centralization). The run-digest binding extends the pre-existing analystFindingDigest snapshot->canonical->hash pattern (feedback-trajectory-review.ts:62-6
• Real-world viability: Error paths are covered by tests: pre-abort (no call/no charge), missing ledger (reject), incomplete cost receipt (reject), mid-call cancellation with late receipt settled via waitForIdle (upstream-evaluators.test.ts:105-145,203-248), and non-finite/edge comparison inputs (benchmark-comparison.test.ts:189-241). abortableEvaluation races execute vs abort for CODE evaluators (upstream-evaluators.ts:
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 1

No concerns — sound change, no better or existing approach found. ✅

---

What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass	What it asks
Heuristic	Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication	Do added function/class names already exist elsewhere in the repo?
Value Audit	What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit	Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

_{value-audit · 20260730T014213Z}