Skip to content

fix(analyst-loop): forward same-run findings - #658

Open
drewstone wants to merge 1 commit into
mainfrom
fix/analyst-chain-findings
Open

fix(analyst-loop): forward same-run findings#658
drewstone wants to merge 1 commit into
mainfrom
fix/analyst-chain-findings

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

Summary

  • expose agent-eval's opt-in same-run finding chain through runAnalystLoop
  • preserve independent analysts by default
  • document registration-order behavior and verify it with the real Eval registry

Verification

  • focused tests: 19 passed
  • full tests: 1,859 passed; 6 skipped
  • lint, typecheck, docs check, and package verification passed

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — 5944cb23

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

tangletools · auto-approval · reason: drewstone_author · 2026-07-29T20:08:17Z

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Value Audit — sound

Verdict sound
Concerns 0 (none)
Heuristic 0.0s
Duplication 0.0s
Interrogation 102.0s (2 bridge agents)
Total 102.0s

💰 Value — sound

Forwards agent-eval's existing opt-in same-run finding-chain option through runAnalystLoop, which was previously swallowing it; clean, minimal, defaults to off, tested against the real registry.

  • What it does: Exposes the chainFindings?: boolean option on RunAnalystLoopOpts and threads it into the registryOptions object passed to registry.run/registry.runStream. When on, the underlying AnalystRegistry populates each later analyst's ctx.upstreamFindings with findings emitted by earlier analysts in the same run (registration order = dependency order). When off (the default), behavior is unch
  • Goals it achieves: Close a real gap: the orchestrator was constructing the registry options object internally and silently dropping chainFindings, so callers had no way to turn on same-run finding chaining through runAnalystLoop even though the underlying registry fully supported it (agent-eval accumulates findings in registration order and populates ctx.upstreamFindings when the flag is set, with dedicated `r
  • Assessment: Good change. It is a minimal, correct forwarding that matches the exact established grain in the same object literal (...(opts.x ? { x: opts.x } : {})), defaults to off to preserve existing behavior, and adds the option to the narrowed registry type shapes so the typed contract stays honest. The two new tests use the REAL AnalystRegistry (not the stub) to prove both the on-path (second analyst
  • Better / existing approach: none — this is the right approach. Verified chainFindings/upstreamFindings is the real, mature capability in agent-eval (default-registry-Brxr728w.d.ts:230 RegistryRunOpts.chainFindings; default-registry-CHmdy2An.js:2088 populates ctx.upstreamFindings from an accumulating allFindings array; renderUpstreamFindings exists at d.ts:139). There is no pre-existing passthrough in runAnalystLoop — it
  • Model: opencode/zai-coding-plan/glm-5.2
  • Bridge attempts: 2
  • Bridge warning: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content

🎯 Usefulness — sound

A minimal, opt-in pass-through that exposes agent-eval's native same-run finding chain through the orchestrator, tested against the real registry — coherent, non-competing, and reachable.

  • Integration: Reachable and correctly wired. run-analyst-loop.ts:148 forwards opts.chainFindings into the curated registryOptions object alongside priorFindings/costLedger/signal, using the same conditional-spread idiom. The underlying agent-eval AnalystRegistry natively consumes chainFindings and populates ctx.upstreamFindings (confirmed in node_modules/@tangle-network/agent-eval/dist/default-registry-Brxr728w
  • Fit with existing patterns: Fits the grain precisely. It mirrors the established forwarding pattern for the sibling priorFindings option and the other curated registry opts; the orchestrator deliberately curates which fields cross into the registry rather than blanket-spreading (since RunAnalystLoopOpts carries loop-only fields like findingsStore/baselineRunId that must not reach the registry), and this change respects that
  • Real-world viability: Holds up because the orchestrator's role here is pure pass-through; the hard parts (sequential/dependency-ordered execution, populating upstreamFindings, handling a failed earlier analyst) are owned by the agent-eval registry, an established component. The flag defaults off, so every existing analyst suite keeps identical behavior — zero regression surface. The forwarding guard `opts.chainFindings
  • Model: opencode/zai-coding-plan/glm-5.2
  • Bridge attempts: 1

No concerns — sound change, no better or existing approach found. ✅


What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass What it asks
Heuristic Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication Do added function/class names already exist elsewhere in the repo?
Value Audit What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

value-audit · 20260729T201019Z

@tangletools

Copy link
Copy Markdown
Contributor

✅ No Blockers — 5944cb23

Review health 100/100 · Reviewer score 86/100 · Confidence 80/100 · 3 findings (3 low)

glm: Correctness 86 · Security 86 · Testing 86 · Architecture 86

Reviewer score is advisory once the run is complete and the verdict has no blockers.

Full multi-shot audit completed 4/4 planned shots over 6 changed files. Global verifier still owns final merge decision.

🟡 LOW opts prose example omits the newly-documented chainFindings — docs/api/runtime.md

The prose for RegistryAnalyzeProjection.opts still reads 'Optional run opts (e.g. priorFindings) forwarded verbatim to the registry.' even though this same opts object now also carries the newly added chainFindings. The type is correct (it is Parameters<AnalystRegistryLike['run']>[2], which includes both), so this is cosmetic only — but for discoverability the example could read 'e.g. priorFindings, chainFindings'. No behavioral impact.

🟡 LOW Chaining not exercised through the streaming (runStream) path — tests/analyst-loop.test.ts

captureSameRunUpstreamFindings does not pass onEvent, so runRegistry takes the registry.run() branch (run-analyst-loop.ts:153 requires both runStream AND onEvent). The chainFindings flag is built into the same registryOptions object for both branches (run-analyst-loop.ts:146-152), so the gap is low-risk, but the streaming+chaining combination is untested. Consider adding onEvent to one chaining case to cover the runStream forwarding path. Not a blocker.

🟡 LOW Explicit chainFindings:false case not tested — tests/analyst-loop.test.ts

The 'isolated' test passes chainFindings as undefined. The loop forwards chainFindings only when !== undefined (run-analyst-loop.ts:148), so explicit false would also be forwarded to the registry — but that explicit-false path is not asserted. Minor coverage gap; the undefined default is the more important case and is covered.


tangletools · 2026-07-29T20:13:24Z · trace

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Approved — 3 non-blocking findings — 5944cb23

Full multi-shot audit completed 4/4 planned shots over 6 changed files. Global verifier still owns final merge decision.

Full immutable report for this review: trace

Summary comment for this run: full summary


tangletools · 2026-07-29T20:13:24Z · immutable trace

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants