feat(trace-analyst): expose transport-neutral tools - #486
Conversation
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — 670fee5a
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
tangletools · auto-approval · reason: drewstone_author · 2026-07-29T20:28:05Z
tangletools
left a comment
There was a problem hiding this comment.
🟡 Value Audit — sound-with-nits
| Verdict | sound-with-nits |
| Concerns | 1 (1 weak-concern) |
| Heuristic | 0.0s |
| Duplication | 0.0s |
| Interrogation | 76.1s (2 bridge agents) |
| Total | 76.1s |
💰 Value — sound-with-nits
Maps the seven existing Ax trace tools to framework-neutral EvalToolDef-shaped descriptors via a thin delegating mapper — coherent, in-grain, single source of truth; only a pre-existing tool-def triplication worth noting.
- What it does: Adds buildTraceAnalysisToolDescriptors() (src/trace-analyst/tools.ts:199) which takes the same store and returns TraceAnalysisToolDescriptor[] — each carrying the stable 'traces' namespace, name, description, the Ax-generated JSON input schema (tool.parameters), and a handler that delegates to the original Ax tool.func. Exports it from src/trace-analyst/index.ts:89-93 and surfaces it in scripts/ve
- Goals it achieves: Decouple trace read access from the Ax transport so non-Ax hosts reuse the bounded reads (validation, byte ceilings, store errors) with zero drift. The mapper is deliberately NOT a reimplementation: handlers call tool.func and schemas come from tool.parameters, so both surfaces share one definition.
- Assessment: Sound and squarely in the codebase's grain. The repo already has a transport-neutral tool interface (EvalToolDef, src/eval-tools.ts:29) and an explicit 'framework-neutral tool defs' philosophy (src/fuzz/tools.ts:1-8). This change extends EvalToolDef rather than inventing a new shape, derives from Ax (the internal analyst loop's real transport) instead of hand-writing a second JSON-Schema source, a
- Better / existing approach: Considered two alternatives and rejected both. (a) A generic axFunctionToEvalToolDef converter: only one consumer exists (trace-analyst), and the trace mapper enforces a namespace invariant ('traces') plus a parameters-present guard that a generic couldn't — premature abstraction. (b) Rewriting the seven tools descriptors-first then wrapping into Ax for the analyst loop: would force hand-written J
- Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 2
- Bridge warning: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content
🎯 Usefulness — sound
A thin, correct adapter that re-exposes the seven existing bounded trace reads over the codebase's established EvalToolDef transport-neutral pattern, delegating to the same Ax handlers so nothing can drift.
- Integration: Fully reachable as a public export: trace-analyst/index.ts:89-95 re-exports it through src/traces.ts into the package.json './traces' subpath (dist/traces.js). The descriptor's handler signature (args, ctx?: {signal?}) matches EvalToolDef.handler exactly (eval-tools.ts:33) and threads the AbortSignal to tool.func (tools.ts:219). No in-repo production caller exists yet — only the test, docs/trace-a
- Fit with existing patterns: Slots cleanly into the grain. EvalToolDef is THE established transport-neutral tool descriptor (eval-tools.ts:29-34), already backing three tools via makeEvalTools plus a toOpenAiTool converter. TraceAnalysisToolDescriptor extends it (tools.ts:190), narrowing only the namespace to the literal 'traces' so hosts can encode it per their own convention (docs explicitly say 'the host owns namespace enc
- Real-world viability: Holds up off the happy path. The handler delegates to the identical bounded store operations, so the per-call byte ceiling (oversized fallback), 1..200 page limit, 1..500 max_matches, regex compile-check, and typed store errors (TraceNotFoundError) all propagate identically to the Ax path — proven by the parity test suite (11/11 passing, including byte-ceiling, five validation-error cases, and a m
- Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 1
💰 Value Audit
🟡 Transport-neutral tool-def shape now exists in three near-identical forms [duplication] ``
EvalToolDef (src/eval-tools.ts:29), ExploreToolDef (src/fuzz/tools.ts:13), and now TraceAnalysisToolDescriptor (src/trace-analyst/tools.ts:190) are structurally the same {name, description, parameters, handler} shape; ExploreToolDef is byte-for-byte identical to EvalToolDef, and the new type is EvalToolDef plus one literal-typed namespace field. The new type correctly EXTENDS rather than redeclares, so this PR doesn't add a fresh duplicate — but it does add a third name for one concept. A future
What this audit checks
It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.
| Pass | What it asks |
|---|---|
| Heuristic | Vague title? Whitespace-only or cruft-bearing diff? (content signals only) |
| Duplication | Do added function/class names already exist elsewhere in the repo? |
| Value Audit | What does it do? What goal does it achieve? Is it good? Better architecture or already-exists? |
| Usefulness Audit | Does it integrate and fit? Will it hold up in real use and actually get used? |
Findings are concerns, not blocks — the human reviewer decides what to do with them.
✅ No Blockers —
|
tangletools
left a comment
There was a problem hiding this comment.
✅ Approved — 3 non-blocking findings — 670fee5a
Full multi-shot audit completed 4/4 planned shots over 7 changed files. Global verifier still owns final merge decision.
Full immutable report for this review: trace
Summary comment for this run: full summary
tangletools · 2026-07-29T20:35:24Z · immutable trace
Exports the seven existing bounded trace-analysis reads as transport-neutral descriptors derived from the same Ax definitions and handlers. This lets Runtime and other hosts bind trace reads without copying schemas, store calls, validation, or byte limits.
Proof:
No Runtime, MCP, profile, candidate, lineage, or new-store dependency is introduced.