Skip to content

feat(trace-analyst): expose transport-neutral tools - #486

Merged
drewstone merged 5 commits into
mainfrom
feat/transport-neutral-trace-tools
Jul 29, 2026
Merged

feat(trace-analyst): expose transport-neutral tools#486
drewstone merged 5 commits into
mainfrom
feat/transport-neutral-trace-tools

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

Exports the seven existing bounded trace-analysis reads as transport-neutral descriptors derived from the same Ax definitions and handlers. This lets Runtime and other hosts bind trace reads without copying schemas, store calls, validation, or byte limits.

Proof:

  • 4,224 tests passed, 3 skipped
  • 38 focused trace tests passed
  • typecheck, lint, build, package verification passed
  • built package read two real JSONL traces
  • critical audit approved with no blocking findings

No Runtime, MCP, profile, candidate, lineage, or new-store dependency is introduced.

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — 670fee5a

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

tangletools · auto-approval · reason: drewstone_author · 2026-07-29T20:28:05Z

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Value Audit — sound-with-nits

Verdict sound-with-nits
Concerns 1 (1 weak-concern)
Heuristic 0.0s
Duplication 0.0s
Interrogation 76.1s (2 bridge agents)
Total 76.1s

💰 Value — sound-with-nits

Maps the seven existing Ax trace tools to framework-neutral EvalToolDef-shaped descriptors via a thin delegating mapper — coherent, in-grain, single source of truth; only a pre-existing tool-def triplication worth noting.

  • What it does: Adds buildTraceAnalysisToolDescriptors() (src/trace-analyst/tools.ts:199) which takes the same store and returns TraceAnalysisToolDescriptor[] — each carrying the stable 'traces' namespace, name, description, the Ax-generated JSON input schema (tool.parameters), and a handler that delegates to the original Ax tool.func. Exports it from src/trace-analyst/index.ts:89-93 and surfaces it in scripts/ve
  • Goals it achieves: Decouple trace read access from the Ax transport so non-Ax hosts reuse the bounded reads (validation, byte ceilings, store errors) with zero drift. The mapper is deliberately NOT a reimplementation: handlers call tool.func and schemas come from tool.parameters, so both surfaces share one definition.
  • Assessment: Sound and squarely in the codebase's grain. The repo already has a transport-neutral tool interface (EvalToolDef, src/eval-tools.ts:29) and an explicit 'framework-neutral tool defs' philosophy (src/fuzz/tools.ts:1-8). This change extends EvalToolDef rather than inventing a new shape, derives from Ax (the internal analyst loop's real transport) instead of hand-writing a second JSON-Schema source, a
  • Better / existing approach: Considered two alternatives and rejected both. (a) A generic axFunctionToEvalToolDef converter: only one consumer exists (trace-analyst), and the trace mapper enforces a namespace invariant ('traces') plus a parameters-present guard that a generic couldn't — premature abstraction. (b) Rewriting the seven tools descriptors-first then wrapping into Ax for the analyst loop: would force hand-written J
  • Model: opencode/zai-coding-plan/glm-5.2
  • Bridge attempts: 2
  • Bridge warning: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content

🎯 Usefulness — sound

A thin, correct adapter that re-exposes the seven existing bounded trace reads over the codebase's established EvalToolDef transport-neutral pattern, delegating to the same Ax handlers so nothing can drift.

  • Integration: Fully reachable as a public export: trace-analyst/index.ts:89-95 re-exports it through src/traces.ts into the package.json './traces' subpath (dist/traces.js). The descriptor's handler signature (args, ctx?: {signal?}) matches EvalToolDef.handler exactly (eval-tools.ts:33) and threads the AbortSignal to tool.func (tools.ts:219). No in-repo production caller exists yet — only the test, docs/trace-a
  • Fit with existing patterns: Slots cleanly into the grain. EvalToolDef is THE established transport-neutral tool descriptor (eval-tools.ts:29-34), already backing three tools via makeEvalTools plus a toOpenAiTool converter. TraceAnalysisToolDescriptor extends it (tools.ts:190), narrowing only the namespace to the literal 'traces' so hosts can encode it per their own convention (docs explicitly say 'the host owns namespace enc
  • Real-world viability: Holds up off the happy path. The handler delegates to the identical bounded store operations, so the per-call byte ceiling (oversized fallback), 1..200 page limit, 1..500 max_matches, regex compile-check, and typed store errors (TraceNotFoundError) all propagate identically to the Ax path — proven by the parity test suite (11/11 passing, including byte-ceiling, five validation-error cases, and a m
  • Model: opencode/zai-coding-plan/glm-5.2
  • Bridge attempts: 1

💰 Value Audit

🟡 Transport-neutral tool-def shape now exists in three near-identical forms [duplication] ``

EvalToolDef (src/eval-tools.ts:29), ExploreToolDef (src/fuzz/tools.ts:13), and now TraceAnalysisToolDescriptor (src/trace-analyst/tools.ts:190) are structurally the same {name, description, parameters, handler} shape; ExploreToolDef is byte-for-byte identical to EvalToolDef, and the new type is EvalToolDef plus one literal-typed namespace field. The new type correctly EXTENDS rather than redeclares, so this PR doesn't add a fresh duplicate — but it does add a third name for one concept. A future


What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass What it asks
Heuristic Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication Do added function/class names already exist elsewhere in the repo?
Value Audit What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

value-audit · 20260729T202943Z

@tangletools

Copy link
Copy Markdown
Contributor

✅ No Blockers — 670fee5a

Review health 100/100 · Reviewer score 86/100 · Confidence 80/100 · 3 findings (3 low)

glm: Correctness 86 · Security 86 · Testing 86 · Architecture 86

Reviewer score is advisory once the run is complete and the verdict has no blockers.

Full multi-shot audit completed 4/4 planned shots over 7 changed files. Global verifier still owns final merge decision.

🟡 LOW New trace tool descriptors get no runtime value assertion — scripts/verify-package-exports.mjs

buildTraceAnalysisToolDescriptors({ store }) is constructed and TRACE_ANALYST_TOOL_NAMESPACE is assigned, but neither value is inspected at runtime — they are only pushed into the void [...] array at lines 419-420 so tsc type-checks them. Compare packedMissingTraceCode (line 248) which asserts .code, or hostedEventValid which runs a schema parse. The harness proves the symbols import and type-check from the packed tarball, but does not assert e.g. descriptors.length === 7 or traceToolNamespace === 'traces'. Low impact: th

🟡 LOW Validation-error parity only sampled for 2 of 7 tools — src/trace-analyst/tools.test.ts

The it.each at tools.test.ts:114 covers queryTraces (limit/offset) and searchTrace (regex/max_matches) but omits viewSpans (span_ids shape), viewTrace/searchSpan (empty trace_id/span_id via assertString), and the filters.regex_pattern empty-string guard in parseFilters (tools.ts:263). Defensible because the descriptor is a uniform await tool.func(args, ...) passthrough with zero per-tool logic, so drift is impossible by construction — but the parity property is only regression-locked for a sample. Fix: add the remaining 5 tools to the it.each table, or document that the delegation test at :67 is the canonical parity guard.

🟡 LOW Abort-signal forwarding branch is untested — src/trace-analyst/tools.ts

The handler closes with ctx?.signal ? { abortSignal: ctx.signal } : undefined (tools.ts:219), matching the convention in loop.ts:103. Every test calls descriptor.handler(args) with NO second argument (e.g. tools.test.ts:95, :106, :137), so the truthy-signal branch is never executed. A future refactor that broke signal forwarding (e.g. dropped ctx passthrough) would pass all current tests. Impact is low because the store interface has no abort-aware operations today, but the descriptor promises to thread the signal. Fix: add a test passing a ctx with an AbortSignal and assert it reaches tool.func (e.g. via an already-aborted signal that still no-ops but exercises the branch), or at minimum a type/shape assertion.


tangletools · 2026-07-29T20:35:24Z · trace

tangletools
tangletools previously approved these changes Jul 29, 2026

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Approved — 3 non-blocking findings — 670fee5a

Full multi-shot audit completed 4/4 planned shots over 7 changed files. Global verifier still owns final merge decision.

Full immutable report for this review: trace

Summary comment for this run: full summary


tangletools · 2026-07-29T20:35:24Z · immutable trace

@drewstone
drewstone merged commit a264a4f into main Jul 29, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants