gather: deterministic context-pack tool (aft_gather)

[iceteaSA]

gather: deterministic context-pack tool (aft_gather)

What

One new tool — aft_gather — assembles a bounded "context pack" (ranked, deduped, budgeted verbatim code evidence) in a single call. It replaces the multi-turn search → outline → zoom → callgraph read chain an agent otherwise runs to build context around a question or a symbol.

Two modes (mutually exclusive):

• question: "how does X work?" — seeds from handle_semantic_search (same pipeline as aft_search, all lanes/fallbacks)
• symbol + filePath — seeds from the callgraph (impact depth-1 callers + call_tree depth-1 callees)

Seeds expand one hop through the callgraph, dedupe by canonicalized (file, symbol) with seeds winning, and render via render_symbol_within_budget until a hard line budget (default 400, cap 800) is spent. Everything past the cut appears as one-line stubs under ## Beyond budget (zoom to expand) — nothing is silently dropped.

Why

Agents burn serial turns assembling context: search, then outline the hits, then zoom the symbols, then chase callers. Each turn round-trips through the model. A pack returns evidence (verbatim bodies with file:line headers), not conclusions — the agent reasons over it directly, ready to attach to a subagent dispatch.

Measured on a real config repo (3 questions, one call each vs. the manual chain):

• "how does the lane-verdict cache flow end-to-end" → complete 4-file chain (cache shapes, verdict logic, write/read, plugin deny hook) in one pack, used=283/400. Manual baseline: 5-6 tool calls.
• "how does score-tap decide when to write an executor row" → the full decision chain (event filter → dedup → context sources → verdict parse → DB insert) in one pack at budget=200.
• An unrehearsed cross-file question → correct core symbols plus their docstrings, first try.

Independently reproduced on the aft codebase itself: question: "how does bash output compression dispatch pick a compressor" → seeds=15, used=226/400, one pack assembling the full dispatch chain (compress → gate → compress_with_registry_exit_code 20-compressor array → Compressor trait → install path → subc mirror) that otherwise takes a 4–5-call search→zoom chain.

Honest degradation

The pack never lies about its own quality:

• While the semantic index is building, long NL queries degrade to lexical-only file-level hits. The pack renders them as visible file:line (no containing symbol) stubs and flags the header with degraded=semantic-index-building (partial results — retry when index ready) — detected via the response's semantic_status field, cleared as soon as one real seed resolves. No blocking or retry inside the tool.
• Grep-fallback hits ({file, line_text, line}, no symbol name) resolve to their containing symbol by line containment — definitions, call sites, and comment hits all upgrade to the enclosing symbol. Hits with no containing symbol stay visible as stubs.
• Unresolved external/stdlib callees collapse to one summary line ((N unresolved external calls omitted)) instead of drowning the stub list; unresolved seeds and callers are never suppressed.

Implementation

• crates/aft/src/commands/gather.rs — Rust-side composition: calls handle_semantic_search / impact_result / call_tree_result / render_symbol_within_budget directly (shared &AppContext, no bridge round-trips, no parallel reimplementation of search).
• Wiring: main.rs dispatch arm, subc_translate.rs mapping, TS factory packages/opencode-plugin/src/tools/gather.ts + registration (same tier as aft_callgraph — depends on the callgraph store).
• No new dependencies, no config surface beyond the tool args, no LLM calls, no caching.
• Follows the tri-state honest-reporting convention (protocol.rs Response doc-comment): success:false+code for un-performable calls (e.g. invalid_request on a bad mode combo), success:true with a visible degraded/stub pack for partial results — never a bare empty success.

Tests

22 unit tests in gather.rs, including red-checked regressions (each confirmed to fail against pre-fix code): mid-codepoint truncation panic, duplicate-symbol line-anchored resolution, abs/rel path dedupe, containing-symbol resolution via a real TreeSitterProvider, callee-only stub suppression driven through the production build_pack path, and degradation-flag presence/absence/mixed cases.

Limitations (deliberate scope)

• 1-hop expansion only — multi-hop was deliberately excluded: the budget math and ranking get harder and the packs get noisier.
• Callgraph-dependent by design: neighbor expansion quality follows the callgraph store's freshness, same as aft_callgraph.
• Budget counted in lines, not tokens — matches aft's existing budget idiom across zoom/outline.

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

---

Summary by cubic

Adds aft_gather_context, a single‑call, deterministic context‑pack builder that returns ranked, deduped, verbatim code within a fixed line budget, replacing the multi‑step search→outline→zoom→callgraph chain. Exposed as gather in Rust and aft_gather_context in @opencode on the “all” surface.

• New Features

  • Two modes (XOR): question (semantic seeds) or symbol+filePath (impact callers + call‑tree callees), 1‑hop expansion; dedupe by (file, symbol) with seeds winning.
  • Hard line budget (default 400, max 800) with per‑symbol balancing; overflow listed under “Beyond budget”; unresolved external callees collapse to one summary; unresolved seeds/callers stay visible.
  • Grep‑fallback hits resolve to containing symbols; no‑symbol hits render as visible stubs.
  • Honest header flags: degraded=semantic-index-building and neighbors=skipped(callgraph-unavailable); exact used lines computed post‑render.
  • Robustness: repo‑relative path resolution, abs/rel normalization, Unicode‑safe query truncation, best‑match selection for same‑name symbols by start_line. Tests cover truncation, dedupe, stubs, suppression, and path resolution.
  • Tool surface: added to @opencode as aft_gather_context (ALL‑only; inventory updated), with XOR schema validation and budget min/max enforced.

• Migration

  • Call with { question } or { symbol, filePath }; optional budget 1–800 (default 400).
  • Neighbor expansion requires the callgraph store in symbol mode; in question mode, neighbors are skipped with a header notice if unavailable.

^{Written for commit 0de9d84. Summary will update on new commits.}

Greptile Summary

This PR introduces aft_gather — a single-call, deterministic context-pack builder that replaces the serial search → outline → zoom → callgraph pattern agents previously needed for gathering code context. It is well-designed and thoroughly tested (22 unit tests), with honest degradation flags, callee-only stub suppression, path normalization, and Unicode-safe truncation, all wired correctly through the Rust dispatch, subc-translate, and TypeScript plugin layers.

• New gather command (gather.rs): two mutually exclusive modes (question via semantic search, symbol via callgraph impact), 1-hop neighbor expansion, dedup by (file, symbol) with seeds winning, and a hard line budget (default 400, cap 800) with over-budget symbols rendered as visible stubs rather than silently dropped.
• Off-by-one in line accounting for truncated/menu symbols: format!("{}\n{}\n{}", header, body, truncated_note) yields different .lines().count() results depending on whether truncated_note is empty (trailing \n ignored by Rust's .lines()) versus non-empty (no trailing \n, extra line counted). The +1 separator then overcounts by 1 per truncated symbol, causing used= to over-report and the budget guard to be slightly more conservative — directly contradicting the "exact used lines" stated design goal fixed in earlier iterations.

Confidence Score: 4/5

• Safe to merge after fixing the line-accounting overcount in render_symbol_section; the rest of the implementation is solid and well-tested.
• The only defect is in render_symbol_section: the format!("{}\n{}\n{}", header, body, truncated_note) pattern silently over-counts lines_used by 1 for every symbol that receives a truncation or menu note, causing the used= header to over-report and the budget guard to be slightly more aggressive than intended. Earlier review rounds fixed analogous undercounting bugs in the separator path, and this is the same class of problem on the opposite side. The fix is a one-line restructuring of the format call. All other logic — dedup, path normalization, callee suppression, degradation flags, TypeScript wiring — is correct and comprehensively exercised by the 22 unit tests.
• crates/aft/src/commands/gather.rs — specifically the render_symbol_section function and corresponding line-counting in build_pack.

Important Files Changed

Filename	Overview
crates/aft/src/commands/gather.rs	New 1460-line file implementing the context-pack builder. Contains a subtle off-by-one in line accounting for symbols with truncation/menu notes — the format string produces different .lines() counts depending on whether truncated_note is empty, causing used= to overcount by 1 per such symbol and the budget guard to be slightly more conservative than intended. Core logic (dedup, path normalization, callee suppression, degradation flags) is well-structured and thoroughly tested with 22 unit tests.
packages/opencode-plugin/src/tools/gather.ts	TypeScript tool adapter for aft_gather_context. Mode validation mirrors Rust-side translate_gather rules (XOR, symbol+filePath together, neither). Budget coercion and conditional rawArgs assembly are correct. Return type is Promise<string> and response.text is always set in the Rust success path.
packages/opencode-plugin/src/tool-registration.ts	gatherTools spread is unconditional (matching callgraph pattern); aft_gather_context added to ALL_ONLY_TOOLS so it is filtered from minimal/recommended surfaces. Registration is correct.
crates/aft/src/subc_translate.rs	translate_gather provides more granular mode validation than the Rust handler (symbol-without-filePath caught with specific message) and correctly resolves filePath against project_root before passing to the backend.
crates/aft/src/main.rs	Single dispatch arm added for "gather" → handle_gather. Clean, no issues.

Sequence Diagram

sequenceDiagram
    participant Agent
    participant GatherTS as gather.ts (TS)
    participant GatherRS as gather.rs (Rust)
    participant SemanticSearch
    participant CallgraphStore

    Agent->>GatherTS: "aft_gather_context({question|symbol+filePath, budget})"
    GatherTS->>GatherTS: XOR mode validation
    GatherTS->>GatherRS: tool_call("gather", rawArgs)

    alt question mode
        GatherRS->>SemanticSearch: "handle_semantic_search(query, top_k=15)"
        SemanticSearch-->>GatherRS: results[] + semantic_status
        GatherRS->>GatherRS: resolve grep-fallback hits via line containment
        GatherRS->>GatherRS: sort seeds by score
        GatherRS->>CallgraphStore: impact_result(seed) + call_tree_result(seed) per seed
        CallgraphStore-->>GatherRS: callers[] + callees[]
    else symbol mode
        GatherRS->>GatherRS: validate_path(filePath) → absolute path
        GatherRS->>CallgraphStore: "impact_result(filePath, symbol, depth=1)"
        CallgraphStore-->>GatherRS: callers[]
        GatherRS->>CallgraphStore: "call_tree_result(filePath, symbol, depth=1)"
        CallgraphStore-->>GatherRS: callees[]
    end

    GatherRS->>GatherRS: dedup_by_file_and_name (seeds win)
    GatherRS->>GatherRS: build_pack: render each candidate within per_symbol_budget
    Note over GatherRS: Budget guard: lines_used + section_lines + 1 > budget → stub
    Note over GatherRS: Suppress unresolved external callees, keep seeds/callers visible
    GatherRS->>GatherRS: "Build header last (used=exact_count)"
    GatherRS-->>GatherTS: "{text: "## gather pack | ..."}"
    GatherTS-->>Agent: pack text

Loading

_{Reviews (13): Last reviewed commit: "gather: deterministic context-pack tool ..." | Re-trigger Greptile}

[cubic-dev-ai]

4 issues found across 7 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[iceteaSA]

Follow-up on the two maintainability notes from the Greptile summary (they weren't separate review threads, so noting here) — both addressed in cf09b9b9:

• degraded expression flagged as always-false in the non-empty-seeds path — intentional: degraded only applies when zero symbol-level seeds resolve (a building index yields no seeds; a ready index with seeds is never degraded). Logic unchanged; added a comment on the why so it doesn't read as dead code.
• callee-suppression string-match coupling — extracted UNRESOLVED_MARKER ("(symbol not resolved)") and CALLEE_PROVENANCE_PREFIX ("callee-of-") consts, referenced by both the producer (render_symbol_section err arm / collect_callees_for_seed) and the consumer (build_pack suppression guard) so the match can't silently drift. Only test-site literals remain, intentionally — a test pointing at the const couldn't catch const drift.

Matched/rendered text is byte-identical; 23/23 gather tests green.

[iceteaSA]

Rebased onto current main (690bed59) and renamed the tool to aft_gather_context — the longer name reads more clearly at the call site.

The rename covers the advertised tool name and the agent-facing error strings only. The wire command stays "gather" (TS→Rust transport), as do the file and function names, matching how aft_callgraph maps to callers/call_tree/impact from navigation.ts. Both manifest entries were updated: the REG-V049-OC-ALL set and the HOSTONLY-V049-006 allowlist row.

Rebase conflicts were confined to ARCHITECTURE.md and STRUCTURE.md; all code auto-merged. tools/structure.ts and tools/lsp.ts are gone from main, so I kept your shorter tool lists and only re-inserted gather.ts.

Verified: registration-parity 10, tool-surface-transport-invariant 2, tools 10, cargo test gather 23, bun run build and cargo build clean. The built bundle contains aft_gather_context and zero occurrences of the old name.

Still one commit. Two notes on the full plugin suite, both pre-existing on main rather than from this branch: the two e2e outline command failures and the format_on_edit ones. main's own CI is currently red on 1d1a6968, 1da22bfe, and ef57355f, and 690bed59's run was cancelled — happy to hold this until that's sorted if you'd rather rebase onto a green base.