fix(router): use the LLM to pick from the full roster on low-confidence routes (v0.292.5) - #537
Merged
Merged
Conversation
…ce routes (v0.292.5)
Without an embedder the keyword scorer is weak — a task's words rarely match an
agent's description verbatim ("build a feature" doesn't lexically hit engineer),
so it disambiguated to the wrong few (app-packager/marketer/newsletter). The LLM
step now, on any low-confidence keyword result (disambiguate/none), picks the
single best-fit agent from EVERY agent's description instead of only re-ranking a
near-tie shortlist. So "build a feature" / "which agent can help me build a
feature" → engineer. Confident keyword routes are left alone (no LLM cost);
UNSURE keeps the keyword decision; no LLM configured → unchanged.
Fixes the wrong list behind Cockpit's "Route to an agent instead" and lifts
routing quality across Cockpit/Discord/Slack.
Verified: typecheck + build clean; test:governance 159/159 + tier-A + capability;
stubbed-model e2e (keyword-weak → full-roster LLM pick → engineer; confident
keyword → 0 LLM calls; no key → keyword-only unchanged).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the "wrong list" behind Cockpit's "Route to an agent instead" — and the underlying routing weakness it exposed.
Root cause
Routing "build a feature" returned
app-packager, marketer, newsletter-writer— notengineer. Without an embedder, the keyword scorer matches word overlap, and "build a feature" doesn't lexically hit engineer's description ("writes code, debugs, ships fixes"). So it disambiguated to the wrong few. The LLM step existed but only re-ranked the near-tie shortlist — it never reconsidered the full fleet, so it couldn't surface engineer.Fix
On any low-confidence keyword result (
disambiguate/none), if an LLM is configured, it now picks the single best-fit agent from every agent's description (llmTieBreak→llmPick, over the full roster). A confident keywordrouteis left alone (no LLM cost);UNSUREkeeps the keyword decision; no LLM configured → unchanged keyword-only behavior.Result — the reported case and its variants now resolve correctly:
This lifts routing quality across Cockpit, Discord, and Slack (all go through
chooseAgent), now that a Claude key is configured.Verification
typecheck+buildclean;test:governance→ 159/159 + tier-A 18/18 + capability 18/18After merge
Deploy, then in Cockpit hit "Route to an agent instead" on that ask answer (or just type "build a feature") → it should route to engineer, not the marketing agents.
src/edge/router.ts.🤖 Generated with Claude Code