Make search_issues semantic by default

[kelsey-myers]

Summary

Make search_issues use semantic search by default on hosts that support it, falling back to lexical on GHES.

Why

Closes github/plan-track-agentic-toolkit#729

An A/B experiment on the remote server compared lexical, semantic, hybrid and agent-chosen variants of search_issues over a week. Semantic reduced the number of searches an agent needed per session to 1.7, against lexical's 2.4, with query length roughly flat. Fewer searches means fewer round-trips of results into the agent's context.

Semantic search is not available on GitHub Enterprise Server, where the API returns 403 for search_type=semantic, so those hosts keep lexical behaviour.

What changed

• search_issues sends search_type=semantic on dotcom and GHEC.
• Free-text quotes are stripped from the query, since quoting asks the backend for an exact phrase match and defeats semantic matching. Quotes around qualifier values (label:"needs triage") are preserved.
• The tool and query descriptions now describe the engine the host will actually use. Steering a caller toward natural language is wrong advice on a lexical-only instance.
• New WithHost tool option, so tools with host-specific capabilities can adapt. utils.ParseHostType exposes the classification parseAPIHost was already doing internally.
• search_pull_requests is unaffected.

advanced_search is deliberately left alone. The semantic query class already extends the advanced one, so the parameter changes nothing when semantic runs, and on the lexical fallback it makes things worse: under advanced search an unparenthesised boolean binds repo: to only its own clause, so the search escapes the repository the caller asked about. Prod-verified over four query shapes; the existing field. conditional in prepareSearchArgs is unchanged, since field genuinely does require it.

Eval details — how the query description was chosen

The query parameter description was picked by A/B testing four phrasings against a set of benchmark cases that prompt with synonym lists ("auth, authentication, or login problems") to see whether the model reaches for boolean OR. OR isn't supported by semantic search, so the query silently degrades to lexical.

Measured as: of the queries emitted for those cases, how many contained OR.

Wording	OR usage	sonnet-4.5	gemini-2.5-pro	haiku-4.5
A — shipped "When the user gives alternative wordings, include them as plain words rather than joining them with OR."	6/15 (40%)	1/5	5/5	0/5
B "Semantic matching already finds related wording, so synonyms don't need listing or joining with OR."	8/19 (42%)	3/7	5/5	0/7
"Avoid boolean OR operators, which fall back to lexical search."	9/17 (53%)	4/5	5/5	0/7
lexical description (control)	7/12 (58%)	3/5	4/4	0/3
C "Semantic matching already covers related wording and synonyms." (no mention of OR)	10/17 (59%)	3/5	5/5	2/7

Findings:

• Naming the alternative behaviour is what works. A and B both do, and both beat the negative phrasing.
• Negation alone does nothing. "Avoid boolean OR" is indistinguishable from a description that never mentions OR.
• Silence is worst. C was the only condition where haiku emitted OR at all.
• gemini-2.5-pro is unmoved by any wording, 5/5 throughout.

MCP impact

• Tool schema or behavior changed

search_issues descriptions changed, and the underlying search engine changes on non-GHES hosts. Parameters are unchanged.

Prompts tested (tool changes only)

• "Find issues about login failing after a password reset in github/github-mcp-server"
• "Search for issues labelled needs triage in this repo"
• "Any open issues about slow rendering performance?"

Security / limits

• Data exposure, filtering, or token/size limits considered

Semantic issue search sits in a separate API rate limit family: 10 requests/minute versus 30/minute for standard search. This applies per request based on search_type, so making semantic the default lowers the effective rate limit for issue search on supported hosts. The experiment recorded no rate limit errors across ~76k semantic calls.

Tool renaming

• I am not renaming tools as part of this PR

Lint & tests

• Linted locally with ./script/lint
• Tested locally with ./script/test

Docs

• Updated (README / docs / examples)

[kelsey-myers]

Reasoning for why OSS as opposed to the remote server is detailed in github/plan-track-agentic-toolkit#729 (tradeoffs dropdown) — tl;dr: making a change to expose Client.BaseURL() to handle GHES cases was relatively cheap, we keep one source of truth for the tool description and avoid patching the tool with semantic-specific overrides in the remote repo, and make future changes easier to eval.

I'm open to discussing this though!

[Copilot]

Pull request overview

Makes issue search semantic by default on supported hosts, with lexical fallback for GHES.

Changes:

• Adds host-aware tool configuration and semantic search mode.
• Preserves qualifier quotes while removing free-text quotes.
• Updates tests, snapshots, and generated documentation.

Show a summary per file

File	Description
README.md	Updates query guidance.
pkg/utils/api.go	Exposes host classification.
pkg/github/tools.go	Adds host-aware tool options.
pkg/github/search_utils.go	Implements semantic search preparation.
pkg/github/search_semantic_test.go	Tests quote handling and descriptions.
pkg/github/issues.go	Makes issue search host-aware.
pkg/github/issues_test.go	Updates semantic request expectations.
pkg/github/inventory.go	Propagates tool options.
pkg/github/__toolsnaps__/search_issues.snap	Updates default schema snapshot.
pkg/github/__toolsnaps__/search_issues_ff_fields_param.snap	Updates flagged schema snapshot.
internal/ghmcp/server.go	Configures host type for stdio.
docs/insiders-features.md	Updates generated query guidance.
docs/feature-flags.md	Updates generated query guidance.

Review details

• Files reviewed: 13/13 changed files
• Comments generated: 2
• Review effort level: Medium

[Copilot]

Review details

Suppressed comments (3)

pkg/github/issues.go:1679

• Using one translation key for both engine-specific defaults breaks the host-aware description in HTTP mode. TranslationHelper caches the first value for a key (pkg/translations/translations.go:35-50), and HTTP startup first builds default-dotcom tools in initGlobalToolScopeMap (pkg/http/server.go:158-159) before constructing the GHES inventory. The semantic text is therefore cached and returned here even when mode is lexical. Use distinct semantic and lexical translation keys (while deciding how to preserve the existing override key) so the advertised engine matches the handler.

			Description: t("TOOL_SEARCH_ISSUES_DESCRIPTION", toolDescription),

pkg/github/search_utils.go:163

• Blindly deleting phrase quotes can turn literal words into search operators. For example, "auth OR login failure" becomes auth OR login failure; the uppercase OR is then interpreted as boolean syntax and, as described in this PR, forces lexical fallback. That defeats the purpose of unquoting and changes the phrase's meaning. Parse free-text quoted segments so reserved operators inside them are neutralized (for example by lowercasing them) when the quotes are removed.

	protected := qualifierQuotePattern.ReplaceAllString(query, "${1}"+sentinel+"${2}"+sentinel)
	stripped := strings.ReplaceAll(protected, `"`, "")
	return strings.ReplaceAll(stripped, sentinel, `"`)

pkg/github/issues.go:1694

• The request-level GHES fallback is not covered: the new description test checks only schema text, while every handler case constructs the zero-value semantic tool. Add a handler test using WithHost(utils.HostTypeGHES) that verifies search_type is omitted (and that semantic quote rewriting is not applied). This directly guards the fallback whose purpose is to prevent GHES 403 responses.

			result, err := searchIssuesHandler(ctx, deps, args, mode, options...)

• Files reviewed: 12/12 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[RossTarrant]

We might need to account for the translation cache here. In HTTP mode initGlobalToolScopeMap builds the tools without WithHost before the host-aware inventory is created. That caches the semantic description, which means a GHES server can advertise semantic search even though it correctly executes lexical search

Instead could we pass the configured host into the scope-map construction which should keep the description and behavior aligned