Explorer FTS Track 5: GO/NO-GO decision gate

[rdhyee]

BLUF (plain English, updated 2026-07-17)

Where this stands: the new fast search shipped as the site default this morning
(PR #335; old search
survives at ?fts=off). That was a relative call — the new path beats the old on every
measured row — not a pass of this gate: the §7 end-to-end budgets below measure
search plus the page-wide refiltering that follows it, and that shared machinery
now dominates for either backend.

The next step, and our position — just take it: build the pin-overlay UI
(option C, already adopted in #164/#165 — search fills the side panel and pins matches
on the globe, instead of refiltering the whole page). It's the piece that makes search
feel instant and makes this gate's budgets winnable as written. Est. days, not weeks.

The only reasons we'd hold off — say so if any of these bite:

1. Runway — grant ends 7/31; pin-overlay competes with Zenodo closeout and the
   ~1-day coherence-manifest v0 (Substrate↔UI coherence: release manifest so sibling data files can't silently diverge #334). Our read: v0 + pin-overlay both fit if we start now.
2. UX re-confirm — option C changes behavior: search would pin results on top
   rather than filter the facet counts/table. The decision is from May; if anyone's
   expectation has drifted, now's the moment.
3. SME ranking verdicts (A/B links)
   could force ranking tuning first — but that work is independent and can run in parallel.

Absent an objection on 1–3, pin-overlay implementation starts next. Historical gate
spec follows below, unchanged.

---

Updated 2026-05-08 (rounds 1 + 2 per Codex review on #165). Round 1 added quality-gate cells. Round 2 sharpened: (a) "non-empty" is necessary but not sufficient — concept-only and stopword-heavy checks now require top-K relevance, not just any results; (b) added pathological-behavior hard-fails covering tokenizer parity, all-stopword queries, duplicate terms, edge-length tokens, missing display-join rows, filter composition; (c) NO-GO framing makes hosted-search a permanent contingency for either v1 failure or future v2+ quality requirements.

Sub-issue of #165. Depends on #171 (browser query prototype + benchmark data).

Goal

Mechanical decision gate. No budget renegotiation here — the budgets in #169's SEARCH_INDEX_V1.md are the contract.

Decision criterion

Does the prototype meet ALL of the cells below — every latency/bytes target, every quality target, and every hard-fail check?

A fast-but-mediocre search that fails any quality cell or any hard-fail check is NOT a GO, regardless of how it performs on the latency/bytes table.

Performance gates (hard)

metric	contract	prototype	pass?
cold first search (P50)	≤ 2 s	(fill)
warm repeat-same-query search	≤ 500 ms	(fill)
warm new-query-after-warm-up search	≤ 500 ms	(fill)
filter-composed cold search	≤ 3 s	(fill)
bytes transferred cold	≤ 5 MB	(fill)
bytes transferred warm	≤ 1 MB	(fill)

Quality gates (hard, not advisory)

metric	contract	prototype	pass?
top-3 overlap vs hand-labeled set	≥ TBD%	(fill)
top-10 overlap vs hand-labeled set	≥ TBD%	(fill)
top-10 overlap vs DuckDB FTS local oracle	≥ TBD%	(fill)
concept-only top-3 relevance: each of ceramic, bone, mammal (+ 1-2 more) returns ≥ 2 of 3 hand-labeled known-good PIDs	2/3 each	(fill)
stopword-heavy near-equivalence: top-K Jaccard between pottery from Cyprus and pottery Cyprus (stopword-stripped form)	≥ 0.8 top-10	(fill)

Numeric thresholds get filled in once #167 baseline + #171 prototype + DuckDB FTS oracle numbers land, so we know what "beats ILIKE" and "approaches BM25 oracle" mean on the canonical query set.

Hard-fail checks (any single failure = NO-GO)

Semantics

check	pass?
Concept-only queries (ceramic, bone, mammal) all return non-empty results
Concept-only queries hit the top-3 relevance bar in the quality table above
Stopword-heavy queries (pottery from Cyprus) return non-empty results
Stopword-heavy queries hit the near-equivalence bar in the quality table above
Diacritic queries (Çatalhöyük) match the diacritic-stripped index

Tokenizer + query parsing

check	pass?
Tokenizer parity: Python and JS produce identical token sequences for every term in the curated benchmark (not just the regression set)
All-stopword query (a the of) yields a controlled empty state with helpful copy, not an error or a full-corpus dump
Duplicate terms (pottery pottery cyprus) produce the same top-K result identity as pottery cyprus, within ranking-order tolerance
Empty / 1-char / very-long token queries: do not fetch broad shards; return an empty or error-with-copy state without long stalls
Wildcard literals (%, _) tokenize without errors

Display + composition

check	pass?
Missing display-join rows: substrate hit whose pid has no row in samples_map_lite does not crash and does not silently drop a top hit (either show with placeholder or document as known limit)
Filter composition matches a labeled expectation (one of two modes per (query, filter) pair). (a) Pair has a hand-labeled expected filtered top-K (in tests/search_benchmark.json); the substrate's filtered top-K must match it. (b) Filter is chosen such that ALL hand-labeled unfiltered top-K results satisfy it (e.g., source filter whose set covers every top-K result's source); the filtered top-K must equal the unfiltered top-K. The earlier "implicitly satisfies" wording was too loose — a top result that doesn't satisfy the filter legitimately drops out, so a raw top-K change is not necessarily a bug; the invariant has to be tied to labeled expectation. Tested on at least 3 distinct (query, filter) pairs.

Two outcomes

GO

• All performance cells pass.
• All quality cells pass.
• All hard-fail checks pass.
• Open ship issue: remove ?fts=v1 flag, route doSearch() permanently to substrate path, deprecate the ILIKE path.
• Update query-spec.qmd:225 to describe the substrate-backed search.
• Close Improve Interactive Explorer full-text search substrate #165 once ship issue lands.

A v1 GO does not close the hosted-search-backend question. It defers it. See NO-GO framing below for why hosted search remains a permanent contingency for v2+ requirements (richer analyzers, phrase search, typo tolerance, v2 field growth).

NO-GO

• At least one cell fails.
• File Explorer FTS Track 6: Hosted-search backend issue with:
  • the failed-cell data attached (which budgets, which quality, which hard-fails)
  • a starter requirements doc referencing Solr searchText semantics from query-spec.qmd:213-221
  • the DuckDB FTS local oracle numbers from Explorer FTS Track 4: Browser query prototype + benchmark #171 §5 as the relevance bar to clear
  • explicit framing: hosted-search is the answer if the static substrate is structurally limited; static-site constraint should not permanently cap search quality
• Keep the ?fts=v1 flag in place as a measurement tool until the hosted backend lands.
• Close Improve Interactive Explorer full-text search substrate #165 with a pointer to the hosted-search issue.

Hosted-search backend as a permanent contingency

The Track 6 hosted-search-backend issue may be triggered by either:

• (a) v1 GO/NO-GO failure — at least one cell fails the gate above.
• (b) Post-ship v2+ requirements — even on a v1 GO, future quality requirements (phrase search, typo tolerance, richer analyzers, v2 field growth that exceeds the static substrate's byte budget) may exceed what a static-Parquet substrate can deliver. When that happens, Track 6 fires for the same reasons the budget data would have triggered it under (a).

Both triggers file the same downstream issue with the same starter requirements doc.

Refs

#165, #169, #171

[rdhyee]

The hard-fail table is directionally right, but it is still too “non-empty result” oriented. Non-empty is necessary, not sufficient.

I would promote these to hard-fail checks or make them explicit subchecks under quality:

• concept-only queries must have known relevant hits in top-3/top-10, not merely any hit
• stopword-heavy query results should match or nearly match the stopword-stripped query (pottery from Cyprus vs pottery Cyprus)
• all-stopword input should produce a controlled empty/helpful state, not a broad query or exception
• tokenizer parity must hold for the exact benchmark terms across Python and JS, not just the standalone regression set
• duplicate terms should not distort ranking unexpectedly (pottery pottery Cyprus)
• empty / 1-char / over-64-char / punctuation-only inputs should not fetch broad shards
• missing display-join rows for top hits should not crash or silently erase the result set

I would keep numeric overlap thresholds too, but the above are semantic invariants. If any of them fail, the prototype is not shippable even if aggregate overlap and latency look acceptable.