#170: FTS Track 3 — offline search index builder + tokenizer parity gate#329
Merged
Merged
Conversation
…hon parity gate Implements SEARCH_INDEX_V1.md against the isamplesorg#169 contract: - tools/search_tokenizer.py — canonical tokenizer (NFKC → lowercase → NFD diacritic strip → non-alphanumeric→space (Unicode-aware) → split → 1..64 length filter). Pure stdlib. Step 4 replaces ALL non-alphanumerics (incl. exotic whitespace) with plain spaces, which makes the split semantics identical across Python and JS by construction. - assets/js/search_tokenizer.js — the JS twin for the isamplesorg#171 browser query path. - tests/search_tokenizer_regression.json — 39 entries (contract: ≥30) covering diacritics, mixed case, hyphenation, IGSN/ark ids, numerics, NFKC folds (fullwidth/ligature/fraction), CJK/Cyrillic/Greek, URLs, 64/65-char length edges, empty/whitespace. expected_tokens generated FROM the Python implementation; the JS suite passing = parity proof. 42 Python + 42 JS tests green. - tools/build_search_index.py — document projection (sample.label / sample.description / sample.place_name from lite / concept.label via decorrelated unnest+join to IdentifiedConcept then vocab_labels prefLabel with URI-tail fallback + counter, mirroring build_frontend_derived.py's planner-safe pattern) → tokenize → token rows {token, pid, field, tf, doc_len} → hash-partitioned shards + df.parquet sidecar + build_stats.json (§10). Shard hash is FNV-1a 32-bit over UTF-8 — deliberately trivial to mirror in JS, since isamplesorg#171's browser reader must locate a token's shard client-side (DuckDB's hash() is not portable). Over-cap shards sub-shard by fnv1a32(pid) % M into shard_XXX_pY.parquet. Streaming build: DuckDB aggregates fragments per (pid,field), Python tokenizes in Arrow batches, intermediates spill to a temp dir, DuckDB does global DF + ordered shard writes. - tests/test_search_index_builder.py — 10-doc E2E fixture per the issue spec: proves URI dereferencing end-to-end ('pottery' returns exactly the <test://Pottery> pids), URI-tail fallback + stats counter, per-field projection + tf/doc_len, shard assignment matches fnv1a32, DF sidecar equals distinct (pid,field) counts, build_stats coverage numbers, and forced sub-sharding via a tiny cap. 8/8 green. Refs isamplesorg#170, isamplesorg#169, isamplesorg#165. CI wiring + full-corpus build_stats.json follow in this PR before review. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjnkWb4HpuLeDbYfzmY3X9
…) + builder fixture Mirrors pipeline-tests.yml's path-scoped pattern. Divergence between the two tokenizer implementations is a hard PR failure per SEARCH_INDEX_V1.md §2. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjnkWb4HpuLeDbYfzmY3X9
…rod build stats
Three empirical findings from the first full-corpus builds (each visible
in tools/build_stats/isamples_202608_search_index_v1_build_stats.json):
1. HOT TOKENS: vocabulary boilerplate ('material', 'object', 'solid'…)
has posting lists on ~5M samples — single tokens larger than the 5 MB
shard cap. The contract's §6 high-frequency rule is TOKEN-level:
hot tokens now live in hot/<fnv1a32 hex>_p{m}.parquet (M sized by
estimate, verified against the cap after writing, re-split at 2×M
until compliant), listed in hot_tokens.json; base shards stay small
so the COMMON query fetches one small file. Near-hot skew (multiple
~3 MB posting lists co-landing in a shard) is handled by promoting
the heaviest tokens out of any over-cap base shard until it fits.
Default shards 64→256. Result: 870 files, ZERO cap violations,
max file 5.17 MB, 1.42 GB total.
2. CONTRACT AMENDMENT (SEARCH_INDEX_V1.md header note): v1-as-written
indexed ZERO results for the benchmark's own example query
'pottery Cyprus' — "Pottery" is not in the 537-concept curated
vocabulary; it reaches samples only as an OpenContext KEYWORD
concept whose label lives on the IdentifiedConcept row. keywords is
pulled forward from v2, and resolution is now vocab.pref_label →
ic.label → URI tail. Post-fix: 'pottery Cyprus' = 1,305 matches
(top hits OpenContext Cyprus arks), 'pottery from Cyprus' identical
(query-time stopword policy), concept-only queries (ceramic/bone/
mammal) all non-empty, 'axial seamount summit caldera' = 284 =
exact ILIKE ground-truth parity.
3. Manifest-disk invariant: when the cap is unreachable (per-file
parquet overhead > cap — pathological, test-only), keep the finest
split on disk rather than deleting it, so hot_tokens.json always
describes real files.
Production build headline (202608, 6,726,892 samples, 506s):
concept.label coverage 100% (contract ≥90%), resolution 100%/99.98%/
100%/100% across material/context/object_type/keywords (contract
≥90%), 132 hot tokens, per-query fetch 1.4–5.5 MB for 1–2 token
queries against the §7 ≤5 MB cold budget.
Tests: 52/52 (tokenizer parity 42 + builder E2E 10, incl. new
keyword-via-ic.label case and hot-isolation semantics).
Refs isamplesorg#170, isamplesorg#169, isamplesorg#165.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjnkWb4HpuLeDbYfzmY3X9
P1 hot-filename hash collisions: fnv1a32 is 32-bit and distinct hot
tokens CAN collide — Codex produced a real pair ('tywtopf1ri' /
'32jnqttihd' → 0xa7c9bf62, verified). Keys are now uniquified with a
-N suffix on collision; readers already locate files via
hot_tokens.json, never by recomputing the hash. Regression test uses
the verified colliding pair and asserts distinct keys + zero
cross-contamination.
P1 tokenizer astral parity: JS length filter counted UTF-16 units
(t.length) where Python counts code points — a 33-char Deseret token
passed Python and failed JS (verified). Now [...t].length; 5 astral
regression entries added (33/64/65 code-point boundaries, mixed
Deseret, mathematical alphanumerics), all parity-green.
P2 USMALLINT truncation observability: doc_len/tf clamps are now
counted with observed maxima in build_stats.json
(usmallint_truncation) + a stderr warning when non-zero. Full 202608
build: ZERO truncations, max doc_len 220, max tf 19 — the claim is
now empirical, not narrative.
P2 final-promotion duplicate postings: exhausting the base-shard
promotion loop after a promotion left the promoted token in BOTH the
base file and hot/. The else branch now rewrites the base shard once
more before recording any violation.
Also per review: SEARCH_INDEX_V1.md §6/§8 now specify the concrete
hot-token layout (hot/<key>_pN.parquet, hot_tokens.json discovery
manifest, fetch-all-sub-files reader rule, fnv1a32 normative, N=256)
and §1's v1/v2 tables reflect the keywords amendment; the builder's
"bounded memory" claim softened to what's actually guaranteed.
Fresh full-corpus build committed: 926 files, 0 cap violations,
max 4.93 MiB, 139 hot tokens, all keys unique. Tests 58 Python +
47 JS, all green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjnkWb4HpuLeDbYfzmY3X9
Contributor
Author
Codex round 1: blocking findings — not LGTM (all verified real, all fixed in 39ef528)
Also per review: contract §6/§8 now specify the full hot-token layout + Fresh full-corpus build committed: 926 files, 0 cap violations, max 4.93 MiB, 139 hot tokens, zero truncations. Tests 58 Python + 47 JS. Round-2 re-review in flight; merge waits on its LGTM. |
…get (Codex round 2) Round-2 P1: a hot token exceeds 5 MiB of postings BY DEFINITION, so a reader that fetches all its sub-files necessarily violates the §7 cold-bytes budget — the amended contract contradicted itself for all 139 hot tokens. Resolution (classic common-term handling): - Hot postings (hot/<key>_p*.parquet) are NEVER fetched at query time; they remain for offline analysis and the isamplesorg#172 oracle. - Query policy (contract §3/§6): hot tokens are COMMON TERMS. Mixed hot+selective queries drop the hot terms from the AND (they sit on ~5M of 6.7M samples — near-zero selectivity; UI copy will say so). All-hot queries rank via a new build-time sidecar: - hot_topk.parquet — each hot token's static single-token BM25 top-500 (field-weighted per §5; IDF + doc_len norm are query-independent for a single token). Full-corpus sidecar: 0.75 MB for 139×500 rows — well under the cap. Multi-hot AND intersects top-K lists (documented approximation). - hot_tokens.json now records query_policy + topk_k + topk_bytes; budgets in §7 hold for EVERY query shape again. Fixture test extended: topk ranks are 1..n, tf ordering respected, K cap enforced. 58 Python + 47 JS green. Fresh full build committed: 927 files, 0 violations, 0 truncations. Note for isamplesorg#171: assets/js/search_substrate.js (stacked branch) still plans hot-file fetches — it will be reworked to the common-term policy when that branch rebases onto this. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjnkWb4HpuLeDbYfzmY3X9
…und 3)
Round-3 blocker fixed — hot_topk now implements §5 correctly:
- per-PID SUMMED field-weighted contributions (was: ranking individual
(pid, field) postings, letting one sample occupy multiple top-K slots
and never ranking its combined score)
- corpus statistics: totalDocs = distinct (pid, field) documents (was:
posting-row count); doc-length normalization uses the PER-FIELD
corpus average (was: per-token average)
- schema simplifies to {token, pid, static_score, rank}
- fixture extended per review: a multi-field pid (label + concept both
matching) proves single-row-per-pid summing beats the single-field
competitor; identical-twin docs prove deterministic pid-ascending
ties.
Round-3 semantic flag addressed with data — "verify all 139 hot tokens
are genuinely low-selectivity" — they are NOT: postings span 81,736
('island', 'genetic', 'guelph' — genuinely selective, promoted for
shard-storage reasons) to 7.5M ('sample'). Policy refined to TWO TIERS,
recorded per token in hot_tokens.json:
- fetchable (total sub-file bytes ≤ cap; 82 tokens on 202608): reader
fetches the token's hot/ sub-files and it joins the AND normally —
no semantic change, still within budget.
- non-fetchable common terms (55 tokens: 'sample', 'material',
'object', 'other', 'solid'… — pure vocabulary boilerplate): dropped
from AND with UI disclosure when other terms survive; all-common
queries rank via hot_topk. isamplesorg#172 benchmarks the semantic cost.
Contract §3/§6 amended accordingly (incl. the §3 AND-rule gap flagged
in round 3).
Fresh full build committed: 902 files, 0 violations, 0 truncations.
58 Python + 47 JS green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjnkWb4HpuLeDbYfzmY3X9
…ound 4) Round-4 blockers: 1. Per-query byte cap was never actually preserved — even two plain base shards can sum past 5 MB (a 4-term query fetches ~9 MB worst case; the earlier smoke run showed exactly that and it went unremarked). §7 is rewritten to say what is TRUE: the build enforces a per-FILE invariant (every base shard, hot sub-file, and the topk sidecar ≤ cap); the per-query cold-bytes/latency budget rows are defined at P50 over the §9 canonical benchmark, which is what isamplesorg#172 mechanically gates; worst case is n_selective_tokens × cap and is computable UP-FRONT — the builder now also emits shard_sizes.json (per-base-shard file bytes) so the reader/benchmark can compute expected transfer before fetching rather than guessing. 2. Contract self-consistency: §6's "hot postings are never fetched" sentence predated the two-tier rule (now scoped to the NON-fetchable tier); the manifest schema line now includes total_bytes/fetchable/ topk_bytes; §7's "budgets hold for every query shape" paragraph (false for multi-term queries) replaced by the guaranteed-vs- measured statement. Fixture asserts shard_sizes.json entries match true file sizes. 58 Python + 47 JS green. (hot_topk scoring confirmed correct in round 4.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjnkWb4HpuLeDbYfzmY3X9
…line-only (Codex round 5) Round-5 blocker: df.parquet (7.9 MB, over-cap) was required by BM25 at query time but neither cap-checked nor accounted. Fix: df ships as a column IN every shard/hot row (constant per token on token-sorted files — RLE-compresses to ~nothing; base total went 539→~550 MB) so the query path needs NO df fetch at all. Corpus BM25 constants (total_documents = distinct (pid,field) docs, per-field avg_doc_len) ride in the KB-scale build_stats.json. df.parquet remains as an offline/oracle artifact, marked as such in contract §4; §7 now enumerates the query path's complete fetch set. Fresh full build committed: 847 files, 0 violations, 0 truncations, df column verified against sidecar in fixture (59 Python + 47 JS green). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjnkWb4HpuLeDbYfzmY3X9
Contributor
Author
Codex 5.6 final verdict: LGTM — no blockers (round 6)The review ledger — every round improved the artifact:
Final shipped artifact (202608, committed build stats): 847 files, every one ≤ 5 MiB cap, 0 violations, 0 truncations, 6,726,892 samples / 17,366,831 documents, 137 hot tokens (82 fetchable / 55 common). 59 Python + 47 JS tests. Merging and uploading to |
rdhyee
added
the
CC+Codex+LGTM
8 tasks
rdhyee
added a commit
to rdhyee/isamplesorg.github.io
that referenced
this pull request
Jul 11, 2026
…ed FNV-1a assets/js/search_substrate.js — the browser query pipeline as PURE functions (no DuckDB, no fetch, no DOM), so every piece of query logic is Node-testable and the explorer's ?fts=v1 wiring (Phase B) stays thin: - fnv1a32: 32-bit FNV-1a with a shift-decomposed multiply (no BigInt); parity with tools/build_search_index.py pinned by tests/search_fnv1a_regression.json (values generated from Python, fixtures include CJK/Cyrillic/Greek inputs). - planQuery: tokenize → query-time stopword drop (contract §3 list) → duplicate-term dedup → shard/hot-file resolution (hot_tokens.json manifest aware) with cross-token file dedup. Distinguishes 'all stopwords' (controlled empty state) from 'no tokens'. - bm25Contribution: k1=1.2 b=0.75, contract §5 field weights. - combineAndRank: AND semantics, per-pid score summation, stable ordering (score desc, pid asc), TOP_K=50. 12/12 unit tests, incl. the isamplesorg#172 hard-fail preconditions: stopword near-equivalence ('pottery from Cyprus' plans identically to 'pottery Cyprus'), duplicate-term identity, all-stopword controlled empty, hot-vs-base file resolution. Phase B (explorer.qmd ?fts=v1 flag wiring + benchmark harness) follows separately. Stacked on isamplesorg#329 (feat/170-search-index-builder). Refs isamplesorg#171, isamplesorg#170, isamplesorg#169. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjnkWb4HpuLeDbYfzmY3X9
rdhyee
added a commit
to rdhyee/isamplesorg.github.io
that referenced
this pull request
Jul 11, 2026
…hot policy Sync with isamplesorg#329's merged contract (§6 two-tier rule + round-5 df embedding): - planQuery returns a mode ('empty'|'allStopwords'|'normal'|'topk'): non-fetchable common terms are dropped from the AND with an ignoredCommon disclosure list (UI must show it); fetchable hot tokens join the AND via their hot/ sub-files; all-common queries plan a single hot_topk.parquet fetch. - expectedBytes computed up-front from shard_sizes.json + manifest total_bytes (§7 transfer accounting; feeds the benchmark metric). - bm25Contribution takes {totalDocs, avgDocLenByField} from build_stats.json (per-field corpus averages — matches the builder's hot_topk scoring exactly); df arrives embedded in rows. 15 unit tests incl. the two-tier cases (drop-with-disclosure, fetchable joins AND, all-common topk mode, expectedBytes accounting). 110/110 across the repo. Refs isamplesorg#171. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjnkWb4HpuLeDbYfzmY3X9
rdhyee
added a commit
to rdhyee/isamplesorg.github.io
that referenced
this pull request
Jul 11, 2026
Feature-flagged wiring of the published sharded index (data.isamples.org/isamples_202608_search_index_v1/, PR isamplesorg#329): - buildSearchFilterSubstrate(term): plans via the unit-tested JS module (tokenize → stopwords → two-tier hot policy → file resolution), then scores in DuckDB SQL (BM25 §5, identical constants to the builder's hot_topk and the module's bm25Contribution; df embedded in rows — the query path fetches ONLY the plan's files). Produces the SAME singleton search_pids table (pid, label, source, place_name, relevance_score) as the interim path, so the table/points/facet-count semi-joins and the side panel are shared unchanged. - topk mode: all-common queries intersect the precomputed per-token top-500 sidecar. - Controlled-empty states for empty/all-stopword queries (§3 copy). - Three KB-scale sidecars (hot_tokens/shard_sizes/build_stats) fetched once on FIRST substrate search — flag-off visitors pay zero. - Disclosure (§3/§6): the results heading lists exactly which very common words were ignored ('material', 'sample', …) — silent semantic change is the one thing the policy forbids. - Route: doSearch's single buildSearchFilter call site branches on ?fts=v1. Default path byte-identical in behavior. Refs isamplesorg#171. Benchmark harness (the isamplesorg#172 input) is the remaining piece. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjnkWb4HpuLeDbYfzmY3X9
This was referenced Jul 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this implements (#170, per SEARCH_INDEX_V1.md / #169)
tools/search_tokenizer.py+assets/js/search_tokenizer.js— canonical tokenizer twins (NFKC → lowercase → diacritic strip → Unicode-aware punctuation→space → split → 1..64 filter). The punctuation step replaces all non-alphanumerics (incl. exotic whitespace) with plain spaces, making split semantics identical across implementations by construction.tests/search_tokenizer_regression.json— 39 entries (contract ≥30); expected tokens generated FROM the Python implementation, so the JS suite passing is the parity proof. New CI workflow (search-index-tests.yml) makes divergence a hard PR failure.tools/build_search_index.py— document projection → token rows{token, pid, field, tf, doc_len}→ 256 FNV-1a-hashed base shards +hot/token isolation +df.parquetsidecar +build_stats.json. FNV-1a-32 is deliberate: Explorer FTS Track 4: Browser query prototype + benchmark #171's browser reader must locate a token's shard in JS.tests/test_search_index_builder.py— 10-doc E2E fixture: URI dereferencing proof ('pottery' → exactly the<test://Pottery>pids), ic.label fallback, URI-tail + counter, tf/doc_len, shard assignment, DF sidecar, hot-isolation semantics. 52/52 tests.What the full corpus taught us (5 build iterations, all reproducible)
material,object,solid…) has posting lists on ~5M samples — bigger than the 5 MB cap per token. Implemented the contract's token-level rule: 132 hot tokens isolated tohot/sub-files (each cap-verified, re-split until compliant), listed inhot_tokens.json; near-hot shard skew handled by promoting heaviest tokens out of over-cap shards. Final: 870 files, zero cap violations, max 5.17 MB, 1.42 GB total.pottery Cyprus— "Pottery" isn't in the curated vocabulary; it reaches samples only as an OpenContext keyword concept. Pulledkeywordsforward from v2 + fall back to the concept's own label before the URI tail. The interim ILIKE search already covers keyword labels, so shipping without = recall regression = automatic Explorer FTS Track 5: GO/NO-GO decision gate #172 NO-GO.pottery Cyprus1,305 matches / 3.8 MB;pottery from Cyprusidentical (stopword policy ✓);ceramic/bone/mammalall non-empty (concept-only gate ✓);axial seamount summit caldera284 = exact parity with the live ILIKE search; no-hit query 0 matches / 1.4 MB.Production build headline (202608, 6,726,892 samples, 506 s)
tools/build_stats/isamples_202608_search_index_v1_build_stats.jsonNot in this PR
Uploading the index to
data.isamples.org(ops step; will follow the release-manifest coherence work), the browser query path (#171), and the GO/NO-GO gate (#172).Codex review pending — gpt-5.6 has been at capacity all morning; review will be run before merge (auto-retry loop is standing by).
🤖 Generated with Claude Code
https://claude.ai/code/session_01AjnkWb4HpuLeDbYfzmY3X9