#170: FTS Track 3 — offline search index builder + tokenizer parity gate

.github/workflows/search-index-tests.yml

@@ -0,0 +1,48 @@
+name: Search-index tests (tokenizer parity + builder)
+
+# The #170 gate: the Python and JS tokenizers must produce identical output
+# for every entry in tests/search_tokenizer_regression.json (both suites run
+# against the same JSON — divergence is a hard fail per SEARCH_INDEX_V1.md
+# §2), and the offline index builder must pass its end-to-end fixture
+# (URI dereferencing, shard structure, DF/doc_len, build_stats).
+on:
+  pull_request:
+    paths:
+      - "tools/search_tokenizer.py"
+      - "tools/build_search_index.py"
+      - "assets/js/search_tokenizer.js"
+      - "tests/search_tokenizer_regression.json"
+      - "tests/test_search_tokenizer.py"
+      - "tests/test_search_index_builder.py"
+      - "tests/unit/search-tokenizer.test.mjs"
+      - "scripts/requirements.txt"
+      - ".github/workflows/search-index-tests.yml"
+  push:
+    branches: [main]
+    paths:
+      - "tools/search_tokenizer.py"
+      - "tools/build_search_index.py"
+      - "assets/js/search_tokenizer.js"
+      - "tests/search_tokenizer_regression.json"
+      - "tests/test_search_tokenizer.py"
+      - "tests/test_search_index_builder.py"
+      - "tests/unit/search-tokenizer.test.mjs"
+  workflow_dispatch:
+
+jobs:
+  tokenizer-parity-and-builder:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+      - name: Install Python deps
+        run: pip install -r scripts/requirements.txt
+      - name: Python tokenizer + builder fixture tests
+        run: python -m pytest tests/test_search_tokenizer.py tests/test_search_index_builder.py -q
+      - name: JS tokenizer parity (same regression JSON)
+        run: node --test tests/unit/search-tokenizer.test.mjs

SEARCH_INDEX_V1.md

@@ -1,5 +1,20 @@
 # Search Index v1 Contract
 
+> **Amendment 2026-07-10** (first full-corpus build, #170): two changes to
+> §1's v1 minimum, discovered empirically. (1) **`keywords` is pulled
+> forward from the v2 list** into `concept.label`'s sources; (2) URI
+> dereferencing falls back to **the concept's own `label` column in the
+> wide** before the URI tail. Rationale: as originally written, v1 indexed
+> ZERO results for the benchmark's own example query `pottery Cyprus` —
+> "Pottery" is not a concept in the 537-entry curated vocabulary; it
+> reaches samples only as an OpenContext *keyword* concept whose label
+> lives on the IdentifiedConcept row. The interim ILIKE search already
+> covers keyword-concept labels (via `build_frontend_derived.py`'s
+> appended `concept_labels`), so v1-as-written was a recall REGRESSION
+> and an automatic #172 NO-GO. Resolution order is now:
+> `vocab_labels.pref_label` → `IdentifiedConcept.label` → URI tail
+> (with the missing-label counter tracking the last case only).
+
 Doc-only contract for the iSamples Explorer's full-text search substrate.
 Closes [#169](https://github.com/isamplesorg/isamplesorg.github.io/issues/169);
 inputs are consumed by the offline builder ([#170](https://github.com/isamplesorg/isamplesorg.github.io/issues/170)),
@@ -36,7 +51,7 @@ name** (entity dot field), not the source parquet column.
 | `sample.label`        | `MaterialSampleRecord.label` (~6.68M coverage)                      | canonical title; near-universal                            |
 | `sample.description`  | `MaterialSampleRecord.description` (~1.61M ≈ 24%)                   | sparse but high-signal where present                       |
 | `sample.place_name`   | `samples_map_lite.parquet.place_name[]` (~2.21M)                    | already proven valuable in current ILIKE search            |
-| `concept.label`       | `material` / `context` / `object_type` URIs dereferenced via `vocab_labels.parquet` (`pref_label`, `lang=en`) | **load-bearing**: facet URIs are near-universal but raw URIs are useless to FTS; dereferenced labels make `pottery`, `ceramic`, `basalt`, `bone`, `marine` work as the user expects |
+| `concept.label`       | `material` / `context` / `object_type` / **`keywords`** concept refs, resolved `vocab_labels.pref_label` → `IdentifiedConcept.label` → URI tail (2026-07-10 amendment — see header note) | **load-bearing**: facet URIs are near-universal but raw URIs are useless to FTS; dereferenced labels make `ceramic`, `basalt`, `bone`, `marine` work — and the keyword-concept labels are the ONLY path by which `pottery` reaches samples |
 
 A sample whose `material` URI is `<…>/Pottery` gets a row
 `{token: 'pottery', pid: …, field: 'concept.label', tf: 1, …}`. One row
@@ -62,7 +77,7 @@ per sample per facet URI per token after tokenization.
 | `curation.label`         | `curation_label`                                      |
 | `curation.description`   | `curation_description`                                |
 | `curation.location`      | `curation_location`                                   |
-| `keywords`               | (if present)                                          |
+| ~~`keywords`~~           | **moved into v1** (2026-07-10 amendment — header note) |
 | `source`                 | `source` enum (low value as FTS — facet UI suffices)  |
 
 ---
@@ -102,6 +117,14 @@ filtering applied to the user input *before* combining tokens.
   empty state with helpful copy ("All terms in your query are common
   words. Add a more specific term to search.") — never a full-corpus
   dump or an error.
+- **Non-fetchable hot (common-term) tokens** (§6): after stopword
+  removal, manifest tokens with `fetchable: false` are ALSO dropped
+  from the AND when at least one other token survives — the UI must
+  disclose exactly which terms were ignored. If ALL surviving tokens
+  are non-fetchable-hot, rank via `hot_topk.parquet` (§6). Fetchable
+  hot tokens participate in the AND normally (their sub-files fit the
+  budget). #172's benchmark quantifies the semantic cost (dropped-term
+  recall, multi-common top-K intersection empty-rate).
 - **No query-language syntax in v1.** No quoted phrases, no
   field-prefix operators, no booleans, no negation, no wildcards, no
   fuzzy. Documented v2 path: phrase quoting first (cheap and
@@ -127,7 +150,15 @@ Two parquet outputs per data version.
 }
 ```
 
-**Sidecar `df.parquet`** (global per-token document frequency):
+**Shipped shard/hot rows additionally carry `df: UINTEGER`** (round-5
+amendment): df is constant per token and the files are token-sorted, so
+the column RLE-compresses to almost nothing — and the query path then
+needs NO extra fetch for IDF. Corpus constants (`total_documents`,
+per-field `avg_doc_len`) ship in `build_stats.json` (KB-sized).
+
+**Sidecar `df.parquet`** (global per-token document frequency — an
+OFFLINE artifact for builds/analysis/#172 oracle; ~8 MB at corpus scale,
+over-cap, never fetched by the query path):
 
 ```
 {
@@ -161,13 +192,51 @@ in `EXPLORER_STATE.md` §6).
 
 ## 6. Partition shape
 
-- Hash-partition by token: `hash(token) % N` shards.
-- Per-shard byte cap: **≤ 5 MB** uncompressed parquet.
-- High-frequency token rule: if a single token's postings would exceed
-  the cap, sub-shard by `hash(pid) % M` within that token's logical
-  shard.
-- Number of top-level shards (`N`): start with **64**; refine in build
-  measurement.
+*(Amended 2026-07-10 with the concrete mechanics from #170's full-corpus
+builds — the original text predates the discovery manifest.)*
+
+- Hash-partition by token: `fnv1a32(utf8(token)) % N` base shards
+  (`shard_000.parquet` … zero-padded). FNV-1a-32 is normative — the
+  browser reader (#171) must compute it in JS.
+- Per-shard byte cap: **≤ 5 MB** parquet FILE bytes (stricter than
+  uncompressed — file bytes are what a browser transfers).
+- **Hot-token rule** (token-level, not shard-level): a token whose
+  postings alone would exceed the cap is REMOVED from its base shard and
+  written to `hot/<key>_p{0..M-1}.parquet`, where `<key>` is the token's
+  fnv1a32 hex (suffixed `-1`, `-2`… on the rare 32-bit collision) and M
+  is sized so each sub-file fits the cap (verified after writing;
+  re-split at 2×M until compliant). Whether a hot token's postings are
+  fetched at query time is decided by the two-tier rule below;
+  NON-fetchable hot postings exist for offline analysis and the #172
+  oracle only (Codex round-2 finding, 2026-07-10; tiering round-3/4).
+- **Two-tier hot-token query rule** (the reconciliation, refined per
+  round-3 review): hotness is a STORAGE property; selectivity is
+  semantic, and the manifest records both. Each hot entry carries
+  `total_bytes` and `fetchable` (= total_bytes ≤ cap).
+  - **Fetchable hot** (e.g. 'island', ~82K postings, ~2.5 MB total):
+    the reader fetches ALL its `hot/` sub-files and treats the token
+    exactly like a normal AND term — still within the cold budget.
+  - **Non-fetchable hot** (true common terms — 'sample', 'material',
+    millions of postings): NEVER fetched. Mixed with ≥1 selective term
+    → dropped from the AND (UI discloses exactly which terms were
+    ignored). All-common queries rank via **`hot_topk.parquet`** — a
+    build-time sidecar holding each hot token's static single-token
+    BM25 top-500 **per-pid summed, field-weighted per §5** (single
+    small file, cap-checked). Multi-common AND intersects top-K lists
+    (documented approximation; #172 benchmarks the empty-rate).
+- **Discovery manifest `hot_tokens.json`** ships beside the shards:
+  `{cap_bytes, query_policy, topk_k, topk_bytes, tokens: {token: {key,
+  sub_files, postings, total_bytes, fetchable}}}`. Reader algorithm:
+  token in manifest → two-tier rule above; otherwise → fetch
+  `shard_{fnv1a32(token) % N}.parquet`.
+- **`shard_sizes.json`** (sidecar): file byte size of every base shard,
+  so the reader can compute a query's expected transfer BEFORE fetching
+  (per-query budgeting + the benchmark's bytes-transferred metric are
+  computed from it, not guessed).
+- Near-hot skew guard: after writing each base shard, its heaviest
+  tokens are promoted to `hot/` until the file fits the cap.
+- Number of top-level shards (`N`): **256** (64 proved too few — ~9 MB
+  base shards on the 202608 corpus); recorded in `build_stats.json`.
 
 ---
 
@@ -178,13 +247,31 @@ this table.
 
 | metric                                          | target           |
 |-------------------------------------------------|------------------|
-| cold first search (P50)                         | ≤ 2 s            |
+| cold first search (P50 over the §9 benchmark)   | ≤ 2 s            |
 | warm repeat-same-query search                   | ≤ 500 ms         |
 | warm new-query-after-warm-up search             | ≤ 500 ms         |
 | filter-composed cold search                     | ≤ 3 s            |
-| bytes transferred cold                          | ≤ 5 MB           |
+| bytes transferred cold (P50 over the §9 benchmark) | ≤ 5 MB        |
 | bytes transferred warm                          | ≤ 1 MB per query |
 
+**What is guaranteed vs measured** (round-4 honesty pass): the BUILD
+enforces a per-FILE invariant — every base shard, hot sub-file, and the
+`hot_topk` sidecar is ≤ cap (5 MiB). A query's cold transfer is the SUM
+of its tokens' files: bounded by `n_selective_tokens × cap` (+ one
+sidecar read for common terms), NOT by a flat 5 MB for arbitrarily long
+queries — a 4-term query can legitimately fetch ~9 MB worst-case. The
+budget row above is therefore defined over the §9 canonical benchmark
+(realistic 1–3 term queries) at P50, which is what the #172 gate
+mechanically evaluates; per-query expected transfer is computable
+up-front from `shard_sizes.json` + the manifest, and the benchmark
+records actual bytes per query. The query path's ONLY fetches are:
+token shard/hot sub-files (each ≤ cap, df embedded), optionally
+`hot_topk.parquet` (≤ cap), and the KB-scale `build_stats.json` +
+`hot_tokens.json` + `shard_sizes.json` loaded once at boot —
+`df.parquet` is offline-only. Row-group pruning under HTTP range
+requests (when #190's fallback doesn't fire) reduces real transfer
+below file sizes; budgets do NOT assume it.
+
 **"Warm" disambiguation** (resolves [#174](https://github.com/isamplesorg/isamplesorg.github.io/issues/174)):
 
 - *Warm-repeat-same-query*: same query, second invocation, same page.
@@ -207,7 +294,8 @@ the specific percentages.
 ## 8. Versioning
 
 - URL pattern: `https://data.isamples.org/isamples_YYYYMM_search_index_v1/<shard>.parquet`
-  with sidecar `df.parquet` and `build_stats.json` (§10).
+  with `hot/<key>_pN.parquet` sub-files and sidecars `df.parquet`,
+  `hot_tokens.json` (§6), and `build_stats.json` (§10).
 - Explorer pins to a specific `YYYYMM` so a dataset rebuild can't
   break a deployed site mid-flight.
 - Index version is tied to data version. v1.x format bumps require

assets/js/search_tokenizer.js

@@ -0,0 +1,38 @@
+// Canonical JS tokenizer for the iSamples search substrate (#169 §2, #170).
+//
+// MUST stay in lockstep with the Python twin `tools/search_tokenizer.py`.
+// Both run against `tests/search_tokenizer_regression.json` in CI; any
+// divergence is a hard failure (SEARCH_INDEX_V1.md §2).
+//
+// Pipeline (order matters, part of the contract):
+//   1. NFKC normalize  2. lowercase  3. diacritic strip (NFD, drop \p{Mn},
+//   NFC)  4. non-alphanumeric -> space (Unicode-aware: \p{L}\p{N} survive,
+//   so `Iron-Age` -> `iron age`, `IGSN:HRV000ABC` -> `igsn hrv000abc`)
+//   5. whitespace split  6. length filter 1..64.
+//
+// Parity note: step 4 replaces every non-alphanumeric character (including
+// all exotic whitespace) with a plain space, so step 5's split semantics
+// cannot diverge between Python str.split() and this implementation.
+//
+// No stemming; no stopword removal here (query-time policy, §3).
+
+export const MAX_TOKEN_LEN = 64;
+
+// Length is counted in Unicode CODE POINTS ([...t].length), not UTF-16
+// units (t.length) — Python's len(str) counts code points, so astral-plane
+// tokens (e.g. Deseret 𐐀, 2 UTF-16 units each) would otherwise pass the
+// filter in Python and fail it here (Codex review of #329, verified).
+const codePointLen = (t) => [...t].length;
+
+export function tokenize(text) {
+    if (!text) return [];
+    let s = String(text).normalize('NFKC');          // 1
+    s = s.toLowerCase();                              // 2
+    s = s.normalize('NFD').replace(/\p{Mn}/gu, '').normalize('NFC'); // 3
+    s = s.replace(/[^\p{L}\p{N}]/gu, ' ');            // 4
+    return s.split(' ')                               // 5
+        .filter(t => {                                // 6
+            const n = codePointLen(t);
+            return n >= 1 && n <= MAX_TOKEN_LEN;
+        });
+}