diff --git a/.github/workflows/search-index-tests.yml b/.github/workflows/search-index-tests.yml new file mode 100644 index 0000000..e57b4e2 --- /dev/null +++ b/.github/workflows/search-index-tests.yml @@ -0,0 +1,48 @@ +name: Search-index tests (tokenizer parity + builder) + +# The #170 gate: the Python and JS tokenizers must produce identical output +# for every entry in tests/search_tokenizer_regression.json (both suites run +# against the same JSON — divergence is a hard fail per SEARCH_INDEX_V1.md +# §2), and the offline index builder must pass its end-to-end fixture +# (URI dereferencing, shard structure, DF/doc_len, build_stats). +on: + pull_request: + paths: + - "tools/search_tokenizer.py" + - "tools/build_search_index.py" + - "assets/js/search_tokenizer.js" + - "tests/search_tokenizer_regression.json" + - "tests/test_search_tokenizer.py" + - "tests/test_search_index_builder.py" + - "tests/unit/search-tokenizer.test.mjs" + - "scripts/requirements.txt" + - ".github/workflows/search-index-tests.yml" + push: + branches: [main] + paths: + - "tools/search_tokenizer.py" + - "tools/build_search_index.py" + - "assets/js/search_tokenizer.js" + - "tests/search_tokenizer_regression.json" + - "tests/test_search_tokenizer.py" + - "tests/test_search_index_builder.py" + - "tests/unit/search-tokenizer.test.mjs" + workflow_dispatch: + +jobs: + tokenizer-parity-and-builder: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + - uses: actions/setup-node@v4 + with: + node-version: "20" + - name: Install Python deps + run: pip install -r scripts/requirements.txt + - name: Python tokenizer + builder fixture tests + run: python -m pytest tests/test_search_tokenizer.py tests/test_search_index_builder.py -q + - name: JS tokenizer parity (same regression JSON) + run: node --test tests/unit/search-tokenizer.test.mjs diff --git a/SEARCH_INDEX_V1.md b/SEARCH_INDEX_V1.md index 291d0ca..5516adb 100644 --- a/SEARCH_INDEX_V1.md +++ b/SEARCH_INDEX_V1.md @@ -1,5 +1,20 @@ # Search Index v1 Contract +> **Amendment 2026-07-10** (first full-corpus build, #170): two changes to +> §1's v1 minimum, discovered empirically. (1) **`keywords` is pulled +> forward from the v2 list** into `concept.label`'s sources; (2) URI +> dereferencing falls back to **the concept's own `label` column in the +> wide** before the URI tail. Rationale: as originally written, v1 indexed +> ZERO results for the benchmark's own example query `pottery Cyprus` — +> "Pottery" is not a concept in the 537-entry curated vocabulary; it +> reaches samples only as an OpenContext *keyword* concept whose label +> lives on the IdentifiedConcept row. The interim ILIKE search already +> covers keyword-concept labels (via `build_frontend_derived.py`'s +> appended `concept_labels`), so v1-as-written was a recall REGRESSION +> and an automatic #172 NO-GO. Resolution order is now: +> `vocab_labels.pref_label` → `IdentifiedConcept.label` → URI tail +> (with the missing-label counter tracking the last case only). + Doc-only contract for the iSamples Explorer's full-text search substrate. Closes [#169](https://github.com/isamplesorg/isamplesorg.github.io/issues/169); inputs are consumed by the offline builder ([#170](https://github.com/isamplesorg/isamplesorg.github.io/issues/170)), @@ -36,7 +51,7 @@ name** (entity dot field), not the source parquet column. | `sample.label` | `MaterialSampleRecord.label` (~6.68M coverage) | canonical title; near-universal | | `sample.description` | `MaterialSampleRecord.description` (~1.61M ≈ 24%) | sparse but high-signal where present | | `sample.place_name` | `samples_map_lite.parquet.place_name[]` (~2.21M) | already proven valuable in current ILIKE search | -| `concept.label` | `material` / `context` / `object_type` URIs dereferenced via `vocab_labels.parquet` (`pref_label`, `lang=en`) | **load-bearing**: facet URIs are near-universal but raw URIs are useless to FTS; dereferenced labels make `pottery`, `ceramic`, `basalt`, `bone`, `marine` work as the user expects | +| `concept.label` | `material` / `context` / `object_type` / **`keywords`** concept refs, resolved `vocab_labels.pref_label` → `IdentifiedConcept.label` → URI tail (2026-07-10 amendment — see header note) | **load-bearing**: facet URIs are near-universal but raw URIs are useless to FTS; dereferenced labels make `ceramic`, `basalt`, `bone`, `marine` work — and the keyword-concept labels are the ONLY path by which `pottery` reaches samples | A sample whose `material` URI is `<…>/Pottery` gets a row `{token: 'pottery', pid: …, field: 'concept.label', tf: 1, …}`. One row @@ -62,7 +77,7 @@ per sample per facet URI per token after tokenization. | `curation.label` | `curation_label` | | `curation.description` | `curation_description` | | `curation.location` | `curation_location` | -| `keywords` | (if present) | +| ~~`keywords`~~ | **moved into v1** (2026-07-10 amendment — header note) | | `source` | `source` enum (low value as FTS — facet UI suffices) | --- @@ -102,6 +117,14 @@ filtering applied to the user input *before* combining tokens. empty state with helpful copy ("All terms in your query are common words. Add a more specific term to search.") — never a full-corpus dump or an error. +- **Non-fetchable hot (common-term) tokens** (§6): after stopword + removal, manifest tokens with `fetchable: false` are ALSO dropped + from the AND when at least one other token survives — the UI must + disclose exactly which terms were ignored. If ALL surviving tokens + are non-fetchable-hot, rank via `hot_topk.parquet` (§6). Fetchable + hot tokens participate in the AND normally (their sub-files fit the + budget). #172's benchmark quantifies the semantic cost (dropped-term + recall, multi-common top-K intersection empty-rate). - **No query-language syntax in v1.** No quoted phrases, no field-prefix operators, no booleans, no negation, no wildcards, no fuzzy. Documented v2 path: phrase quoting first (cheap and @@ -127,7 +150,15 @@ Two parquet outputs per data version. } ``` -**Sidecar `df.parquet`** (global per-token document frequency): +**Shipped shard/hot rows additionally carry `df: UINTEGER`** (round-5 +amendment): df is constant per token and the files are token-sorted, so +the column RLE-compresses to almost nothing — and the query path then +needs NO extra fetch for IDF. Corpus constants (`total_documents`, +per-field `avg_doc_len`) ship in `build_stats.json` (KB-sized). + +**Sidecar `df.parquet`** (global per-token document frequency — an +OFFLINE artifact for builds/analysis/#172 oracle; ~8 MB at corpus scale, +over-cap, never fetched by the query path): ``` { @@ -161,13 +192,51 @@ in `EXPLORER_STATE.md` §6). ## 6. Partition shape -- Hash-partition by token: `hash(token) % N` shards. -- Per-shard byte cap: **≤ 5 MB** uncompressed parquet. -- High-frequency token rule: if a single token's postings would exceed - the cap, sub-shard by `hash(pid) % M` within that token's logical - shard. -- Number of top-level shards (`N`): start with **64**; refine in build - measurement. +*(Amended 2026-07-10 with the concrete mechanics from #170's full-corpus +builds — the original text predates the discovery manifest.)* + +- Hash-partition by token: `fnv1a32(utf8(token)) % N` base shards + (`shard_000.parquet` … zero-padded). FNV-1a-32 is normative — the + browser reader (#171) must compute it in JS. +- Per-shard byte cap: **≤ 5 MB** parquet FILE bytes (stricter than + uncompressed — file bytes are what a browser transfers). +- **Hot-token rule** (token-level, not shard-level): a token whose + postings alone would exceed the cap is REMOVED from its base shard and + written to `hot/_p{0..M-1}.parquet`, where `` is the token's + fnv1a32 hex (suffixed `-1`, `-2`… on the rare 32-bit collision) and M + is sized so each sub-file fits the cap (verified after writing; + re-split at 2×M until compliant). Whether a hot token's postings are + fetched at query time is decided by the two-tier rule below; + NON-fetchable hot postings exist for offline analysis and the #172 + oracle only (Codex round-2 finding, 2026-07-10; tiering round-3/4). +- **Two-tier hot-token query rule** (the reconciliation, refined per + round-3 review): hotness is a STORAGE property; selectivity is + semantic, and the manifest records both. Each hot entry carries + `total_bytes` and `fetchable` (= total_bytes ≤ cap). + - **Fetchable hot** (e.g. 'island', ~82K postings, ~2.5 MB total): + the reader fetches ALL its `hot/` sub-files and treats the token + exactly like a normal AND term — still within the cold budget. + - **Non-fetchable hot** (true common terms — 'sample', 'material', + millions of postings): NEVER fetched. Mixed with ≥1 selective term + → dropped from the AND (UI discloses exactly which terms were + ignored). All-common queries rank via **`hot_topk.parquet`** — a + build-time sidecar holding each hot token's static single-token + BM25 top-500 **per-pid summed, field-weighted per §5** (single + small file, cap-checked). Multi-common AND intersects top-K lists + (documented approximation; #172 benchmarks the empty-rate). +- **Discovery manifest `hot_tokens.json`** ships beside the shards: + `{cap_bytes, query_policy, topk_k, topk_bytes, tokens: {token: {key, + sub_files, postings, total_bytes, fetchable}}}`. Reader algorithm: + token in manifest → two-tier rule above; otherwise → fetch + `shard_{fnv1a32(token) % N}.parquet`. +- **`shard_sizes.json`** (sidecar): file byte size of every base shard, + so the reader can compute a query's expected transfer BEFORE fetching + (per-query budgeting + the benchmark's bytes-transferred metric are + computed from it, not guessed). +- Near-hot skew guard: after writing each base shard, its heaviest + tokens are promoted to `hot/` until the file fits the cap. +- Number of top-level shards (`N`): **256** (64 proved too few — ~9 MB + base shards on the 202608 corpus); recorded in `build_stats.json`. --- @@ -178,13 +247,31 @@ this table. | metric | target | |-------------------------------------------------|------------------| -| cold first search (P50) | ≤ 2 s | +| cold first search (P50 over the §9 benchmark) | ≤ 2 s | | warm repeat-same-query search | ≤ 500 ms | | warm new-query-after-warm-up search | ≤ 500 ms | | filter-composed cold search | ≤ 3 s | -| bytes transferred cold | ≤ 5 MB | +| bytes transferred cold (P50 over the §9 benchmark) | ≤ 5 MB | | bytes transferred warm | ≤ 1 MB per query | +**What is guaranteed vs measured** (round-4 honesty pass): the BUILD +enforces a per-FILE invariant — every base shard, hot sub-file, and the +`hot_topk` sidecar is ≤ cap (5 MiB). A query's cold transfer is the SUM +of its tokens' files: bounded by `n_selective_tokens × cap` (+ one +sidecar read for common terms), NOT by a flat 5 MB for arbitrarily long +queries — a 4-term query can legitimately fetch ~9 MB worst-case. The +budget row above is therefore defined over the §9 canonical benchmark +(realistic 1–3 term queries) at P50, which is what the #172 gate +mechanically evaluates; per-query expected transfer is computable +up-front from `shard_sizes.json` + the manifest, and the benchmark +records actual bytes per query. The query path's ONLY fetches are: +token shard/hot sub-files (each ≤ cap, df embedded), optionally +`hot_topk.parquet` (≤ cap), and the KB-scale `build_stats.json` + +`hot_tokens.json` + `shard_sizes.json` loaded once at boot — +`df.parquet` is offline-only. Row-group pruning under HTTP range +requests (when #190's fallback doesn't fire) reduces real transfer +below file sizes; budgets do NOT assume it. + **"Warm" disambiguation** (resolves [#174](https://github.com/isamplesorg/isamplesorg.github.io/issues/174)): - *Warm-repeat-same-query*: same query, second invocation, same page. @@ -207,7 +294,8 @@ the specific percentages. ## 8. Versioning - URL pattern: `https://data.isamples.org/isamples_YYYYMM_search_index_v1/.parquet` - with sidecar `df.parquet` and `build_stats.json` (§10). + with `hot/_pN.parquet` sub-files and sidecars `df.parquet`, + `hot_tokens.json` (§6), and `build_stats.json` (§10). - Explorer pins to a specific `YYYYMM` so a dataset rebuild can't break a deployed site mid-flight. - Index version is tied to data version. v1.x format bumps require diff --git a/assets/js/search_tokenizer.js b/assets/js/search_tokenizer.js new file mode 100644 index 0000000..f3dc6f3 --- /dev/null +++ b/assets/js/search_tokenizer.js @@ -0,0 +1,38 @@ +// Canonical JS tokenizer for the iSamples search substrate (#169 §2, #170). +// +// MUST stay in lockstep with the Python twin `tools/search_tokenizer.py`. +// Both run against `tests/search_tokenizer_regression.json` in CI; any +// divergence is a hard failure (SEARCH_INDEX_V1.md §2). +// +// Pipeline (order matters, part of the contract): +// 1. NFKC normalize 2. lowercase 3. diacritic strip (NFD, drop \p{Mn}, +// NFC) 4. non-alphanumeric -> space (Unicode-aware: \p{L}\p{N} survive, +// so `Iron-Age` -> `iron age`, `IGSN:HRV000ABC` -> `igsn hrv000abc`) +// 5. whitespace split 6. length filter 1..64. +// +// Parity note: step 4 replaces every non-alphanumeric character (including +// all exotic whitespace) with a plain space, so step 5's split semantics +// cannot diverge between Python str.split() and this implementation. +// +// No stemming; no stopword removal here (query-time policy, §3). + +export const MAX_TOKEN_LEN = 64; + +// Length is counted in Unicode CODE POINTS ([...t].length), not UTF-16 +// units (t.length) — Python's len(str) counts code points, so astral-plane +// tokens (e.g. Deseret 𐐀, 2 UTF-16 units each) would otherwise pass the +// filter in Python and fail it here (Codex review of #329, verified). +const codePointLen = (t) => [...t].length; + +export function tokenize(text) { + if (!text) return []; + let s = String(text).normalize('NFKC'); // 1 + s = s.toLowerCase(); // 2 + s = s.normalize('NFD').replace(/\p{Mn}/gu, '').normalize('NFC'); // 3 + s = s.replace(/[^\p{L}\p{N}]/gu, ' '); // 4 + return s.split(' ') // 5 + .filter(t => { // 6 + const n = codePointLen(t); + return n >= 1 && n <= MAX_TOKEN_LEN; + }); +} diff --git a/tests/search_tokenizer_regression.json b/tests/search_tokenizer_regression.json new file mode 100644 index 0000000..59dd37e --- /dev/null +++ b/tests/search_tokenizer_regression.json @@ -0,0 +1,310 @@ +[ + { + "input": "Çatalhöyük", + "expected_tokens": [ + "catalhoyuk" + ] + }, + { + "input": "Köln", + "expected_tokens": [ + "koln" + ] + }, + { + "input": "São Paulo", + "expected_tokens": [ + "sao", + "paulo" + ] + }, + { + "input": "ÇATALHÖYÜK EXCAVATION", + "expected_tokens": [ + "catalhoyuk", + "excavation" + ] + }, + { + "input": "MaterialSampleRecord", + "expected_tokens": [ + "materialsamplerecord" + ] + }, + { + "input": "iSamples", + "expected_tokens": [ + "isamples" + ] + }, + { + "input": "Iron-Age", + "expected_tokens": [ + "iron", + "age" + ] + }, + { + "input": "co-located", + "expected_tokens": [ + "co", + "located" + ] + }, + { + "input": "Poggio Civitate (Murlo)", + "expected_tokens": [ + "poggio", + "civitate", + "murlo" + ] + }, + { + "input": "Axial Seamount, North Pacific Ocean", + "expected_tokens": [ + "axial", + "seamount", + "north", + "pacific", + "ocean" + ] + }, + { + "input": "Tell es-Sultan/Jericho", + "expected_tokens": [ + "tell", + "es", + "sultan", + "jericho" + ] + }, + { + "input": "Petra — the Treasury", + "expected_tokens": [ + "petra", + "the", + "treasury" + ] + }, + { + "input": "IGSN:HRV000ABC", + "expected_tokens": [ + "igsn", + "hrv000abc" + ] + }, + { + "input": "IGSN:321000001", + "expected_tokens": [ + "igsn", + "321000001" + ] + }, + { + "input": "ark:/28722/r2p24/vdm_19600211", + "expected_tokens": [ + "ark", + "28722", + "r2p24", + "vdm", + "19600211" + ] + }, + { + "input": "1965", + "expected_tokens": [ + "1965" + ] + }, + { + "input": "2.5kg", + "expected_tokens": [ + "2", + "5kg" + ] + }, + { + "input": "40.7128, -74.0060", + "expected_tokens": [ + "40", + "7128", + "74", + "0060" + ] + }, + { + "input": "40°N 74°W", + "expected_tokens": [ + "40", + "n", + "74", + "w" + ] + }, + { + "input": "", + "expected_tokens": [] + }, + { + "input": " ", + "expected_tokens": [] + }, + { + "input": "\t\n", + "expected_tokens": [] + }, + { + "input": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "expected_tokens": [ + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + ] + }, + { + "input": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "expected_tokens": [] + }, + { + "input": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc short", + "expected_tokens": [ + "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc", + "short" + ] + }, + { + "input": "pottery", + "expected_tokens": [ + "pottery" + ] + }, + { + "input": "ceramic", + "expected_tokens": [ + "ceramic" + ] + }, + { + "input": "bone", + "expected_tokens": [ + "bone" + ] + }, + { + "input": "mammal", + "expected_tokens": [ + "mammal" + ] + }, + { + "input": "marine", + "expected_tokens": [ + "marine" + ] + }, + { + "input": "basalt", + "expected_tokens": [ + "basalt" + ] + }, + { + "input": "ＦＵＬＬＷＩＤＴＨ ｔｅｘｔ", + "expected_tokens": [ + "fullwidth", + "text" + ] + }, + { + "input": "fine floral", + "expected_tokens": [ + "fine", + "floral" + ] + }, + { + "input": "½ liter", + "expected_tokens": [ + "1", + "2", + "liter" + ] + }, + { + "input": "the site's 'best' find", + "expected_tokens": [ + "the", + "site", + "s", + "best", + "find" + ] + }, + { + "input": "O'Brien", + "expected_tokens": [ + "o", + "brien" + ] + }, + { + "input": "陶器 сосуд κεραμικός", + "expected_tokens": [ + "陶器", + "сосуд", + "κεραμικος" + ] + }, + { + "input": "sample_id=ABC&type=rock", + "expected_tokens": [ + "sample", + "id", + "abc", + "type", + "rock" + ] + }, + { + "input": "http://w3id.org/isample/material/1.0/pottery", + "expected_tokens": [ + "http", + "w3id", + "org", + "isample", + "material", + "1", + "0", + "pottery" + ] + }, + { + "input": "𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀", + "expected_tokens": [ + "𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨" + ] + }, + { + "input": "𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀", + "expected_tokens": [] + }, + { + "input": "𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀𐐀", + "expected_tokens": [ + "𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨𐐨" + ] + }, + { + "input": "Deseret 𐐐𐐲𐑌𐐼𐑉𐐲𐐼 text", + "expected_tokens": [ + "deseret", + "𐐸𐐲𐑌𐐼𐑉𐐲𐐼", + "text" + ] + }, + { + "input": "𝔉𝔞𝔨𝔢 𝔟𝔬𝔩𝔡", + "expected_tokens": [ + "fake", + "bold" + ] + } +] \ No newline at end of file diff --git a/tests/test_search_index_builder.py b/tests/test_search_index_builder.py new file mode 100644 index 0000000..a13ee30 --- /dev/null +++ b/tests/test_search_index_builder.py @@ -0,0 +1,346 @@ +"""End-to-end fixture test for tools/build_search_index.py (#170 §4). + +Scope: URI dereferencing + document projection + shard structure — the +things the tokenizer tests deliberately do NOT prove. Builds a tiny corpus +(10 docs) with a fixture vocab_labels table, runs the real builder, and +asserts against the produced substrate. +""" + +import json +import subprocess +import sys +from pathlib import Path + +import duckdb +import pytest + +REPO = Path(__file__).resolve().parent.parent +BUILDER = REPO / "tools" / "build_search_index.py" +sys.path.insert(0, str(REPO / "tools")) +from build_search_index import fnv1a32 # noqa: E402 + +SHARDS = 8 # small fixture -> few shards keeps file count readable + +# 10 docs. Materials: 2×Pottery, 1×Ceramic, 1×Bone, 1×unresolvable URI, +# rest no material; one doc carries a KEYWORD concept whose label exists only +# on the IdentifiedConcept row (no vocab entry) — the ic.label fallback path +# that makes OpenContext keyword concepts ('Pottery' etc.) searchable. +# (Issue spec: >=3 docs with resolvable URIs, >=1 without.) +DOCS = [ + # pid, label, description, material_uri, keyword_uri, place_names + ("pid:001", "Red Pottery Bowl", None, "test://Pottery", None, ["Murlo, Italy"]), + ("pid:002", "Pottery sherd", "Iron-Age context", "test://Pottery", None, []), + ("pid:003", "Ceramic figurine", None, "test://Ceramic", None, ["Çatalhöyük"]), + ("pid:004", "Worked bone awl", "carved bone tool", "test://Bone", None, []), + ("pid:005", "Mystery lump", None, "test://weird/UnknownStuff", None, []), + ("pid:006", "Basalt core", "columnar basalt", None, "test://kw/VolcanicRock", ["Axial Seamount summit caldera"]), + ("pid:007", "Soil sample", None, None, None, []), + ("pid:008", "Water sample", "filtered seawater", None, None, ["North Pacific Ocean"]), + ("pid:009", "Coral fragment", None, None, None, []), + ("pid:010", "Obsidian flake", None, None, None, []), +] +# Keyword concept has an IC label but NO vocab entry (the OC-keyword shape). +IC_LABELS = {"test://kw/VolcanicRock": "Volcanic rock"} +VOCAB = [ + ("test://Pottery", "Pottery", "en"), + ("test://Ceramic", "Ceramic", "en"), + ("test://Bone", "Bone", "en"), + # deliberately NO entry for test://weird/UnknownStuff or test://kw/* +] + + +@pytest.fixture(scope="module") +def built_index(tmp_path_factory): + tmp = tmp_path_factory.mktemp("fts_fixture") + con = duckdb.connect() + + # Concept rows get row_ids 100+; samples reference them by row_id array. + uris = sorted({d[3] for d in DOCS if d[3]} | {d[4] for d in DOCS if d[4]}) + uri_rowid = {u: 100 + i for i, u in enumerate(uris)} + con.execute(""" + CREATE TABLE wide ( + row_id BIGINT, pid VARCHAR, otype VARCHAR, label VARCHAR, + description VARCHAR, + p__has_material_category BIGINT[], + p__has_context_category BIGINT[], + p__has_sample_object_type BIGINT[], + p__keywords BIGINT[] + )""") + for i, (pid, label, desc, mat, kw, _places) in enumerate(DOCS): + con.execute( + "INSERT INTO wide VALUES (?, ?, 'MaterialSampleRecord', ?, ?, ?, NULL, NULL, ?)", + [i, pid, label, desc, + [uri_rowid[mat]] if mat else None, + [uri_rowid[kw]] if kw else None]) + for uri, rid in uri_rowid.items(): + con.execute( + "INSERT INTO wide VALUES (?, ?, 'IdentifiedConcept', ?, NULL, NULL, NULL, NULL, NULL)", + [rid, uri, IC_LABELS.get(uri)]) + + con.execute("CREATE TABLE lite (pid VARCHAR, place_name VARCHAR[])") + for pid, _l, _d, _m, _k, places in DOCS: + con.execute("INSERT INTO lite VALUES (?, ?)", [pid, places or None]) + + con.execute("CREATE TABLE vocab (uri VARCHAR, pref_label VARCHAR, lang VARCHAR)") + for row in VOCAB: + con.execute("INSERT INTO vocab VALUES (?, ?, ?)", list(row)) + + wide_p, lite_p, vocab_p = (str(tmp / f"{n}.parquet") for n in ("wide", "lite", "vocab")) + con.execute(f"COPY wide TO '{wide_p}' (FORMAT PARQUET)") + con.execute(f"COPY lite TO '{lite_p}' (FORMAT PARQUET)") + con.execute(f"COPY vocab TO '{vocab_p}' (FORMAT PARQUET)") + + out = tmp / "out" + res = subprocess.run( + [sys.executable, str(BUILDER), + "--wide", wide_p, "--lite", lite_p, "--vocab", vocab_p, + "--outdir", str(out), "--tag", "test_202601", "--shards", str(SHARDS)], + capture_output=True, text=True) + assert res.returncode == 0, res.stderr + root = out / "test_202601_search_index_v1" + q = duckdb.connect() + rows = q.sql(f"SELECT * FROM read_parquet('{root}/shard_*.parquet')").df() + return root, rows + + +def test_outputs_exist(built_index): + root, _ = built_index + assert (root / "df.parquet").exists() + assert (root / "build_stats.json").exists() + assert list(root.glob("shard_*.parquet")) + # shard_sizes.json: every base shard listed with its true file size + sizes = json.loads((root / "shard_sizes.json").read_text()) + assert len(sizes) == SHARDS + for name, size in sizes.items(): + assert (root / name).stat().st_size == size + + +def test_uri_dereferencing_end_to_end(built_index): + """THE core proof: searching 'pottery' finds exactly the pids whose + material URI is — via dereferenced concept labels.""" + _, rows = built_index + hits = rows[(rows.token == "pottery") & (rows.field == "concept.label")] + assert sorted(hits.pid) == ["pid:001", "pid:002"] + + +def test_uri_tail_fallback_and_counter(built_index): + root, rows = built_index + # test://weird/UnknownStuff has no prefLabel -> URI tail 'unknownstuff'. + hits = rows[(rows.token == "unknownstuff") & (rows.field == "concept.label")] + assert list(hits.pid) == ["pid:005"] + stats = json.loads((root / "build_stats.json").read_text()) + assert stats["concept_label_missing_pref_label"] == 1 + assert stats["concept_label_uri_resolution"]["material_missing_pref"] > 0 + + +def test_keyword_concept_via_ic_label_fallback(built_index): + """A keyword concept with no vocab entry resolves through the concept's + own label (contract amendment 2026-07-10) — the path that makes + 'pottery Cyprus' work against OpenContext keyword concepts.""" + _, rows = built_index + hits = rows[(rows.token == "volcanic") & (rows.field == "concept.label")] + assert list(hits.pid) == ["pid:006"] + hits2 = rows[(rows.token == "rock") & (rows.field == "concept.label")] + assert list(hits2.pid) == ["pid:006"] + + +def test_field_projection_and_doc_len(built_index): + _, rows = built_index + # pid:001 label 'Red Pottery Bowl' -> 3 tokens, tf('pottery')=1, doc_len=3 + r = rows[(rows.pid == "pid:001") & (rows.field == "sample.label")] + assert sorted(r.token) == ["bowl", "pottery", "red"] + assert set(r.doc_len) == {3} + assert set(r.tf) == {1} + # place_name projection, diacritics folded + r = rows[(rows.pid == "pid:003") & (rows.field == "sample.place_name")] + assert list(r.token) == ["catalhoyuk"] + # description projection + r = rows[(rows.pid == "pid:002") & (rows.field == "sample.description")] + assert sorted(r.token) == ["age", "context", "iron"] + + +def test_shard_assignment_matches_fnv1a(built_index): + root, rows = built_index + for token in ("pottery", "basalt", "catalhoyuk"): + expected_shard = fnv1a32(token) % SHARDS + for f in root.glob("shard_*.parquet"): + got = duckdb.sql( + f"SELECT count(*) FROM read_parquet('{f}') WHERE token = '{token}'" + ).fetchone()[0] + shard_no = int(f.stem.split("_")[1]) + if shard_no == expected_shard: + assert got > 0, f"{token} missing from its home shard {f.name}" + else: + assert got == 0, f"{token} leaked into {f.name}" + + +def test_df_embedded_in_shards(built_index): + """Round-5: the query path never fetches df.parquet — df ships as a + column in every shard row and must equal the sidecar's value.""" + root, rows = built_index + assert "df" in rows.columns + sidecar = duckdb.sql(f"SELECT * FROM read_parquet('{root}/df.parquet')").df() + merged = rows.merge(sidecar, on="token", suffixes=("_row", "_sidecar")) + assert (merged.df_row == merged.df_sidecar).all() + # BM25 constants ship in build_stats.json + stats = json.loads((root / "build_stats.json").read_text()) + assert stats["total_documents"] == len(rows.groupby(["pid", "field"])) + + +def test_df_sidecar(built_index): + root, rows = built_index + df = duckdb.sql(f"SELECT * FROM read_parquet('{root}/df.parquet')").df() + # 'pottery' appears in: pid:001 label, pid:002 label, pid:001+002 + # concept.label -> 4 (pid, field) docs. + assert int(df[df.token == "pottery"].df.iloc[0]) == 4 + # DF must equal the distinct (pid, field) count for every token. + joined = rows.groupby("token").size() + for token, n in joined.items(): + assert int(df[df.token == token].df.iloc[0]) == n + + +def test_build_stats_field_coverage(built_index): + root, _ = built_index + stats = json.loads((root / "build_stats.json").read_text()) + assert stats["total_samples"] == 10 + assert stats["fields"]["sample.label"]["samples_with_field"] == 10 + assert stats["fields"]["sample.description"]["samples_with_field"] == 4 + assert stats["fields"]["sample.place_name"]["samples_with_field"] == 4 + assert stats["fields"]["concept.label"]["samples_with_field"] == 6 # 5 material + 1 keyword + assert stats["concept_label_uri_resolution"]["keywords_resolved"] == 1.0 + assert stats["shard_hash"] == "fnv1a32(utf8(token)) % shards" + + +def test_hot_token_isolation_on_tiny_cap(tmp_path): + """Force the byte cap to ~0 so every token is 'hot'; assert the token-level + isolation semantics: hot tokens leave the base shards, land in + hot/_p*.parquet, are listed in hot_tokens.json, and no rows are + lost. (At production scale only vocabulary-boilerplate tokens like + 'material' trip this — with posting lists on ~5M samples each.)""" + con = duckdb.connect() + con.execute("CREATE TABLE wide (row_id BIGINT, pid VARCHAR, otype VARCHAR," + " label VARCHAR, description VARCHAR," + " p__has_material_category BIGINT[]," + " p__has_context_category BIGINT[]," + " p__has_sample_object_type BIGINT[]," + " p__keywords BIGINT[])") + # pid:x carries 'pottery' in BOTH label and concept.label (multi-field); + # pid:t1/pid:t2 are identical twins (deterministic tie case). Codex + # round-3: the topk sidecar must rank per-PID SUMMED scores, one row per + # (token, pid), ties broken by pid ascending. + con.execute("INSERT INTO wide VALUES (0, 'pid:x', 'MaterialSampleRecord'," + " 'pottery pottery pottery', NULL, [100], NULL, NULL, NULL)") + con.execute("INSERT INTO wide VALUES (1, 'pid:y', 'MaterialSampleRecord'," + " 'pottery basalt', NULL, NULL, NULL, NULL, NULL)") + con.execute("INSERT INTO wide VALUES (2, 'pid:t1', 'MaterialSampleRecord'," + " 'twin stone', NULL, NULL, NULL, NULL, NULL)") + con.execute("INSERT INTO wide VALUES (3, 'pid:t2', 'MaterialSampleRecord'," + " 'twin stone', NULL, NULL, NULL, NULL, NULL)") + con.execute("INSERT INTO wide VALUES (100, 'test://P', 'IdentifiedConcept'," + " NULL, NULL, NULL, NULL, NULL, NULL)") + con.execute("CREATE TABLE lite (pid VARCHAR, place_name VARCHAR[])") + con.execute("CREATE TABLE vocab (uri VARCHAR, pref_label VARCHAR, lang VARCHAR)") + con.execute("INSERT INTO vocab VALUES ('test://P', 'pottery', 'en')") + wide_p, lite_p, vocab_p = (str(tmp_path / f"{n}.parquet") for n in ("w", "l", "v")) + con.execute(f"COPY wide TO '{wide_p}' (FORMAT PARQUET)") + con.execute(f"COPY lite TO '{lite_p}' (FORMAT PARQUET)") + con.execute(f"COPY vocab TO '{vocab_p}' (FORMAT PARQUET)") + res = subprocess.run( + [sys.executable, str(BUILDER), "--wide", wide_p, "--lite", lite_p, + "--vocab", vocab_p, "--outdir", str(tmp_path / "out"), + "--tag", "cap_test", "--shards", "2", "--shard-cap-mb", "0.000001"], + capture_output=True, text=True) + assert res.returncode == 0, res.stderr + root = tmp_path / "out" / "cap_test_search_index_v1" + + manifest_full = json.loads((root / "hot_tokens.json").read_text()) + manifest = manifest_full + assert "pottery" in manifest["tokens"] + # two-tier policy fields present; at a ~1-byte cap nothing is fetchable + e = manifest["tokens"]["pottery"] + assert e["total_bytes"] > 0 and e["fetchable"] is False + key = manifest["tokens"]["pottery"]["key"] + assert key == f"{fnv1a32('pottery'):08x}" + subs = list((root / "hot").glob(f"{key}_p*.parquet")) + assert len(subs) == manifest["tokens"]["pottery"]["sub_files"] >= 2 + # hot token's rows all present across its sub-files (both pids, no dupes) + hot_rows = duckdb.sql( + f"SELECT pid, field, tf FROM read_parquet('{root}/hot/{key}_p*.parquet') ORDER BY pid, field" + ).fetchall() + # pid:x matches in TWO fields (label tf=3 + concept.label tf=1) + assert hot_rows == [("pid:x", "concept.label", 1), ("pid:x", "sample.label", 3), + ("pid:y", "sample.label", 1)] + # ...and absent from every base shard + base = duckdb.sql( + f"SELECT count(*) FROM read_parquet('{root}/shard_*.parquet') WHERE token='pottery'" + ).fetchone()[0] + assert base == 0 + # hot_topk sidecar (contract §6 common-term rule; Codex round 3): + # ONE row per (token, pid) — per-PID summed field-weighted scores. + assert manifest_full["topk_k"] == 500 + topk = duckdb.sql( + f"SELECT token, pid, static_score, rank FROM read_parquet('{root}/hot_topk.parquet') " + "WHERE token='pottery' ORDER BY rank").fetchall() + assert [r[3] for r in topk] == list(range(1, len(topk) + 1)) + # pid:x appears ONCE despite matching in two fields (label + concept) + assert [r[1] for r in topk] == ["pid:x", "pid:y"] + # multi-field sum: pid:x (label tf=3 + concept.label) beats pid:y (label tf=1) + assert topk[0][2] > topk[1][2] + # deterministic tie: identical twins order by pid ascending + twins = duckdb.sql( + f"SELECT pid, static_score, rank FROM read_parquet('{root}/hot_topk.parquet') " + "WHERE token='twin' ORDER BY rank").fetchall() + assert [t[0] for t in twins] == ["pid:t1", "pid:t2"] + assert twins[0][1] == twins[1][1] + + +def test_hot_key_collision_gets_distinct_files(tmp_path): + """fnv1a32 is 32-bit: distinct tokens CAN collide (this pair was + produced by Codex review and verified: both hash to 0xa7c9bf62). + Colliding hot tokens must get DISTINCT manifest keys and files — + silent overwrite would merge two tokens' postings.""" + a, b = "tywtopf1ri", "32jnqttihd" + assert fnv1a32(a) == fnv1a32(b) # the premise + con = duckdb.connect() + con.execute("CREATE TABLE wide (row_id BIGINT, pid VARCHAR, otype VARCHAR," + " label VARCHAR, description VARCHAR," + " p__has_material_category BIGINT[]," + " p__has_context_category BIGINT[]," + " p__has_sample_object_type BIGINT[]," + " p__keywords BIGINT[])") + con.execute(f"INSERT INTO wide VALUES (0,'pid:a','MaterialSampleRecord','{a}',NULL,NULL,NULL,NULL,NULL)") + con.execute(f"INSERT INTO wide VALUES (1,'pid:b','MaterialSampleRecord','{b}',NULL,NULL,NULL,NULL,NULL)") + con.execute("CREATE TABLE lite (pid VARCHAR, place_name VARCHAR[])") + con.execute("CREATE TABLE vocab (uri VARCHAR, pref_label VARCHAR, lang VARCHAR)") + wide_p, lite_p, vocab_p = (str(tmp_path / f"{n}.parquet") for n in ("w", "l", "v")) + con.execute(f"COPY wide TO '{wide_p}' (FORMAT PARQUET)") + con.execute(f"COPY lite TO '{lite_p}' (FORMAT PARQUET)") + con.execute(f"COPY vocab TO '{vocab_p}' (FORMAT PARQUET)") + res = subprocess.run( + [sys.executable, str(BUILDER), "--wide", wide_p, "--lite", lite_p, + "--vocab", vocab_p, "--outdir", str(tmp_path / "out"), + "--tag", "coll_test", "--shards", "2", "--shard-cap-mb", "0.000001"], + capture_output=True, text=True) + assert res.returncode == 0, res.stderr + root = tmp_path / "out" / "coll_test_search_index_v1" + manifest = json.loads((root / "hot_tokens.json").read_text())["tokens"] + assert a in manifest and b in manifest + assert manifest[a]["key"] != manifest[b]["key"] + # each token's rows are exactly its own pid — no cross-contamination + for tok, pid in ((a, "pid:a"), (b, "pid:b")): + key = manifest[tok]["key"] + rows = duckdb.sql( + f"SELECT DISTINCT token, pid FROM read_parquet('{root}/hot/{key}_p*.parquet')" + ).fetchall() + assert rows == [(tok, pid)], rows + + +def test_no_hot_tokens_at_default_cap(built_index): + """Fixture-scale corpus: nothing is hot at the 5 MB default; base shards + carry everything and hot_tokens.json is an empty manifest.""" + root, rows = built_index + manifest = json.loads((root / "hot_tokens.json").read_text()) + assert manifest["tokens"] == {} + stats = json.loads((root / "build_stats.json").read_text()) + assert stats["hot_tokens"] == 0 + assert stats["shard_cap_violations"] == 0 diff --git a/tests/test_search_tokenizer.py b/tests/test_search_tokenizer.py new file mode 100644 index 0000000..0d36a65 --- /dev/null +++ b/tests/test_search_tokenizer.py @@ -0,0 +1,42 @@ +"""Python tokenizer vs the shared regression set (#170). + +Scope: tokenizer ONLY (SEARCH_INDEX_V1.md §2). URI dereferencing is proved +separately in tests/test_search_index_builder.py. The JS twin runs the same +JSON in tests/unit/search-tokenizer.test.mjs; CI failing either one is the +Python↔JS parity gate. +""" + +import json +import sys +from pathlib import Path + +import pytest + +REPO = Path(__file__).resolve().parent.parent +sys.path.insert(0, str(REPO / "tools")) + +from search_tokenizer import tokenize # noqa: E402 + +REGRESSION = json.loads((REPO / "tests" / "search_tokenizer_regression.json").read_text()) + + +@pytest.mark.parametrize( + "entry", REGRESSION, ids=[repr(e["input"])[:40] for e in REGRESSION] +) +def test_regression_entry(entry): + assert tokenize(entry["input"]) == entry["expected_tokens"] + + +def test_regression_set_is_big_enough(): + # Contract: >= 30 strings (#170 §3). + assert len(REGRESSION) >= 30 + + +def test_none_and_empty(): + assert tokenize(None) == [] + assert tokenize("") == [] + + +def test_length_filter_boundaries(): + assert tokenize("x" * 64) == ["x" * 64] + assert tokenize("x" * 65) == [] diff --git a/tests/unit/search-tokenizer.test.mjs b/tests/unit/search-tokenizer.test.mjs new file mode 100644 index 0000000..72f114d --- /dev/null +++ b/tests/unit/search-tokenizer.test.mjs @@ -0,0 +1,36 @@ +// JS tokenizer vs the shared regression set (#170) — the Python↔JS parity +// gate's JS half. The regression JSON's expected_tokens are generated from +// the Python implementation (tools/search_tokenizer.py); this suite passing +// means the two implementations agree on every entry. +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { readFileSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; +import { dirname, join } from 'node:path'; + +import { tokenize, MAX_TOKEN_LEN } from '../../assets/js/search_tokenizer.js'; + +const here = dirname(fileURLToPath(import.meta.url)); +const REGRESSION = JSON.parse( + readFileSync(join(here, '..', 'search_tokenizer_regression.json'), 'utf8')); + +test('regression set is big enough (contract: >= 30)', () => { + assert.ok(REGRESSION.length >= 30, `only ${REGRESSION.length} entries`); +}); + +for (const entry of REGRESSION) { + test(`tokenize ${JSON.stringify(entry.input).slice(0, 40)}`, () => { + assert.deepEqual(tokenize(entry.input), entry.expected_tokens); + }); +} + +test('null/undefined/empty → []', () => { + assert.deepEqual(tokenize(null), []); + assert.deepEqual(tokenize(undefined), []); + assert.deepEqual(tokenize(''), []); +}); + +test('length filter boundaries', () => { + assert.deepEqual(tokenize('x'.repeat(MAX_TOKEN_LEN)), ['x'.repeat(MAX_TOKEN_LEN)]); + assert.deepEqual(tokenize('x'.repeat(MAX_TOKEN_LEN + 1)), []); +}); diff --git a/tools/build_search_index.py b/tools/build_search_index.py new file mode 100644 index 0000000..7f5c192 --- /dev/null +++ b/tools/build_search_index.py @@ -0,0 +1,630 @@ +#!/usr/bin/env python3 +"""Offline builder for the iSamples search substrate v1 (#170). + +Implements SEARCH_INDEX_V1.md: builds the sample-centric document +projection (label / description / place_name / dereferenced concept +labels), tokenizes with the canonical tokenizer (tools/search_tokenizer.py +— kept in JS parity by CI), and emits: + + /_search_index_v1/ + shard_000.parquet .. shard_NNN.parquet token-row substrate (§4) + df.parquet global token DF sidecar + build_stats.json empirical coverage (§10) + +Token-row schema (§4): token VARCHAR, pid VARCHAR, field VARCHAR, +tf USMALLINT, doc_len USMALLINT. + +Sharding (§6): FNV-1a 32-bit over the UTF-8 token, mod --shards. +FNV-1a is deliberate: the browser query path (#171) must compute the +same shard for a query token in JS, so the hash has to be trivially +portable (DuckDB's hash() is not). A shard whose file exceeds the byte +cap is sub-sharded by FNV-1a(pid) % M into shard_XXX_pY.parquet — a +query must then read all sub-files of its shard (documented for #171). + +Memory model: DuckDB aggregates text fragments per (pid, field) and +streams them in Arrow batches; Python tokenizes and counts; token rows +spill to intermediate parquets every ~2M rows; DuckDB then does the +global DF + per-shard ordered writes. The PYTHON side is bounded (batch ++ 2M-row buffer); DuckDB's aggregation/sort working set still scales +with the corpus (fine at 6.7M samples on a dev machine; not a hard +guarantee). + +Usage: + python tools/build_search_index.py \ + --wide ~/Data/.../isamples_202608_wide.parquet \ + --lite https://data.isamples.org/isamples_202608_samples_map_lite_v3.parquet \ + --vocab https://data.isamples.org/vocab_labels_202608.parquet \ + --outdir /tmp/search_index --tag isamples_202608 +""" + +from __future__ import annotations + +import argparse +import json +import os +import sys +import tempfile +import time +from collections import Counter +from pathlib import Path + +import duckdb +import pyarrow as pa +import pyarrow.parquet as pq + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +from search_tokenizer import tokenize # noqa: E402 + +V1_FIELDS = ("sample.label", "sample.description", "sample.place_name", "concept.label") +# CONTRACT AMENDMENT (2026-07-10, discovered on the first full-corpus build): +# `keywords` is pulled forward from the contract's v2 list, and label +# resolution falls back to the concept's own `label` column in the wide +# before the URI tail. Without both, the v1 index REGRESSES the benchmark's +# own example query: 'pottery Cyprus' → 0 results, because "Pottery" is not +# a concept in the 537-entry curated vocabulary — it reaches samples only as +# an OpenContext keyword concept whose label lives on the IdentifiedConcept +# row. The interim ILIKE search already covers keywords (via +# build_frontend_derived.py's appended concept_labels), so shipping v1 +# without them would be a recall regression and an automatic #172 NO-GO. +CONCEPT_DIMS = { + "material": "p__has_material_category", + "context": "p__has_context_category", + "object_type": "p__has_sample_object_type", + "keywords": "p__keywords", +} +TOKEN_ROW_SCHEMA = pa.schema([ + ("token", pa.string()), + ("pid", pa.string()), + ("field", pa.string()), + ("tf", pa.uint16()), + ("doc_len", pa.uint16()), + ("shard", pa.uint16()), +]) + + +def fnv1a32(data: str) -> int: + """FNV-1a 32-bit over UTF-8 bytes. JS twin must match exactly (#171).""" + h = 0x811C9DC5 + for byte in data.encode("utf-8"): + h ^= byte + h = (h * 0x01000193) & 0xFFFFFFFF + return h + + +def uri_tail(uri: str) -> str: + """Fallback fragment for a concept URI with no prefLabel: its last path + segment (e.g. <…/material/1.0/pottery> -> 'pottery').""" + return uri.rstrip("/").rsplit("/", 1)[-1] if uri else "" + + +def fragment_relation(con: duckdb.DuckDBPyConnection, wide: str, lite: str, vocab: str) -> None: + """Materialize `fragments(pid, field, text, resolved)` — the document + projection of SEARCH_INDEX_V1.md §1 — plus per-dim URI resolution stats. + + `resolved` is only meaningful for concept.label rows: TRUE when the URI + dereferenced to a SKOS prefLabel, FALSE when we fell back to the URI + tail (build-stat counter feeds off it). + """ + # Decorrelated unnest+join, mirroring scripts/build_frontend_derived.py's + # `mat` CTE — a correlated `row_id IN (SELECT unnest(...))` subquery blows + # up the planner at 20M-row scale (documented there). + concept_selects = [] + for dim, col in CONCEPT_DIMS.items(): + concept_selects.append(f""" + SELECT ex.pid, + 'concept.label' AS field, + COALESCE(v.pref_label, ic.label) AS pref_label, + ic.uri, + '{dim}' AS dim + FROM ( + SELECT s.pid, u.rid + FROM wide_samples s, UNNEST(s.{col}) AS u(rid) + ) ex + JOIN ic ON ic.row_id = ex.rid + LEFT JOIN vocab v ON v.uri = ic.uri + """) + concepts_union = " UNION ALL ".join(concept_selects) + con.execute(f""" + CREATE TEMP TABLE wide_samples AS + SELECT pid, label, description, + {', '.join(CONCEPT_DIMS.values())} + FROM read_parquet('{wide}') + WHERE otype = 'MaterialSampleRecord' AND pid IS NOT NULL; + + CREATE TEMP TABLE ic AS + SELECT row_id, pid AS uri, label FROM read_parquet('{wide}') + WHERE otype = 'IdentifiedConcept'; + + CREATE TEMP VIEW vocab AS + SELECT uri, pref_label FROM read_parquet('{vocab}') + WHERE lang = 'en' OR lang IS NULL; + + CREATE TEMP TABLE concept_fragments AS + SELECT pid, field, pref_label, uri, dim, + (pref_label IS NOT NULL) AS resolved + FROM ({concepts_union}); + + CREATE TEMP TABLE fragments AS + SELECT pid, 'sample.label' AS field, label AS text, TRUE AS resolved + FROM wide_samples WHERE label IS NOT NULL AND label != '' + UNION ALL + SELECT pid, 'sample.description', description, TRUE + FROM wide_samples WHERE description IS NOT NULL AND description != '' + UNION ALL + SELECT l.pid, 'sample.place_name', t.place, TRUE + FROM read_parquet('{lite}') l, UNNEST(l.place_name) AS t(place) + WHERE t.place IS NOT NULL AND t.place != '' + UNION ALL + SELECT pid, field, + CASE WHEN resolved THEN pref_label ELSE uri END AS text, + resolved + FROM concept_fragments; + """) + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--wide", required=True) + ap.add_argument("--lite", required=True) + ap.add_argument("--vocab", required=True) + ap.add_argument("--outdir", required=True) + ap.add_argument("--tag", required=True, help="e.g. isamples_202608") + # 256 default: the 202608 corpus yields ~570 MB of non-hot base rows; + # 64 shards made ~9 MB base files against the 5 MB cap. 256 → ~2.2 MB + # average with headroom for growth (v1.5 event/site fields). + ap.add_argument("--shards", type=int, default=256) + ap.add_argument("--shard-cap-mb", type=float, default=5.0) + ap.add_argument("--batch-rows", type=int, default=200_000) + args = ap.parse_args() + + t0 = time.time() + out_root = Path(args.outdir) / f"{args.tag}_search_index_v1" + out_root.mkdir(parents=True, exist_ok=True) + + con = duckdb.connect() + fragment_relation(con, args.wide, args.lite, args.vocab) + + total_samples = con.sql( + "SELECT count(DISTINCT pid) FROM wide_samples").fetchone()[0] + # URI-resolution stats per dim (contract §10) — computed in SQL, cheap. + resolution = {} + for dim in CONCEPT_DIMS: + row = con.sql(f""" + SELECT count(*) FILTER (resolved), count(*) + FROM concept_fragments WHERE dim = '{dim}' + """).fetchone() + total = row[1] or 1 + resolution[f"{dim}_resolved"] = round(row[0] / total, 4) + resolution[f"{dim}_missing_pref"] = round((total - row[0]) / total, 4) + missing_pref_count = con.sql( + "SELECT count(*) FROM concept_fragments WHERE NOT resolved").fetchone()[0] + + # For concept fragments that missed a prefLabel, tokenize the URI TAIL, + # not the full URI (contract: URI-tail fallback). Handled here by + # rewriting those texts before aggregation. + con.execute(""" + UPDATE fragments + SET text = regexp_extract(rtrim(text, '/'), '([^/]+)$', 1) + WHERE field = 'concept.label' AND NOT resolved; + """) + + # Aggregate fragments per (pid, field) so each streamed row carries the + # COMPLETE document for that pair (tf/doc_len computable in one visit). + docs = con.sql(""" + SELECT pid, field, list(text) AS texts + FROM fragments + GROUP BY pid, field + """) + + field_stats = {f: {"samples_with_field": 0, "total_tokens": 0} for f in V1_FIELDS} + # USMALLINT truncation is a contract-semantics change when it fires, so + # it is COUNTED, not silent (Codex review of #329): build_stats records + # both counters + observed maxima; a non-zero count prints a warning. + trunc = {"doc_len_docs": 0, "tf_rows": 0, "max_doc_len": 0, "max_tf": 0} + tmp_dir = tempfile.mkdtemp(prefix="search_index_rows_") + tmp_files: list[str] = [] + buf: dict[str, list] = {k: [] for k in ("token", "pid", "field", "tf", "doc_len", "shard")} + buf_rows = 0 + + def flush() -> None: + nonlocal buf, buf_rows + if not buf_rows: + return + table = pa.table( + { + "token": pa.array(buf["token"], pa.string()), + "pid": pa.array(buf["pid"], pa.string()), + "field": pa.array(buf["field"], pa.string()), + "tf": pa.array(buf["tf"], pa.uint16()), + "doc_len": pa.array(buf["doc_len"], pa.uint16()), + "shard": pa.array(buf["shard"], pa.uint16()), + }, + schema=TOKEN_ROW_SCHEMA, + ) + path = os.path.join(tmp_dir, f"rows_{len(tmp_files):05d}.parquet") + pq.write_table(table, path) + tmp_files.append(path) + buf = {k: [] for k in buf} + buf_rows = 0 + + reader = docs.fetch_arrow_reader(batch_size=args.batch_rows) + for batch in reader: + pids = batch.column("pid").to_pylist() + fields = batch.column("field").to_pylist() + texts_col = batch.column("texts").to_pylist() + for pid, field, texts in zip(pids, fields, texts_col): + tokens: list[str] = [] + for t in texts: + tokens.extend(tokenize(t)) + if not tokens: + continue + raw_len = len(tokens) + trunc["max_doc_len"] = max(trunc["max_doc_len"], raw_len) + if raw_len > 65535: + trunc["doc_len_docs"] += 1 + doc_len = min(raw_len, 65535) + fs = field_stats[field] + fs["samples_with_field"] += 1 + fs["total_tokens"] += raw_len + for token, tf in Counter(tokens).items(): + trunc["max_tf"] = max(trunc["max_tf"], tf) + if tf > 65535: + trunc["tf_rows"] += 1 + buf["token"].append(token) + buf["pid"].append(pid) + buf["field"].append(field) + buf["tf"].append(min(tf, 65535)) + buf["doc_len"].append(doc_len) + buf["shard"].append(fnv1a32(token) % args.shards) + buf_rows += 1 + if buf_rows >= 2_000_000: + flush() + flush() + + if not tmp_files: + print("ERROR: no token rows produced — empty inputs?", file=sys.stderr) + return 1 + + rows_glob = os.path.join(tmp_dir, "rows_*.parquet") + + # df.parquet: documents are (pid, field) pairs; rows are already unique + # per (token, pid, field), so DF is a plain count. NOTE (round 5): this + # sidecar is an OFFLINE artifact (8 MB at corpus scale, over-cap) — the + # query path never fetches it, because df is EMBEDDED as a column in + # every shipped shard/hot row below (sorted-by-token runs RLE-compress + # to ~nothing) and corpus constants ride in build_stats.json. + df_path = out_root / "df.parquet" + con.execute(f""" + COPY ( + SELECT token, count(*)::UINTEGER AS df + FROM read_parquet('{rows_glob}') + GROUP BY token ORDER BY token + ) TO '{df_path}' (FORMAT PARQUET); + """) + + # Hot-token isolation + per-shard ordered writes (§6). + # + # The contract's high-frequency rule is TOKEN-level, not shard-level: a + # token whose postings alone would blow the byte cap is pulled OUT of its + # base shard and written to its own sub-file set, sub-sharded by + # fnv1a32(pid) % M with M sized so each sub-file fits the cap. The base + # shard then stays small, so a #171 reader fetches: + # normal token -> its base shard (small), located by fnv1a32(token)%N; + # hot token -> hot/_p{0..M-1}.parquet, with + # hotness + M read from hot_tokens.json. + # (First full-corpus build: 'material'/'object'/'solid' etc. — vocabulary + # boilerplate present on ~5M samples — have posting lists >100 MB; the + # earlier shard-level split forced readers to fetch M files for EVERY + # token, breaking the §7 cold-bytes budget for the common case.) + # + # Cap is enforced against parquet FILE bytes (what a browser actually + # transfers), which is stricter than the contract's "uncompressed". + cap_bytes = int(args.shard_cap_mb * 1024 * 1024) + est = con.sql(f""" + SELECT avg(len(token) + len(pid) + len(field) + 4) FROM + (SELECT * FROM read_parquet('{rows_glob}') LIMIT 500000) + """).fetchone()[0] or 30.0 + # Planning ratio is only a first guess (observed file ratios vary by + # token entropy); every hot token's sub-files are VERIFIED against the + # cap after writing and re-split at 2×M until they fit. + compress_ratio = 0.7 + hot = con.sql(f""" + SELECT token, count(*) AS n + FROM read_parquet('{rows_glob}') + GROUP BY token + HAVING count(*) * {est} * {compress_ratio} > {cap_bytes} + """).fetchall() + hot_dir = out_root / "hot" + hot_manifest: dict[str, dict] = {} + if hot: + hot_dir.mkdir(exist_ok=True) + con.execute("CREATE TEMP TABLE hot_tokens (token VARCHAR PRIMARY KEY)") + con.executemany("INSERT INTO hot_tokens VALUES (?)", [[t] for t, _ in hot]) + shard_max_bytes = 0 + shard_files = 0 + cap_violations: list[str] = [] + _hot_keys_used: set[str] = set() + + def write_hot_token(token: str, n: int) -> None: + nonlocal shard_max_bytes, shard_files + # FNV-1a is 32-bit, so distinct hot tokens CAN collide (Codex review + # of #329 produced a real pair: 'tywtopf1ri'/'32jnqttihd' → a7c9bf62). + # Readers locate hot files via hot_tokens.json (never by recomputing + # the hash), so uniqueness only has to hold here: suffix on collision. + base_key = f"{fnv1a32(token):08x}" + key = base_key + seq = 0 + while key in _hot_keys_used: + seq += 1 + key = f"{base_key}-{seq}" + _hot_keys_used.add(key) + tok_sql = token.replace("'", "''") + m_count = max(2, -(-int(n * est * compress_ratio) // cap_bytes)) + max_attempts = 6 # 2x per retry; 6 doublings is plenty + for attempt in range(max_attempts): + paths = [] + for m in range(m_count): + path = hot_dir / f"{key}_p{m}.parquet" + con.execute(f""" + COPY ( + SELECT r.token, r.pid, r.field, r.tf, r.doc_len, d.df + FROM read_parquet('{rows_glob}') r + JOIN read_parquet('{df_path}') d USING (token) + WHERE r.token = '{tok_sql}' + AND (CAST(hash(r.pid) AS UBIGINT) % {m_count}) = {m} + ORDER BY r.pid, r.field + ) TO '{path}' (FORMAT PARQUET); + """) + paths.append(path) + biggest = max(p.stat().st_size for p in paths) + if biggest <= cap_bytes: + break + if attempt < max_attempts - 1: + for p in paths: # re-split finer and retry + p.unlink() + m_count *= 2 + else: + # Cap unreachable (e.g. per-file parquet overhead > cap). + # KEEP the finest split — manifest must always describe + # what is actually on disk — and record the violation. + cap_violations.append(f"hot/{key}: {biggest/1e6:.1f} MB after retries") + shard_max_bytes = max(shard_max_bytes, biggest) + shard_files += m_count + total_bytes = sum( + (hot_dir / f"{key}_p{m}.parquet").stat().st_size + for m in range(m_count)) + # Two-tier policy (Codex round 3): many hot tokens are hot for + # STORAGE reasons (promoted because their shard co-landed with other + # big tokens), not semantic commonness — e.g. 'island'/'genetic' at + # ~100k postings are genuinely selective and their dedicated files + # total only ~2.5 MB. If a token's whole posting set fits the cold + # budget, the reader just fetches its sub-files like a normal token; + # only tokens whose postings EXCEED the budget get the common-term + # drop/topk treatment. + hot_manifest[token] = {"key": key, "sub_files": m_count, "postings": n, + "total_bytes": total_bytes, + "fetchable": total_bytes <= cap_bytes} + + for token, n in hot: + write_hot_token(token, n) + if not hot: + con.execute("CREATE TEMP TABLE hot_tokens (token VARCHAR PRIMARY KEY)") + for shard in range(args.shards): + shard_path = out_root / f"shard_{shard:03d}.parquet" + # Write, then promote the shard's heaviest tokens to hot/ until the + # file fits the cap. Handles near-hot skew: a few ~3 MB posting lists + # co-landing in one shard can't be fixed by shard count alone. + for _attempt in range(12): + con.execute(f""" + COPY ( + SELECT r.token, r.pid, r.field, r.tf, r.doc_len, d.df + FROM read_parquet('{rows_glob}') r + JOIN read_parquet('{df_path}') d USING (token) + WHERE r.shard = {shard} + AND r.token NOT IN (SELECT token FROM hot_tokens) + ORDER BY r.token, r.pid, r.field + ) TO '{shard_path}' (FORMAT PARQUET); + """) + size = shard_path.stat().st_size + if size <= cap_bytes: + break + heaviest = con.sql(f""" + SELECT token, count(*) AS n + FROM read_parquet('{rows_glob}') + WHERE shard = {shard} + AND token NOT IN (SELECT token FROM hot_tokens) + GROUP BY token ORDER BY n DESC LIMIT 1 + """).fetchone() + if heaviest is None: + break + con.execute("INSERT INTO hot_tokens VALUES (?)", [heaviest[0]]) + write_hot_token(heaviest[0], heaviest[1]) + else: + # Attempts exhausted with the LAST promotion never re-written: + # without this final rewrite the just-promoted token would sit in + # BOTH the base file and hot/ — duplicate postings that inflate + # tf-side scores (Codex review of #329). Rewrite once more so the + # base shard reflects every promotion, then record the violation + # if it still doesn't fit. + con.execute(f""" + COPY ( + SELECT r.token, r.pid, r.field, r.tf, r.doc_len, d.df + FROM read_parquet('{rows_glob}') r + JOIN read_parquet('{df_path}') d USING (token) + WHERE r.shard = {shard} + AND r.token NOT IN (SELECT token FROM hot_tokens) + ORDER BY r.token, r.pid, r.field + ) TO '{shard_path}' (FORMAT PARQUET); + """) + size = shard_path.stat().st_size + if size > cap_bytes: + cap_violations.append(f"{shard_path.name}: {size/1e6:.1f} MB after promotions") + shard_max_bytes = max(shard_max_bytes, size) + shard_files += 1 + # hot_topk.parquet — the all-hot-query path (Codex round-2 finding: a + # hot token's full postings exceed the cold-bytes budget BY DEFINITION, + # so readers must never need them). For each hot token, precompute its + # top-K postings by STATIC single-token BM25 (IDF and doc_len norm are + # both query-independent for a single token), field-weighted per §5. + # Query policy (contract §3 amendment): queries mixing hot + selective + # terms DROP the hot terms from AND matching (common-term rule); pure- + # hot queries rank via this sidecar. K=500 keeps headroom for source/ + # facet post-filtering while staying a single small file. + # Corpus statistics per contract §5 / Codex round 3: totalDocs is the + # number of (pid, field) DOCUMENTS (rows are unique per (token,pid,field), + # so distinct (pid,field) must be counted explicitly), and doc-length + # normalization uses the PER-FIELD corpus average. Computed + # unconditionally: the browser reader needs both as BM25 constants (they + # ship in build_stats.json — round-5: the query path must not depend on + # the over-cap df.parquet sidecar). + total_docs_for_idf = con.sql(f""" + SELECT count(*) FROM ( + SELECT pid, field FROM read_parquet('{rows_glob}') + GROUP BY pid, field) + """).fetchone()[0] + topk_path = out_root / "hot_topk.parquet" + if hot_manifest: + hot_list_sql = ",".join( + "'" + t.replace("'", "''") + "'" for t in hot_manifest) + con.execute(f""" + COPY ( + WITH field_avg AS ( + SELECT field, avg(doc_len) AS avg_dl FROM ( + SELECT pid, field, any_value(doc_len) AS doc_len + FROM read_parquet('{rows_glob}') + GROUP BY pid, field) + GROUP BY field + ), hot_rows AS ( + SELECT r.token, r.pid, r.field, r.tf, r.doc_len, + d.df, fa.avg_dl + FROM read_parquet('{rows_glob}') r + JOIN read_parquet('{out_root / "df.parquet"}') d USING (token) + JOIN field_avg fa USING (field) + WHERE r.token IN ({hot_list_sql}) + ), contrib AS ( + SELECT token, pid, + (CASE field + WHEN 'sample.label' THEN 3.0 + WHEN 'concept.label' THEN 2.5 + WHEN 'sample.place_name' THEN 2.0 + ELSE 1.0 END) + * ln((({total_docs_for_idf} - df + 0.5) / (df + 0.5)) + 1) + * (tf * 2.2) / (tf + 1.2 * (0.25 + 0.75 * doc_len / avg_dl)) + AS c + FROM hot_rows + ), per_pid AS ( + -- §5: rank per PID by the SUM of field-weighted + -- contributions, not per (pid, field) posting. + SELECT token, pid, sum(c) AS static_score + FROM contrib GROUP BY token, pid + ), ranked AS ( + SELECT *, row_number() OVER ( + PARTITION BY token ORDER BY static_score DESC, pid + ) AS rank + FROM per_pid + ) + SELECT token, pid, round(static_score, 4) AS static_score, rank + FROM ranked WHERE rank <= 500 + ORDER BY token, rank + ) TO '{topk_path}' (FORMAT PARQUET); + """) + topk_bytes = topk_path.stat().st_size + if topk_bytes > cap_bytes: + cap_violations.append(f"hot_topk.parquet: {topk_bytes/1e6:.1f} MB") + else: + topk_bytes = 0 + # shard_sizes.json — file bytes of every base shard, so the reader can + # compute a query's expected transfer BEFORE fetching (contract §6/§7, + # round-4 review: per-query budgeting must be computable, not guessed). + with open(out_root / "shard_sizes.json", "w") as f: + json.dump({f"shard_{i:03d}.parquet": + (out_root / f"shard_{i:03d}.parquet").stat().st_size + for i in range(args.shards)}, f, indent=1) + f.write("\n") + with open(out_root / "hot_tokens.json", "w") as f: + json.dump({ + "cap_bytes": cap_bytes, + "query_policy": ("TWO-TIER (contract §6): tokens with fetchable=true " + "— reader fetches ALL their hot/ sub-files (total fits " + "the cold budget) and they join the AND normally. " + "Tokens with fetchable=false are COMMON TERMS: dropped " + "from AND when >=1 other term survives (UI must " + "disclose); all-common queries rank via " + "hot_topk.parquet (per-pid summed static BM25 " + "top-500). Non-fetchable postings files exist for " + "offline/#172-oracle use only."), + "sub_shard_hash": ("sub-file membership uses DuckDB hash(pid) — a " + "consumer that DOES read hot postings fetches ALL " + "sub_files; the split function never needs " + "client-side computation"), + "topk_k": 500, + "topk_bytes": topk_bytes, + "tokens": hot_manifest}, f, indent=1) + f.write("\n") + if cap_violations: + print("WARNING: shard-cap violations (raise --shards or lower ratio):", + *cap_violations[:10], sep="\n ", file=sys.stderr) + + top_df = con.sql(f""" + SELECT token, count(*) AS df FROM read_parquet('{rows_glob}') + GROUP BY token ORDER BY df DESC LIMIT 20 + """).fetchall() + total_uncompressed = con.sql(f""" + SELECT sum(len(token) + len(pid) + len(field) + 4) + FROM read_parquet('{rows_glob}') + """).fetchone()[0] + + stats = { + "data_version": args.tag, + "built_at_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), + "total_samples": total_samples, + "total_documents": total_docs_for_idf, + "fields": { + f: { + "samples_with_field": field_stats[f]["samples_with_field"], + "total_tokens": field_stats[f]["total_tokens"], + "avg_doc_len": round( + field_stats[f]["total_tokens"] + / max(field_stats[f]["samples_with_field"], 1), 2), + } + for f in V1_FIELDS + }, + "concept_label_uri_resolution": resolution, + "concept_label_missing_pref_label": missing_pref_count, + "usmallint_truncation": { + "doc_len_docs_truncated": trunc["doc_len_docs"], + "tf_rows_truncated": trunc["tf_rows"], + "max_doc_len_observed": trunc["max_doc_len"], + "max_tf_observed": trunc["max_tf"], + }, + "shard_count": args.shards, + "shard_files": shard_files, + "hot_tokens": len(hot_manifest), + "shard_cap_violations": len(cap_violations), + "shard_hash": "fnv1a32(utf8(token)) % shards", + "shard_max_size_mb": round(shard_max_bytes / 1024 / 1024, 2), + "total_bytes_uncompressed": int(total_uncompressed or 0), + "build_seconds": round(time.time() - t0, 1), + "top_df_tokens": [[t, n] for t, n in top_df], + } + with open(out_root / "build_stats.json", "w") as f: + json.dump(stats, f, indent=1) + f.write("\n") + + for p in tmp_files: + os.unlink(p) + os.rmdir(tmp_dir) + + if trunc["doc_len_docs"] or trunc["tf_rows"]: + print(f"WARNING: USMALLINT truncation occurred — doc_len docs: " + f"{trunc['doc_len_docs']}, tf rows: {trunc['tf_rows']} " + f"(max observed doc_len {trunc['max_doc_len']}, tf {trunc['max_tf']}) " + "— BM25 semantics degraded for those documents", file=sys.stderr) + print(f"built {args.tag}_search_index_v1: {shard_files} shard files, " + f"max {stats['shard_max_size_mb']} MB, " + f"{total_samples:,} samples, {stats['build_seconds']}s") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/build_stats/isamples_202608_search_index_v1_build_stats.json b/tools/build_stats/isamples_202608_search_index_v1_build_stats.json new file mode 100644 index 0000000..984f427 --- /dev/null +++ b/tools/build_stats/isamples_202608_search_index_v1_build_stats.json @@ -0,0 +1,135 @@ +{ + "data_version": "isamples_202608", + "built_at_utc": "2026-07-10T22:58:34Z", + "total_samples": 6726892, + "total_documents": 17366831, + "fields": { + "sample.label": { + "samples_with_field": 6723722, + "total_tokens": 37802291, + "avg_doc_len": 5.62 + }, + "sample.description": { + "samples_with_field": 1652569, + "total_tokens": 30849932, + "avg_doc_len": 18.67 + }, + "sample.place_name": { + "samples_with_field": 2263648, + "total_tokens": 17311647, + "avg_doc_len": 7.65 + }, + "concept.label": { + "samples_with_field": 6726892, + "total_tokens": 87138723, + "avg_doc_len": 12.95 + } + }, + "concept_label_uri_resolution": { + "material_resolved": 1.0, + "material_missing_pref": 0.0, + "context_resolved": 1.0, + "context_missing_pref": 0.0, + "object_type_resolved": 1.0, + "object_type_missing_pref": 0.0, + "keywords_resolved": 1.0, + "keywords_missing_pref": 0.0 + }, + "concept_label_missing_pref_label": 0, + "usmallint_truncation": { + "doc_len_docs_truncated": 0, + "tf_rows_truncated": 0, + "max_doc_len_observed": 220, + "max_tf_observed": 19 + }, + "shard_count": 256, + "shard_files": 847, + "hot_tokens": 129, + "shard_cap_violations": 0, + "shard_hash": "fnv1a32(utf8(token)) % shards", + "shard_max_size_mb": 4.93, + "total_bytes_uncompressed": 7245750192, + "build_seconds": 516.4, + "top_df_tokens": [ + [ + "sample", + 7538687 + ], + [ + "material", + 5083658 + ], + [ + "object", + 4962091 + ], + [ + "other", + 4677450 + ], + [ + "solid", + 4531240 + ], + [ + "feature", + 4025455 + ], + [ + "sampled", + 4023625 + ], + [ + "any", + 4023265 + ], + [ + "individual", + 3164007 + ], + [ + "cm", + 2593846 + ], + [ + "organic", + 2305649 + ], + [ + "natural", + 2235884 + ], + [ + "rock", + 1802337 + ], + [ + "section", + 1791371 + ], + [ + "1", + 1490718 + ], + [ + "organism", + 1444732 + ], + [ + "core", + 1419466 + ], + [ + "animal", + 1340485 + ], + [ + "of", + 1240868 + ], + [ + "asia", + 1210156 + ] + ] +} diff --git a/tools/search_tokenizer.py b/tools/search_tokenizer.py new file mode 100644 index 0000000..31233dc --- /dev/null +++ b/tools/search_tokenizer.py @@ -0,0 +1,48 @@ +"""Canonical Python tokenizer for the iSamples search substrate (#169 §2, #170). + +MUST stay in lockstep with the JS twin `assets/js/search_tokenizer.js`. +Both implementations are run against `tests/search_tokenizer_regression.json` +in CI; any divergence is a hard failure (SEARCH_INDEX_V1.md §2). + +Pipeline (order matters, and is part of the contract): + 1. Unicode NFKC normalization (canonical + compatibility composition — + folds full-width forms, ligatures, etc.). + 2. Lowercase. + 3. Diacritic strip: NFD decomposition, drop combining marks (Mn), then + NFC recomposition of what's left. + 4. Replace every non-alphanumeric character with a space (punctuation + strip — hyphens, colons, slashes, quotes all become separators, so + `Iron-Age` → `iron age`, `IGSN:HRV000ABC` → `igsn hrv000abc`). + "Alphanumeric" is Unicode-aware (`str.isalnum()`), so CJK, Greek, + Cyrillic text survives. + 5. Whitespace split. + 6. Length filter: keep tokens with 1 <= len <= 64. + +No stemming. No stopword removal here — stopwords are indexed at build +time and dropped at QUERY time only (SEARCH_INDEX_V1.md §3), which this +module deliberately does not implement. +""" + +from __future__ import annotations + +import unicodedata + +MAX_TOKEN_LEN = 64 + + +def tokenize(text: str | None) -> list[str]: + """Tokenize one text fragment per the v1 contract. Deterministic, pure.""" + if not text: + return [] + # 1. NFKC compatibility normalization. + s = unicodedata.normalize("NFKC", text) + # 2. Lowercase. + s = s.lower() + # 3. Diacritic strip: NFD, drop combining marks, recompose. + s = unicodedata.normalize("NFD", s) + s = "".join(ch for ch in s if unicodedata.category(ch) != "Mn") + s = unicodedata.normalize("NFC", s) + # 4. Non-alphanumeric -> space (Unicode-aware). + s = "".join(ch if ch.isalnum() else " " for ch in s) + # 5-6. Split + length filter. + return [t for t in s.split() if 1 <= len(t) <= MAX_TOKEN_LEN]