Skip to content

Add Intl.Segmenter modes: intl-word and grapheme - #25

Merged
krkarma777 merged 1 commit into
masterfrom
feature/intl-segmenter
Aug 31, 2026
Merged

Add Intl.Segmenter modes: intl-word and grapheme#25
krkarma777 merged 1 commit into
masterfrom
feature/intl-segmenter

Conversation

@krkarma777

Copy link
Copy Markdown
Owner

Closes #15

Summary

  • `mode: 'intl-word'`: locale-aware word segmentation via `Intl.Segmenter` — unspaced scripts (Japanese/Chinese/Thai) now diff by word where `word` mode sees one opaque letter-run (私は猫が好きです → 私は犬が好きです isolates 猫→犬)
  • `mode: 'grapheme'`: cluster-safe character diff — ZWJ emoji (👨‍👩‍👧) and combining sequences stay whole where `char` splits code points
  • New `locale` option (BCP 47, defaults to runtime locale); Segmenter instances cached per granularity+locale; clear `TypeError` on runtimes without `Intl.Segmenter`
  • `refine` drops `intl-word` pairs to grapheme granularity; `heuristic` composes
  • Both modes route through the generic `diffTokens` pipeline; the fast scanner modes are untouched

Test Plan

  • 7 new tests: Japanese word isolation (with the word-mode contrast), ZWJ replacement, refine-to-grapheme, 200 mixed-script fuzz round-trips, heuristic composition, empty inputs
  • Full suite 50/50, typecheck clean (lib bumped to ES2022 for Segmenter types; emit target unchanged)

@krkarma777
krkarma777 merged commit 9168b90 into master Aug 31, 2026
5 checks passed
@krkarma777
krkarma777 deleted the feature/intl-segmenter branch August 31, 2026 02:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: Intl.Segmenter-based word and grapheme segmentation modes

1 participant