feat(fsst): rewire writer/reader adapters onto the fsst module#295
Merged
Conversation
PR 5 of #287: reduce the two FSST production files to thin wire adapters over the standalone vortex-fsst module. Only production code touched; no test files changed (that is PR 6's scope). The wire format is unchanged. Deleted (writer): the ~250-line inline SymbolTable/SymbolCandidate machinery — training loop (trainSymbolTable), stratified sampling (sampleRowIndices), the open-addressing longest-match index, candidate counting, and per-row compress. All of that now lives in CompressorBuilder/Compressor/Matcher. Deleted (reader): the byte-by-byte decompressString loop. Decoding now goes through Decompressor (the unconditional 8-byte-store Algorithm 1). Kept (both): all wire/metadata plumbing — UTF-8 conversion, ProtoFSSTMetadata with narrowestUnsigned ptype selection, the EncodeNode/EncodeResult two-child tree (uncompLens/codesOff), arena allocation of every output buffer, the readUnsigned ptype dispatch, and the VarBinArray.OffsetMode result. The TRAINING_SAMPLE_SEED reproducibility constant is retained and now feeds CompressorBuilder.seed(). Wire-order code remapping (the subtle part — over-explained on purpose): The Compressor numbers its codes in gain-descending order, but the vortex.fsst wire format lays symbols out in length order (all multi-byte length-ascending, then length-1 last; mirrors Rust FSSTData::validate_symbol_lengths). So the writer must express BOTH the symbol table AND the code stream in wire codes. wireOrder = compressor.codesSortedByLength() — wireOrder[i] is the internal (gain-descending) code that belongs at wire position i. internalToWire[wireOrder[i]] = i — the inverse permutation, mapping every internal code the compressor emits to its wire position. Symbol table buffers: at wire position i write packedSymbol(wireOrder[i]) and symbolLength(wireOrder[i]) — the symbol living at wire slot i is internal symbol wireOrder[i]. Code stream: compressor.compress emits internal codes; each non-escape code byte c is rewritten to internalToWire[c]. The escape byte 0xFF and the single literal byte following it are copied through untouched — the literal is raw data, not a code, and 0xFF never collides with a code since codes are 0..254 (numSymbols <= 255). Reader side: it reads the wire symbol table into code-indexed arrays and a code b in the stream indexes wire slot b directly, so encoder position i and decoder index i refer to the same symbol — the file is self-consistent and round-trips. Verified: (1) writer,reader unit suite green with zero test changes (1713 tests); (2) checkstyle/javadoc gate green; (3) JavaWritesRustReadsIntegration javaWriter_jniReader_fsstUtf8Column — the real Rust reference reader decodes this encoder's output; (4) RustJavaReaderComparison fsst.vortex fixture (our decoder reads a Rust-written FSST file) and FileSizeComparison highCardinalityUtf8_javaVsJni (FSST still dispatch-selected) green; (5) full reactor build green; plus a throwaway 11-row driver proving encode→remap→wire symbol table→Decompressor reproduces every input byte-for-byte with the wire table in valid length order. Also fixed the empty-table read: with 0 trained symbols the writer floors the symbol buffers to a 1-byte placeholder, so the decoder derives the symbol count from symbolsBuf.byteSize()/8 (floors to 0) rather than symbolLensBuf.byteSize(). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR 5 of #287 — the only PR touching existing production code. Reduces
writer/FsstEncodingEncoderandreader/FsstEncodingDecoderto thin wire adapters over the standalonefsstmodule (PRs 1-4, already merged). Wire format is unchanged; no test files modified (that's PR 6).SymbolTable/SymbolCandidatemachinery — training, stratified sampling, the open-addressing match index. All of that now lives inCompressorBuilder/Compressor/Matcher.decompressStringloop, replaced byDecompressor's unconditional-8-byte-store decode, with the documented+7slack allocation and a read-only slice back to the true logical length so callers still see an exactly-sized buffer.ProtoFSSTMetadata+narrowestUnsignedptype selection, the two-childEncodeNode/EncodeResulttree, arena allocation,readUnsigneddispatch,VarBinArray.OffsetMode.TRAINING_SAMPLE_SEEDnow feedsCompressorBuilder.seed().wireOrder/internalToWireinverse-permutation logic in full; I reviewed it independently and the arithmetic on both the symbol-table population and the code-stream remapping checks out.Two real bugs were caught and fixed earlier in this sequence (PR 3's boundary-overrun match, since carried through PR 4) — none new found in this PR; I independently re-verified the interop gates myself rather than just trusting the agent's report, given this PR's risk level.
Test plan
./mvnw test -pl writer,reader -am— 1713 tests, 0 failures, zero test-file changes./mvnw verify -pl writer,reader -am— checkstyle/javadoc cleanJavaWritesRustReadsIntegrationTest(full class, 217 tests incl.javaWriter_jniReader_fsstUtf8Column) — the real Rust reference reader decoding this encoder's outputRustJavaReaderComparisonIntegrationTest(fsst.vortexfixture — our decoder reading a Rust-written file) andFileSizeComparisonIntegrationTest#highCardinalityUtf8_javaVsJni(FSST still dispatch-selected) — 251 tests total across all three classes, 0 failures, 2 unrelated skips./mvnw verify -DskipTestsat root — full reactor SUCCESS./mvnw javadoc:javadoc -pl writer,reader— zero new diagnostics🤖 Generated with Claude Code