feat(fsst): MemorySegment hot paths + unconditional-store decode#294
Merged
Conversation
Add MemorySegment-native overloads of Compressor#compress and Decompressor#decompress alongside the existing byte[] versions, so the fsst module speaks FFM at its hot-path boundary — matching this project's MemorySegment-not-JNI convention (CLAUDE.md) and how PR 5's writer/reader adapters will drive it with zero intermediate heap copies. The overloads read/write directly against caller-provided segments; no scratch buffers are allocated inside them. Decompressor's MemorySegment overload implements the FSST paper's Algorithm 1 "unconditional 8-byte store" trick: for each non-escape code, one little-endian 8-byte store of the packed symbol, then advance the output cursor by only the symbol's true length. The store width is a compile-time constant and the loop body stays uniform (hot-loop rule). The overwritten slack bytes are covered by the next store, except for the final symbol of the final row — so callers MUST allocate the output with at least 7 bytes of trailing slack past the total decoded length. This +7 precondition is documented prominently in the javadoc; PR 5 will actually allocate it in FsstEncodingDecoder. The MemorySegment compress overload carries the exact pos + length <= end boundary guard introduced in PR 3: loadWord zero-pads past the input end and the branch-free Matcher has no notion of end, so without the guard a trained symbol with trailing zeros can spuriously match the padding and decode to more bytes than were compressed, corrupting NUL-tailed/binary rows. Ported carefully since a port can silently drop a boundary check that is not obviously part of the core algorithm. loadWord for segment input assembles the word byte by byte bounded by both end and the segment's own size (a wide unconditional read is unsafe here, unlike the same-array byte[] over-read). Tests: byte[] vs MemorySegment parity across empty/single-byte/ repetitive/incompressible/boundary-overrun inputs; the 8-byte-store trick over multiple consecutive rows into one shared +7-slack segment with no cross-row corruption; and the MemorySegment port of PR 3's compress_symbolWithTrailingZeros_doesNotMatchPastEndOfInput regression. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR 4 of #287: adds
MemorySegment-native overloads ofCompressor#compress/Decompressor#decompressalongside the existingbyte[]versions, matching this project's FFM-everywhere convention (CLAUDE.md: "never JNI... uses FFM") at the module's hot-path boundary — zero intermediate heap copies for PR 5's writer/reader adapters to drive.Decompressor'sMemorySegmentoverload implements the paper's Algorithm 1 "unconditional 8-byte store" trick: one little-endian 8-byte store per code, advance by only the true length. Callers must allocateoutwith +7 bytes of trailing slack (documented prominently); PR 5 will do the actual allocation.Compressor'sMemorySegmentoverload carries the exactpos + length <= endboundary guard introduced in PR 3 — a port can silently drop a check that isn't obviously part of the "core algorithm," so this was verified carefully.loadWord(MemorySegment...)assembles the word byte-by-byte bounded by both the logicalendand the segment's ownbyteSize()— unlike the byte[] version, a wide unconditional read isn't safe here since the caller-provided segment has no guaranteed trailing slack.Test plan
./mvnw test -pl fsst -am— 50 tests (7 new), 0 failures — parity (byte[] vs MemorySegment, including the boundary-overrun case), the 8-byte-store trick across multiple rows with no cross-row corruption, and the MemorySegment port of PR 3's boundary regression./mvnw verify -pl fsst -am— checkstyle clean./mvnw javadoc:javadoc -pl fsst— zero output./mvnw verify -DskipTestsat root — full reactor SUCCESS🤖 Generated with Claude Code