feat(hotblocks): expose CF_TABLES memtable size and count - #106
Merged
Conversation
Production runs the stock 64 MB x 2 memtable, and both costs that motivated ADR 0002 trace back to it. The write stalls are the ceiling being hit -- immutable_memtables peaks at exactly 2 on every stalled pod and 1 on every healthy one, while files_at_level0 stays at 15 against a stop trigger of 36 and pending_compaction_bytes at 5.9% of its limit. And a 64 MB buffer flushes small L0 files continuously, which is most of the device write bill: at production I/O parity the bench goes 39.1 -> 16.3 GB at 512 MB, with the curve still falling at 1024. Defaults keep RocksDB's own values so the flags are inert until a deployment sets them. Scoped to CF_TABLES, which takes essentially all the write volume, so the memory cost is paid once rather than per column family. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
--rocksdb-write-buffer-mband--rocksdb-max-write-buffers. Defaults are RocksDB's own (64 MB × 2), so this is inert until a deployment sets them.Why
Production sets no memtable option at all.
The write stalls are the memtable ceiling, not compaction debt. Over 7 d,
write_stoppedfired on exactly two stacks, and the discriminator is exact:immutable_memtablesmaxfiles_at_level0maxpending_compaction_bytesmaxmax_write_buffer_numberis 2, soimmutable_memtables == 2is the stop condition, and it partitions the pods perfectly. L0 peaks at 15 against a stop trigger of 36; pending compaction peaks at 5.9 % of its hard limit.mainnet-internal-db-1spends 1.2 % of wall time write-stopped (~2 h/week).A 64 MB buffer is also most of the device write bill. Bench sweep at production I/O parity (Zstd WAL, 8 background jobs, direct I/O), free-run 45 × 2000:
Two effects separate cleanly: count fixes the stall (128 × 4 is already at zero with writes unchanged) and size drives volume (512 × 2 and 512 × 4 are identical). Commit max at 512 × 4 is 118 ms against 1569 ms as deployed; throughput rises 39 %.
Recommended deployment value
--rocksdb-write-buffer-mb 512 --rocksdb-max-write-buffers 4— 2 GB of memtable, scoped toCF_TABLES.Before rollout
max_bytes_for_level_baseis untouched at 256 MB, now smaller than one memtable — the numbers above are a floor on what tuning can do, not a ceiling.Full record:
docs/measurements/2026-07-27-rocksdb-memtable-tuning.md.Bearing on ADR 0002
Same bench, same build, same session: mdbx writes 13.27 GB free-run / 3.28 GB paced, against the tuned reference's 16.28 / 4.69 GB — a 1.2–1.4× gap where the ADR gates the port on ≥ 3×. At 1024 MB × 4 RocksDB writes less than mdbx. The engine case now rests on commit tails (mdbx p99 379 µs vs 3743 µs) and per-dataset isolation, not on write amplification.
🤖 Generated with Claude Code