cachedb_perf: high-performance local memory cache built on modern kernel features

[Lt-Flash]

Summary

cachedb_perf is a local in-memory cachedb backend, selected by URL scheme (perf://). It implements the standard cachedb_funcs vtable, so every module that takes a cachedb_url (topology_hiding, dialog profiles, sql_cacher, ...) and the core cache_* script functions work unchanged — changing the URL is the whole migration.

It exists because cachedb_local's single fixed-size table and per-bucket locking become the bottleneck at high churn: at 50k resident entries and the topology-hiding access pattern, cachedb_perf does 448 ns/op vs 1,013 ns (2.3×; 7.9× vs cachedb_local's 512-bucket default), pure writes are 5.6–9.9× faster and stay flat with concurrency, and end-to-end topology hiding sustains 6,000 CPS where cachedb_local breaks at ~3,900.

How: lock-free reads (per-bucket version + copy-out validation; writers hold a bucket lock for ~hundreds of ns), named collections that grow at runtime, instant expiry with a cheap background sweep, and records in a size-class slab arena — in core shm or a dedicated reservation (memory_backing), with full reclaim in the dedicated modes. Keys hash with MurmurHash3 (the core's hash collides on sequential keys).

Optional, each degrading explicitly when absent: persistence to any db_* backend, cluster warm-up sync, and cross-node pull-on-miss — over the clusterer's bin links, the clusterer_controller plane, or the module's own udp/tcp sockets.

Outside the module, one optional cachedb capability: CACHEDB_CAP_GET_BUF, an allocation-free read into the caller's buffer. Backends and consumers that do not know it are unaffected.

Usage

loadmodule "cachedb_perf.so"
modparam("cachedb_perf", "cache_collections", "th=16")
modparam("cachedb_perf", "cachedb_url", "perf:///th")

loadmodule "topology_hiding.so"
modparam("topology_hiding", "th_state_url", "perf:///th")

From the script the backend id is perf (or perf:<group> when a grouped URL is declared): cache_store("perf", "call-$ci", "$var(state)", 3600);

Parameters

parameter	default
cache_collections	default=14	collections, name[=log2(buckets)];... — the size is only a starting point, tables grow
cachedb_url	perf://	URL(s) to resolve; perf:///name picks the collection; repeatable
memory_backing	auto	where records live: core shm, own-hg (dedicated huge-page arena), own (dedicated slab in shm, full reclaim)
arena_hugepage_mb	0	huge-page reservation size for the records; 0 = plain shm
arena_hugepage_cap_mb / arena_profile	—	growth cap / sizing profile of the dedicated arena
reclaim_keep, reclaim_quiet_s, reclaim_cooloff_s, reclaim_giveback	1/5/10/1	own-backing reclaim tuning
expiry_sweep_period	1	seconds between reclaim passes (expiry itself is instant)
growth_load_factor / growth_budget	2 / 4096	entries-per-bucket target / max splits per tick
db_url, db_table, db_mode, persist_collections	—	persistence: backend, table, auto load/save policy, which collections
sync_cluster_id, sync_shtag	—	cluster + sharing tag for warm-up sync
replicate_collections, pull_on_miss	—	which collections may pull from peers; repair misses transparently
pull_transport	bin	how pulls travel: bin, clctr, or module-owned udp/tcp sockets
pull_bind, pull_port	—	the udp/tcp transport's own address
pull_timeout_ms, pull_negative_ms	50/300	pull backstop / negative-answer cache
event_expired_collections, arena_selftest, htable_selftest	—	opt-in events and startup self-tests

MI (perf_stats, introspection, perf_save/perf_load/perf_sync), statistics and events are documented in the module README.

Docs, testing

Everything measured, with graphs, lives in the branch: doc/STUDY.md (the study), doc/PR-NOTES.md (progress notes), bench/README.md (the test/benchmark harness). Builds clean with gcc and clang under -Werror; no new dependencies.