ggml-cpu: add opt-in Q2_0 VNNI64 four-row decode - #95
Conversation
github-actions
Bot
added
documentation
|
Hi Thanks for the PR, is that something we can upstream to main llama.cpp This seems in draft mode did we mean to merge it? And also has lots of extra files that probably wont want to merge. |
There was a problem hiding this comment.
Pull request overview
Adds an opt-in Q2_0 packed-cache decode path for AVX-512 VNNI CPU inference.
Changes:
- Adds Q2_0 packing formats and implementation.
- Integrates scalar and SIMD decode routes into CPU repacking.
- Adds parity tooling and performance evidence.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
ggml/src/ggml-cpu/CMakeLists.txt |
Builds the packing source. |
ggml/src/ggml-cpu/repack.cpp |
Implements repacking and decode routes. |
third_party/tbkern/include/tbkern/format.h |
Defines packed Q2_0 formats. |
third_party/tbkern/include/tbkern/q2_pack.h |
Declares the packing API. |
third_party/tbkern/src/q2_pack.c |
Implements Q2_0 packing. |
scripts/tbkern_compare.py |
Compares parity artifacts. |
docs/tbkern/phase9-four-row-vnni64.md |
Documents four-row decoding. |
docs/tbkern/phase9-prism-evidence.md |
Records validation evidence. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| target_include_directories(${GGML_CPU_NAME} PRIVATE . ggml-cpu) | ||
|
|
||
| # Resolve from the project root inside this function to avoid caller-scope shadowing. | ||
| set(TBKERN_ROOT "${CMAKE_SOURCE_DIR}/third_party/tbkern") |
| /* Mainline GGUF Q2_0 native block covers 64 weights (QK2_0). PrismML's native | ||
| * variant uses group-128 blocks with identical code semantics. All tbkern code | ||
| * parameterizes the group size G at runtime; these are the only legal values. */ | ||
| #define TBK_QK2_0 64 /* mainline GGUF Q2_0 block length */ |
| * Ground truth: mainline llama.cpp GGML_TYPE_Q2_0 (ggml-common.h / ggml-quants.c). | ||
| * See docs/FORMAT.md for the normative spec and bit-exact diagrams. | ||
| * | ||
| * RULE (docs/CONTEXT.md §Rules): after Phase 0 this header may only be changed | ||
| * by the orchestrator. Code against it; do not edit it. | ||
| */ |
| /* Code semantics (2-bit quant code -> ternary/quaternary weight value): | ||
| * 00 = -1, 01 = 0, 10 = +1, 11 = +2. value = (code - 1) * scale. | ||
| * The unsigned datum u = code = (w + 1) in {0..3} is what the VNNI path feeds | ||
| * to vpdpbusd (see docs/FORMAT.md §VNNI). +2 (code 3) is legal Q2_0 but never |
| @@ -0,0 +1,155 @@ | |||
| /* tbkern/format.h — FROZEN Phase-0 contract. | |||
| int tbk_pack_from_q2(const uint8_t *gguf_q2_blocks, int M, int K, int G, | ||
| tbk_layout_kind layout, tbk_mat *out) { |
| static bool tbkern_q2_0_enabled() { | ||
| return tbkern_q2_0_scalar_enabled() || tbkern_q2_0_avx2_enabled() || tbkern_q2_0_vnni_enabled() || tbkern_q2_0_vnni64_enabled() || tbkern_q2_0_vnni64_4r_enabled(); |
|
I ran the benchmark again, it seems any performance gains I had were on an earlier version of your main branch here and were superseded after rebasing from main. I ran this again today with this opt-in turned on and performance was worse, so not sure it would be worth merging at this point. Happy to clean it up if you want to keep this opt-in as an option. Thanks for supporting the CPU path for this quantization as it could be useful for batch inference on CPUs. |
Summary
This draft reduces the Prism CPU proposal to the minimal Q2_0 backend change and its evidence:
Scope
Kept:
Removed from the proposal:
Real-weight evidence
Model: Ternary-Bonsai-27B-Q2_0.gguf
Host: Google Cloud C3 highmem-8, Intel Xeon Platinum 8481C, 4 physical cores plus SMT, 8 ggml threads
Command: llama-bench -m MODEL -p 0 -n 16 -t 8 -ngl 0 -b 32 -ub 32
Baseline is unmodified Prism on the same model, host, command, and thread count:
The historical matched set shows approximately a 40% target-decode improvement.
Correctness was bit-identical on the real model:
The selector remains opt-in because the recorded perplexity workload was slower even though PPL matched. Attention, norms, RoPE, softmax, GDN recurrence/convolution, KV cache, sampling, scheduling, and GPU execution remain Prism-native.
Validation
Raw model files and logs are not included.