llama/ggml: Hadamard weight-fold runtime, ported to prism-v7 by bri-prism · Pull Request #121 · PrismML-Eng/llama.cpp

bri-prism · 2026-08-26T01:23:35Z

Ports the Hadamard weight-fold runtime onto prism-v7. Supersedes #119 and #120, which are based on prism-v6 and cannot be merged now that the README marks that branch do-not-build.

What

Runtime support for models whose weights were Hadamard-folded offline, so a folded checkpoint can be packed to a low-bit format and still compute the original function:

A prism.hadamard v1 GGUF contract (block size, transform, axis, sign mode, explicit sign vectors, per-tensor roles). The loader validates it, materializes one rotation tensor and one sign tensor per (width, buffer type), and applies the activation-side transform immediately before every folded matmul. A runtime that does not understand the contract refuses to load rather than silently skipping the transform.
FWHT kernels extended from 512 to 1024 and 2048 wide, on CUDA and Metal, with F16 inputs accepted on CPU, CUDA, Metal and Vulkan. Backends without a matching kernel width fall back to a plain matmul against the real rotation matrix, so results stay correct everywhere.
Inverse transform after the token-embedding lookup, so an embedding table can stay in the quantized latent format instead of a dequantized copy.
Grouped V-head order for folded linear-attention out_proj, with the runtime permuting the activation instead: a post-fold column permutation on the rotation axis cannot be refolded without destroying the quantized codes.
A graph-build check that every folded weight actually receives its transform, so an architecture whose matmul path bypasses the transform helper fails loudly instead of computing nonsense.
convert_hf_to_gguf picks up a hadamard_packing.json manifest and writes the contract.

Porting notes

Metal moved its shaders into ggml/src/ggml-metal/kernels/ on this branch, so the FWHT kernel changes were re-landed in kernels/misc.metal rather than cherry-picked. Everything else applied as-is.

Validation

test-backend-ops -o MUL_MAT_HADAMARD: 18/18 on CUDA (RTX 4090, SM89, 2/2 backends) and 17/17 on Metal (3/3 backends), against the CPU reference, including the wide F32 and F16 cases and the fused sign-flip path at the three real contraction widths.
End to end on a folded group-128 checkpoint: loads with 402 folded weights and 1 inverse-lookup table, and decodes at the same rate as on the previous branch (23.12 vs 23.16 tok/s).
Numerics were certified on the previous branch and are unchanged by the port: logit KL divergence against the same model in unfolded F16 form sits at the measurement floor (mean 3.6e-4, same top-1 98.7%, PPL ratio statistically zero).

One thing worth deciding

A file quantized before the Q2_0_g128 to PQ2_0 rename records ftype 142 while this branch uses 141, so it loads and computes correctly but prints file type = unknown, may not work. Anything quantized on this branch gets the right value automatically. If a legacy alias in the ftype name switch is wanted, say so and I will add it here; I left it out because the renumber looked deliberate.

(cherry picked from commit 9146baa)

(cherry picked from commit 14d8bb5)

Register-resident warp kernel handles N up to 2048 for F32/F16 inputs. test-backend-ops MUL_MAT_HADAMARD passes 14/14 on H200 (SM90) vs CPU reference, including H2048 F32/F16. (cherry picked from commit db4c6ee)

Load prism.hadamard.* v1 metadata (block size, transform, axis, sign mode, folded weight names), materialize one persistent rotation tensor per (block size, buffer type), and apply x' = Hx immediately before every folded mul_mat / mul_mat_id via the FWHT hint. Identity signs only; unsupported contract values fail loading loudly. (cherry picked from commit 2f2ddf0)

Pick up hadamard_packing.json from the model dir, validate the fold contract (schema v1, identity signs, last-axis), map folded tensor names to GGUF names, and write the prism.hadamard.* keys. (cherry picked from commit ddc5ed9)

Metal advertises every mul_mat type combo, but the Hadamard-hint fallback for F16 inputs at widths without a dedicated FWHT kernel requests a nonexistent f32 x f16 mul_mv pipeline and crashes on the null pipeline. Decline the op in supports_op so it falls back to CPU instead. test-backend-ops MUL_MAT_HADAMARD: MTL0 13/13 (F16 wide-block case now correctly unsupported), 3/3 backends pass. (cherry picked from commit fe88d0c)

With mmap-backed buffer types (Metal shared buffers), weight->buffer is not assigned until load_all_data, so the rotation setup threw 'weight has no buffer' on every folded model. CUDA allocates buffers up front, which masked the ordering bug. (cherry picked from commit f37f047)

Instantiate the existing simdgroup-register FWHT template for 1024/2048 (F32 and F16 inputs) and extend the supported-size gate, replacing the dense rotation-matmul fallback for wide Hadamard-hint blocks. test-backend-ops MUL_MAT_HADAMARD: MTL0 14/14. Removes most of the Hadamard-hint decode overhead on Apple silicon versus the fallback. (cherry picked from commit 610ddad)

…aths The activation-side transform is applied only where the graph goes through build_lora_mm/build_lora_mm_id, but the converter accepted any mappable tensor and the loader accepted any weight name. Architectures that multiply projection weights with raw ggml_mul_mat (deepseek2 wq_a, kimi-linear, plm) would load a folded GGUF and silently skip the transform. Gate the contract on both ends: the converter refuses archs outside a verified set (llama, qwen3, qwen3moe, qwen35, qwen35moe, qwen3next) and tensors outside an explicit projection-kind allowlist, and the loader enforces the same checks on prism.hadamard.weight_names so folded GGUFs from other tools refuse to load rather than run wrong math. (cherry picked from commit e76b304)

…fp16 The fwht_f16 shaders declare float16_t, but VK_KHR_shader_float16_int8 is only enabled when device->fp16 is set, so unconditional creation could fail pipeline setup on devices without shader-float16 (or under GGML_VK_DISABLE_F16) even when no F16 FWHT is ever dispatched. Guard both the subgroup and shmem variants; ggml_vk_can_use_fwht already treats the null pipeline as unsupported and falls back to the rotation-matrix matmul. (cherry picked from commit 4edee06)

sign_mode 'explicit' carries per-width sign vectors in GGUF metadata (prism.hadamard.sign_widths + flattened sign_values). The loader materializes one F32 sign tensor per (width, buffer type) and the graph applies x' = H (s * x): elementwise sign flip, then the blockwise FWHT hint matmul. Identity mode is unchanged; unknown modes still refuse to load. Converter accepts schema v2 manifests with an explicit signs table. (cherry picked from commit c49590b)

Tensors listed in prism.hadamard.inverse_weight_names store latent (rotated) rows; the graph restores the primal basis right after the token-embedding lookup: h = s * (H z). This lets the embedding table stay in the quantized latent format instead of a dequantized primal copy. The rotation and sign tensors are shared with the matmul-side transforms; the direct-embeddings input path is untouched. (cherry picked from commit 2e85ea4)

The converter's tiled V-head reorder permutes out_proj's input axis, which is the rotation axis of a folded latent; a post-fold column permutation breaks the blockwise Hadamard correspondence and cannot be refolded without destroying the ternary codes. For folded out_proj tensors the converter now keeps the training (grouped) V order and sets prism.hadamard.gdn_v_grouped; the runtime permutes the activation tiled->grouped (reshape+permute+cont) before the sign flip and rotation. (cherry picked from commit e71a350)

Wrapper prefixes are stripped from tensor names before modify_tensors, so the exact-name check against the manifest never fired and the tiled V reorder still permuted folded rotation axes. (cherry picked from commit e9e4ea3)

An architecture whose matmul path bypasses the transform helpers would load a Hadamard-folded GGUF cleanly and silently compute wrong results. After reserving the worst-case graph, walk it once: every mul_mat consuming a folded weight must take the hint matmul's output (through reshape/view) built from that weight's rotation, and every get_rows of a latent table must feed an inverse transform. Violations abort context creation with the offending tensor name. (cherry picked from commit 34c9b5c)

The scheduler splits cross-backend paths with copy tensors (a CPU-mapped latent embedding feeding a GPU FWHT), which breaks the producer chain the check follows and produced a false positive. Verify the pristine graph right after build_graph, once per context, and log the consumers of an unverified lookup before aborting. (cherry picked from commit 8362c89)

The sign multiply was a separate full pass over the activation (4.3% of traced GPU time, ~1250 launches per run). Detect the mul+reshape+FWHT-hint subgraph in the CUDA graph evaluator and multiply the sign vector during the transform kernel's load instead. Backend-internal: the graph and every fallback path are unchanged. test-backend-ops gains fused-pattern cases at widths 5120/6144/17408. (cherry picked from commit deb4dfd)

Two fixes surfaced by a real folded checkpoint against the new verified-path allowlist: - The allowlist rejected two kinds that are on the build_lora_mm path and covered by the graph-walk check: output.weight (the lm head) and blk.N.attn_gate (the linear-attention z projection). A checkpoint folding either failed to load or convert. - Inverse (lookup-side) transforms inherited the buffer type of a host-mapped embedding table, bouncing the per-token inverse across the PCIe boundary: measured 3x decode loss on an RTX 4090 (27.7 -> 80.9 tok/s once device-side) while NVLink-class parts hid it. The inverse rotation and signs now use the buffer type the forward rotations chose. (cherry picked from commit 52ad701)

Copilot

Pull request overview

Adds runtime and conversion support for Hadamard-folded GGUF models, including activation transforms and backend acceleration.

Changes:

Defines and validates the prism.hadamard GGUF contract.
Applies forward/inverse transforms and verifies graph coverage.
Extends CPU, CUDA, Metal, and Vulkan FWHT support.

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
`tests/test-backend-ops.cpp`	Adds wide, F16, and signed FWHT tests.
`src/llama-model.h`	Stores Hadamard metadata and transforms.
`src/llama-model.cpp`	Loads contracts and allocates transform tensors.
`src/llama-model-loader.cpp`	Adds arbitrary-key metadata templates.
`src/llama-graph.h`	Defines graph transform structures.
`src/llama-graph.cpp`	Applies transforms around matmuls and embeddings.
`src/llama-context.h`	Tracks graph verification state.
`src/llama-context.cpp`	Verifies transform coverage.
`ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp`	Generates F16 FWHT shaders.
`ggml/src/ggml-vulkan/vulkan-shaders/fwht.comp`	Supports F16 shader input.
`ggml/src/ggml-vulkan/ggml-vulkan.cpp`	Dispatches F16 FWHT pipelines.
`ggml/src/ggml-metal/kernels/misc.metal`	Adds F16 and wide FWHT kernels.
`ggml/src/ggml-metal/ggml-metal-ops.cpp`	Dispatches typed FWHT kernels.
`ggml/src/ggml-metal/ggml-metal-device.m`	Validates Metal FWHT support.
`ggml/src/ggml-metal/ggml-metal-device.h`	Exposes FWHT capabilities.
`ggml/src/ggml-metal/ggml-metal-device.cpp`	Selects typed Metal pipelines.
`ggml/src/ggml-cuda/ggml-cuda.cu`	Adds signed FWHT fusion.
`ggml/src/ggml-cuda/fwht.cuh`	Declares signed FWHT dispatch.
`ggml/src/ggml-cuda/fwht.cu`	Adds wide, F16, and signed kernels.
`ggml/src/ggml-cpu/ops.cpp`	Supports F16 FWHT input.
`ggml/src/ggml-cpu/ggml-cpu.cpp`	Advertises mixed-type support.
`ggml/src/ggml-cpu/ggml-cpu.c`	Converts F16 matmul input.
`conversion/qwen.py`	Handles grouped folded projections.
`conversion/base.py`	Reads manifests and writes GGUF metadata.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

bri-prism · 2026-08-26T16:48:55Z

+            ml.get_arr("prism.hadamard.sign_widths", sign_widths);
+            ml.get_arr("prism.hadamard.sign_values", sign_values);


Correct, fixed in c0e4baf. With sign_widths empty the loop never runs, off stays 0, and the trailing off != sign_values.size() check compares 0 against 0 and passes, so the contract validates and the sign table stays empty. Explicit mode now requires at least one width.

bri-prism · 2026-08-26T16:48:56Z

+                if len(vec) != width or any(v not in (-1, 1) for v in vec):
+                    raise ValueError(f"invalid sign vector for width {width}")


Fixed in c0e4baf. The converter now applies the same rule as src/llama-model.cpp, rejecting a width that is not positive or not a multiple of block_size, so a manifest that converts also loads.

bri-prism · 2026-08-26T16:49:09Z

+                    # Hadamard-folded latent: the rotation axis must keep the
+                    # training (grouped) V order; the runtime permutes the
+                    # activation tiled->grouped before the transform instead.
+                    self._hadamard_gdn_v_grouped = True


Good catch, and this was the worst of the five since it produces wrong numbers rather than a load failure. Fixed in c0e4baf: the out_proj branch in _transform_nvfp4_weight now takes the same guard as modify_tensors, so a folded out-projection keeps grouped order and sets the grouped flag instead of being column-permuted on the rotation axis. Both paths now go through one helper, which is what let them drift apart in the first place. Same caveat as the CUDA gate: I have not run an NVFP4 prepacked conversion end to end, so this is verified by reading rather than by output.

bri-prism · 2026-08-26T16:48:53Z

+            signs->type == GGML_TYPE_F32 && mul->type == GGML_TYPE_F32 &&
+            (x->type == GGML_TYPE_F32 || x->type == GGML_TYPE_F16) &&


Confirmed and fixed in c0e4baf. ggml_mul_impl builds its result with ggml_dup_tensor(ctx, a), so mul->type follows x->type, and the gate required mul->type == F32 while allowing x to be F16. The whole condition collapsed to F32 only, which means the templated signed half kernel was dead and the F16 signed test was covering the unfused path exactly as you describe. The gate now requires mul->type == x->type with either supported input type. One caveat: this is the CUDA graph gate and I have no NVIDIA host right now, so it is not covered by the 17/17 MUL_MAT_HADAMARD run I did on Metal. It needs a CUDA run before merge.

bri-prism · 2026-08-26T16:49:08Z

+                suffix = "linear_attn.out_proj.weight"
+                if name.endswith(suffix) and any(n.endswith(suffix) for n in self.hadamard_folded_names()):


Fixed in c0e4baf. The suffix test is replaced with a per-tensor match that compares the full name modulo leading wrapper prefixes, so layer identity is preserved and a partially folded checkpoint no longer puts every layer's out_proj in grouped order.

…order scoping Five review findings, all real. The signed F16 fusion was unreachable. ggml_mul builds its result with ggml_dup_tensor, so mul->type follows x->type, and the gate required mul->type == F32 while allowing x to be F16. The condition collapsed to F32 only, so the templated signed half kernel never ran and the F16 signed test covered the unfused path. The gate now requires mul->type == x->type with either supported input type. An explicit sign contract with no widths passed validation, left the sign table empty, and was then read as identity mode, which silently changes the model function. Explicit mode now requires at least one width. The converter accepted zero-width and non-block-aligned sign vectors that the runtime rejects, so a manifest could convert into a GGUF this same branch cannot load. It now applies the runtime's width rule. Grouped V order was selected by tensor-kind suffix, so one folded out_proj in the manifest put every layer's out_proj in grouped order. A partially folded checkpoint then computed the non-folded layers with mismatched ordering. The match is now per-tensor, comparing full names modulo wrapper prefixes. The prepacked NVFP4 repack path bypassed that guard entirely: it column-permuted folded out-projections on the rotation axis, which cannot be refolded, and never set the grouped flag. It now takes the same branch as modify_tensors. test-backend-ops -o MUL_MAT_HADAMARD: 17/17 on Metal, 3/3 backends. The CUDA gate change is not covered by that run; it needs an NVIDIA host.

bri-prism added 18 commits August 25, 2026 18:12

ggml: support F16 input for FWHT matmul hint

6d25d20

(cherry picked from commit 9146baa)

ggml-cuda: advertise F16 Hadamard inputs

98a3706

(cherry picked from commit 14d8bb5)

ggml-cuda: extend FWHT matmul hint to 1024/2048 blocks

2fa1eae

Register-resident warp kernel handles N up to 2048 for F32/F16 inputs. test-backend-ops MUL_MAT_HADAMARD passes 14/14 on H200 (SM90) vs CPU reference, including H2048 F32/F16. (cherry picked from commit db4c6ee)

convert: match Hadamard-folded out_proj by suffix

cc92dd0

Wrapper prefixes are stripped from tensor names before modify_tensors, so the exact-name check against the manifest never fired and the tiled V reorder still permuted folded rotation axes. (cherry picked from commit e9e4ea3)

bri-prism requested a review from khosravipasha August 26, 2026 01:23

github-actions Bot added ggml Vulkan testing Apple Metal conversion CUDA labels Aug 26, 2026

khosravipasha requested a balanced review from Copilot August 26, 2026 01:52

Copilot started reviewing on behalf of khosravipasha August 26, 2026 01:53 View session

Copilot AI reviewed Aug 26, 2026

View reviewed changes

This was referenced Aug 26, 2026

llama/ggml: Hadamard weight-fold runtime (prism.hadamard GGUF contract, FWHT to 2048) #119

Closed

llama/ggml: explicit signs, latent embeddings, grouped out_proj for the Hadamard contract #120

Closed

Navigation Menu

Uh oh!

llama/ggml: Hadamard weight-fold runtime, ported to prism-v7 - #121

llama/ggml: Hadamard weight-fold runtime, ported to prism-v7#121
bri-prism wants to merge 19 commits into
prism-v7from
hadamard-v7

bri-prism commented Aug 26, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

bri-prism Aug 26, 2026

Uh oh!

bri-prism Aug 26, 2026

Uh oh!

bri-prism Aug 26, 2026

Uh oh!

bri-prism Aug 26, 2026

Uh oh!

bri-prism Aug 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		ml.get_arr("prism.hadamard.sign_widths", sign_widths);
		ml.get_arr("prism.hadamard.sign_values", sign_values);

		if len(vec) != width or any(v not in (-1, 1) for v in vec):
		raise ValueError(f"invalid sign vector for width {width}")

		signs->type == GGML_TYPE_F32 && mul->type == GGML_TYPE_F32 &&
		(x->type == GGML_TYPE_F32 \|\| x->type == GGML_TYPE_F16) &&

		suffix = "linear_attn.out_proj.weight"
		if name.endswith(suffix) and any(n.endswith(suffix) for n in self.hadamard_folded_names()):

Uh oh!

Conversation

bri-prism commented Aug 26, 2026

What

Porting notes

Validation

One thing worth deciding

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

bri-prism Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

bri-prism Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

bri-prism Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

bri-prism Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

bri-prism Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants