llama/ggml: explicit signs, latent embeddings, grouped out_proj for the Hadamard contract - #120
Closed
bri-prism wants to merge 8 commits into
Closed
llama/ggml: explicit signs, latent embeddings, grouped out_proj for the Hadamard contract#120bri-prism wants to merge 8 commits into
bri-prism wants to merge 8 commits into
Conversation
sign_mode 'explicit' carries per-width sign vectors in GGUF metadata (prism.hadamard.sign_widths + flattened sign_values). The loader materializes one F32 sign tensor per (width, buffer type) and the graph applies x' = H (s * x): elementwise sign flip, then the blockwise FWHT hint matmul. Identity mode is unchanged; unknown modes still refuse to load. Converter accepts schema v2 manifests with an explicit signs table.
Tensors listed in prism.hadamard.inverse_weight_names store latent (rotated) rows; the graph restores the primal basis right after the token-embedding lookup: h = s * (H z). This lets the embedding table stay in the quantized latent format instead of a dequantized primal copy. The rotation and sign tensors are shared with the matmul-side transforms; the direct-embeddings input path is untouched.
The converter's tiled V-head reorder permutes out_proj's input axis, which is the rotation axis of a folded latent; a post-fold column permutation breaks the blockwise Hadamard correspondence and cannot be refolded without destroying the ternary codes. For folded out_proj tensors the converter now keeps the training (grouped) V order and sets prism.hadamard.gdn_v_grouped; the runtime permutes the activation tiled->grouped (reshape+permute+cont) before the sign flip and rotation.
Wrapper prefixes are stripped from tensor names before modify_tensors, so the exact-name check against the manifest never fired and the tiled V reorder still permuted folded rotation axes.
An architecture whose matmul path bypasses the transform helpers would load a Hadamard-folded GGUF cleanly and silently compute wrong results. After reserving the worst-case graph, walk it once: every mul_mat consuming a folded weight must take the hint matmul's output (through reshape/view) built from that weight's rotation, and every get_rows of a latent table must feed an inverse transform. Violations abort context creation with the offending tensor name.
The scheduler splits cross-backend paths with copy tensors (a CPU-mapped latent embedding feeding a GPU FWHT), which breaks the producer chain the check follows and produced a false positive. Verify the pristine graph right after build_graph, once per context, and log the consumers of an unverified lookup before aborting.
The sign multiply was a separate full pass over the activation (4.3% of traced GPU time, ~1250 launches per run). Detect the mul+reshape+FWHT-hint subgraph in the CUDA graph evaluator and multiply the sign vector during the transform kernel's load instead. Backend-internal: the graph and every fallback path are unchanged. test-backend-ops gains fused-pattern cases at widths 5120/6144/17408.
bri-prism
force-pushed
the
hadamard-explicit-signs
branch
from
August 24, 2026 01:58
9c5a1a4 to
deb4dfd
Compare
Two fixes surfaced by a real folded checkpoint against the new verified-path allowlist: - The allowlist rejected two kinds that are on the build_lora_mm path and covered by the graph-walk check: output.weight (the lm head) and blk.N.attn_gate (the linear-attention z projection). A checkpoint folding either failed to load or convert. - Inverse (lookup-side) transforms inherited the buffer type of a host-mapped embedding table, bouncing the per-token inverse across the PCIe boundary: measured 3x decode loss on an RTX 4090 (27.7 -> 80.9 tok/s once device-side) while NVLink-class parts hid it. The inverse rotation and signs now use the buffer type the forward rotations chose.
This was referenced Aug 26, 2026
Author
|
Closing in favour of #121, which is the same work ported onto prism-v7. This targets prism-v6, and the prism-v7 README now marks that branch as a stale mid-migration snapshot that should not be built from, so this cannot land as-is. Nothing is lost. #121 carries all 18 commits from this branch and this one, rebased onto prism-v7, with the Metal FWHT kernels re-landed into the new ggml-metal/kernels layout since v7 moved the shaders out of the monolithic metal file. Verified there: 18/18 MUL_MAT_HADAMARD on CUDA (RTX 4090, 2/2 backends) and 17/17 on Metal (3/3 backends), and a folded checkpoint loads with 402 folded weights plus 1 inverse-lookup table and decodes at the same rate as on this branch. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #119, stacked on
hadamard-gguf-contract. That PR shipped the identity-sign contract; this one adds what a real trained checkpoint needs.What
sign_mode "explicit"carries per-width sign vectors in GGUF metadata (prism.hadamard.sign_widthsplus a flattenedsign_values). The loader materializes one F32 sign tensor per (width, buffer type) alongside the rotation, and the graph appliesx' = H (s * x): an elementwise sign flip, then the existing blockwise FWHT hint matmul. Identity mode is unchanged and unknown modes still refuse to load.prism.hadamard.inverse_weight_namesstore latent (rotated) rows; the graph restores the primal basis right after the token-embedding lookup,h = s * (H z), so an embedding table can stay in the quantized latent format instead of a dequantized copy. The direct-embeddings input path is untouched.prism.hadamard.gdn_v_grouped; the runtime permutes the activation tiled-to-grouped (reshape, permute, cont) before the sign flip and rotation.Why
Trained Hadamard checkpoints use seeded sign flips and rotate every projection including the embedding and output head. Without these three pieces the contract could only represent identity-sign folds on plain matmuls.
Validation
test-backend-ops -o MUL_MAT_HADAMARD: 14/14 on CUDA (SM90) and Metal against the CPU reference.