Skip to content

llama/ggml: explicit signs, latent embeddings, grouped out_proj for the Hadamard contract - #120

Closed
bri-prism wants to merge 8 commits into
hadamard-gguf-contractfrom
hadamard-explicit-signs
Closed

llama/ggml: explicit signs, latent embeddings, grouped out_proj for the Hadamard contract#120
bri-prism wants to merge 8 commits into
hadamard-gguf-contractfrom
hadamard-explicit-signs

Conversation

@bri-prism

Copy link
Copy Markdown

Follow-up to #119, stacked on hadamard-gguf-contract. That PR shipped the identity-sign contract; this one adds what a real trained checkpoint needs.

What

  • Explicit sign vectors. sign_mode "explicit" carries per-width sign vectors in GGUF metadata (prism.hadamard.sign_widths plus a flattened sign_values). The loader materializes one F32 sign tensor per (width, buffer type) alongside the rotation, and the graph applies x' = H (s * x): an elementwise sign flip, then the existing blockwise FWHT hint matmul. Identity mode is unchanged and unknown modes still refuse to load.
  • Inverse transform for latent embedding tables. Tensors listed in prism.hadamard.inverse_weight_names store latent (rotated) rows; the graph restores the primal basis right after the token-embedding lookup, h = s * (H z), so an embedding table can stay in the quantized latent format instead of a dequantized copy. The direct-embeddings input path is untouched.
  • Grouped-order linear-attention out_proj. The converter's tiled V-head reorder permutes out_proj's input axis, which is the rotation axis of a folded latent; a post-fold column permutation breaks the blockwise correspondence and cannot be refolded without destroying the quantized codes. For folded out_proj tensors the converter now keeps the grouped V order and sets prism.hadamard.gdn_v_grouped; the runtime permutes the activation tiled-to-grouped (reshape, permute, cont) before the sign flip and rotation.
  • Converter accepts schema v2 manifests (signs table, per-tensor roles), plus two missing template instantiations that only surface with the GNU linker.

Why

Trained Hadamard checkpoints use seeded sign flips and rotate every projection including the embedding and output head. Without these three pieces the contract could only represent identity-sign folds on plain matmuls.

Validation

  • test-backend-ops -o MUL_MAT_HADAMARD: 14/14 on CUDA (SM90) and Metal against the CPU reference.
  • End to end on a large hybrid-attention checkpoint packed with every projection folded (explicit signs, block 1024): logit KL divergence against the same model in unfolded F16 form lands at the measurement noise floor (mean KLD 3.7e-4, median 3.1e-4, same top-1 98.7%, mean ln PPL ratio -0.00005), i.e. the folded quantized pack is equivalent to the reference to the limit of the measurement.
  • The activation-side transform stack measures at roughly 10% of GPU time on an H200 trace (FWHT 4.8%, sign multiplies 4.3%, permute copies about 2%), with obvious fusion headroom later.

sign_mode 'explicit' carries per-width sign vectors in GGUF metadata
(prism.hadamard.sign_widths + flattened sign_values). The loader
materializes one F32 sign tensor per (width, buffer type) and the graph
applies x' = H (s * x): elementwise sign flip, then the blockwise FWHT
hint matmul. Identity mode is unchanged; unknown modes still refuse to
load. Converter accepts schema v2 manifests with an explicit signs table.
Tensors listed in prism.hadamard.inverse_weight_names store latent
(rotated) rows; the graph restores the primal basis right after the
token-embedding lookup: h = s * (H z). This lets the embedding table
stay in the quantized latent format instead of a dequantized primal
copy. The rotation and sign tensors are shared with the matmul-side
transforms; the direct-embeddings input path is untouched.
The converter's tiled V-head reorder permutes out_proj's input axis,
which is the rotation axis of a folded latent; a post-fold column
permutation breaks the blockwise Hadamard correspondence and cannot be
refolded without destroying the ternary codes. For folded out_proj
tensors the converter now keeps the training (grouped) V order and sets
prism.hadamard.gdn_v_grouped; the runtime permutes the activation
tiled->grouped (reshape+permute+cont) before the sign flip and rotation.
Wrapper prefixes are stripped from tensor names before modify_tensors,
so the exact-name check against the manifest never fired and the tiled
V reorder still permuted folded rotation axes.
An architecture whose matmul path bypasses the transform helpers would
load a Hadamard-folded GGUF cleanly and silently compute wrong results.
After reserving the worst-case graph, walk it once: every mul_mat
consuming a folded weight must take the hint matmul's output (through
reshape/view) built from that weight's rotation, and every get_rows of
a latent table must feed an inverse transform. Violations abort context
creation with the offending tensor name.
The scheduler splits cross-backend paths with copy tensors (a CPU-mapped
latent embedding feeding a GPU FWHT), which breaks the producer chain
the check follows and produced a false positive. Verify the pristine
graph right after build_graph, once per context, and log the consumers
of an unverified lookup before aborting.
The sign multiply was a separate full pass over the activation (4.3% of
traced GPU time, ~1250 launches per run). Detect the mul+reshape+FWHT-hint
subgraph in the CUDA graph evaluator and multiply the sign vector during
the transform kernel's load instead. Backend-internal: the graph and every
fallback path are unchanged. test-backend-ops gains fused-pattern cases
at widths 5120/6144/17408.
@bri-prism
bri-prism force-pushed the hadamard-explicit-signs branch from 9c5a1a4 to deb4dfd Compare August 24, 2026 01:58
Two fixes surfaced by a real folded checkpoint against the new
verified-path allowlist:

- The allowlist rejected two kinds that are on the build_lora_mm path and
  covered by the graph-walk check: output.weight (the lm head) and
  blk.N.attn_gate (the linear-attention z projection). A checkpoint
  folding either failed to load or convert.
- Inverse (lookup-side) transforms inherited the buffer type of a
  host-mapped embedding table, bouncing the per-token inverse across the
  PCIe boundary: measured 3x decode loss on an RTX 4090 (27.7 -> 80.9
  tok/s once device-side) while NVLink-class parts hid it. The inverse
  rotation and signs now use the buffer type the forward rotations chose.
@bri-prism

Copy link
Copy Markdown
Author

Closing in favour of #121, which is the same work ported onto prism-v7.

This targets prism-v6, and the prism-v7 README now marks that branch as a stale mid-migration snapshot that should not be built from, so this cannot land as-is.

Nothing is lost. #121 carries all 18 commits from this branch and this one, rebased onto prism-v7, with the Metal FWHT kernels re-landed into the new ggml-metal/kernels layout since v7 moved the shaders out of the monolithic metal file. Verified there: 18/18 MUL_MAT_HADAMARD on CUDA (RTX 4090, 2/2 backends) and 17/17 on Metal (3/3 backends), and a folded checkpoint loads with 402 folded weights plus 1 inverse-lookup table and decodes at the same rate as on this branch.

@bri-prism bri-prism closed this Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant