blog: move the KV, not the work - P2P cache sharing in llm-d - #458
Draft
nilig wants to merge 90 commits into
Draft
blog: move the KV, not the work - P2P cache sharing in llm-d#458nilig wants to merge 90 commits into
nilig wants to merge 90 commits into
Conversation
✅ Deploy Preview for llm-d ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Placeholder draft (draft: true): benchmarks, figures, date, and co-authors still TODO. Signed-off-by: nilig <nili.ifergan@gmail.com>
…simplify decision section Signed-off-by: nilig <nili.ifergan@gmail.com>
Signed-off-by: nilig <nili.ifergan@gmail.com>
Signed-off-by: nilig <nili.ifergan@gmail.com>
Micro pull-vs-recompute crossover, hot-prefix routing comparison, and the 64x16K shared-prefix pool (low-rate per-request table + high-rate saturation table on the fixed build), plus a robustness note linking the findings bundle. Remaining unrun scenarios moved to a future list. Signed-off-by: nilig <nili.ifergan@gmail.com>
Signed-off-by: nilig <nili.ifergan@gmail.com>
Signed-off-by: nilig <nili.ifergan@gmail.com>
Signed-off-by: nilig <nili.ifergan@gmail.com>
Signed-off-by: nilig <nili.ifergan@gmail.com>
Signed-off-by: nilig <nili.ifergan@gmail.com>
Signed-off-by: nilig <nili.ifergan@gmail.com>
Signed-off-by: nilig <nili.ifergan@gmail.com>
Signed-off-by: nilig <nili.ifergan@gmail.com>
…stbed framing The 14x gpt-oss-120b document-QA result (p99 TTFT 2-4x lower, +17% throughput, 3x less run spread vs precise prefix routing) becomes the headline; the crossover table moves to gpt-oss numbers (-68% at 48K) with the Llama-8B corroboration inline; small-model scenarios keep their own testbed section; summary states the when-to-use rule. Signed-off-by: nilig <nili.ifergan@gmail.com>
…antislop pass The four Llama-8B figures get the same headline/spec/source chrome as the gpt-oss figures; text sweep removes filler intensifiers and marker phrasing. Signed-off-by: nilig <nili.ifergan@gmail.com>
docQA figure: precise guide recolored purple->gray so cache-first placement is one color across the post; gpt-oss crossover gets the headline/spec/source chrome and loses a title-label collision. Signed-off-by: nilig <nili.ifergan@gmail.com>
… texts Distinguish the Llama crossover point from gpt-oss winning at 2K and note the single-pair measurement is fleet-size independent; summary states the rule and points at the benchmark instead of repeating its numbers; hot-prefix and P/D sections get the forward pointer and the decode-bound convergence clause; all six images get descriptive alts. Signed-off-by: nilig <nili.ifergan@gmail.com>
Everything it referenced (P2P connector, EPP source selection, the guide, the offloading-connector pieces) is merged by the time this publishes, so the tracking pointer is stale on day one. Signed-off-by: nilig <nili.ifergan@gmail.com>
…und 2) - docQA p99: '2-4x' -> '28-74% reduction, up to 3.9x' (run 2 is 1.4x, not 2x) - hot prefix: '11x tail-latency win' -> '11x lower median latency' (source is p50) - pool TTFT: '25-30% lower' -> '5-30%, 25% at 8 req/s' (6 req/s is 5%) - P/D: 'recovers the affinity ceiling' -> 'nearly recovers (within 6%)' - soften two-run stability claim; add alternation explanation - crossover caption: 'near-flat' -> 'grows far slower than recompute' - gloss NIXL/ZMQ on first use; note NIXL-over-RDMA transport - docQA table split into p50/p95/p99 columns; clarify pull threshold Signed-off-by: nilig <nili.ifergan@gmail.com>
Replaces the placeholder after How P2P Works: consumer/producer schematic showing the EPP source selection, the ZMQ control exchange, the NIXL CPU-to-CPU data path, and the recompute fallback. Signed-off-by: nilig <nili.ifergan@gmail.com>
Per the connector source (P2PSecondaryTierManager owns ZmqTransport and NixlTransport, one bidirectional session per peer): both the control (ZMQ) and data (NIXL) paths run engine-to-engine between the CPU-tier P2P connectors, parallel, ZMQ bidirectional. The sidecar only injects the transfer params and hands off to the engine - it does not speak to the peer. Caption corrected to match. Signed-off-by: nilig <nili.ifergan@gmail.com>
The prefill worker computes the prompt KV and streams it to decode, so the prefill leg is where recomputing a cached prefix is wasted; the header is set against the prefill target and the sidecar injects on that leg. The decode leg already receives full KV over NIXL and has nothing to pull. P/D result numbers left unchanged pending the planned rerun. Signed-off-by: nilig <nili.ifergan@gmail.com>
…er sizing with the TP note Hash agreement needs identical --block-size alongside PYTHONHASHSEED; the CPU tier must be considerably larger than the pod's GPU KV (its value is what GPU evicts and CPU retains), with the ratio computed from measured KV capacity since it shifts with the TP degree. Signed-off-by: nilig <nili.ifergan@gmail.com>
…old calibration Replace the prefill-placement measurement with three paired results on the guide topologies: the pd-disaggregation guide with and without the P2P stack (document Q&A at concurrency 192: p50 11.9s to 1.16s, p99 -25%, +40% throughput), the decoder-to-prefiller pull mechanism under chat multi-turn (477K-1.65M tokens per run, flat per-turn TTFT), and the agentic-serving scenario shapes on P/D (p50 5.22s to 1.09s, +33% throughput, 1.23M tokens pulled, p99 parity from the cold-prefill ceiling). Add a threshold-calibration section with measured pull-vs-recompute crossovers for three models plus tier-sizing rules, and the two applicability boundaries (token-ID round-trip, matched TP). Three figures added; summary lead adjusted to reference the in-post agentic measurement. Signed-off-by: nilig <nili.ifergan@gmail.com>
List all three models' testbeds in the Benchmarks intro (the agentic Qwen topology was only described inline). Rework the threshold-calibration section to defer to the single-request sweep for gpt-oss and Llama - the quick two-point probes behind the removed table disagreed with the sweep's 5-rep medians - keeping Qwen as the worked example on its steady-state rate (~5x at 8K, crossover ~760, threshold 1024). Correct the run-N prompt-length range to 5K-20K in text and figure. Drop the stale 'benchmark numbers are placeholders' line from the draft banner. Signed-off-by: nilig <nili.ifergan@gmail.com>
TP-mismatched peers are supported for non-hybrid-attention models on the V1 model runner (V2-default models forced back via env var); hybrid models such as gpt-oss require matched TP. Replaces the overbroad 'peers must run matched TP' statement. Signed-off-by: nilig <nili.ifergan@gmail.com>
Signed-off-by: nilig <nili.ifergan@gmail.com>
Signed-off-by: nilig <nili.ifergan@gmail.com>
Signed-off-by: nilig <nili.ifergan@gmail.com>
Signed-off-by: nilig <nili.ifergan@gmail.com>
Signed-off-by: nilig <nili.ifergan@gmail.com>
nilig
force-pushed
the
blog/p2p-kv-cache-sharing
branch
from
July 30, 2026 19:43
f68266b to
5b1c3de
Compare
Signed-off-by: nilig <nili.ifergan@gmail.com>
…rge)" This reverts commit 40f04ac. Signed-off-by: nilig <nili.ifergan@gmail.com>
Signed-off-by: nilig <nili.ifergan@gmail.com>
Signed-off-by: nilig <nili.ifergan@gmail.com>
Signed-off-by: nilig <nili.ifergan@gmail.com>
The placement-rule table restated the What This Enables bullets; the closing bullets, the spill-mechanics sentence in the wide-EP section, the threshold re-listing, and the final bold line each restated content already stated once elsewhere. Signed-off-by: nilig <nili.ifergan@gmail.com>
…douts Mechanical review items: unambiguous multiplier directions, exact NIXL name, pull-vs-push clarification, transfer-floor byte arithmetic, threshold-margin note for the wide-EP arm, PYTHONHASHSEED warning admonition, boundary-section relabel to P2P-idle, and per-experiment provenance moved into collapsible measurement notes. Signed-off-by: nilig <nili.ifergan@gmail.com>
Signed-off-by: nilig <nili.ifergan@gmail.com>
Center the post on transfer crossover, wide-EP load spill, and P/D session continuity. Move deployment detail to the guide, retain the negative controls, and add a concise measurement roadmap. Signed-off-by: nilig <nili.ifergan@gmail.com>
Tie the 61.9K figure to the measured mean prompt length and avoid describing the entire prompt as accumulated history. Signed-off-by: nilig <nili.ifergan@gmail.com>
State the generated-turn template constraint and opt-in behavior explicitly. Restore precise operator settings and measurements while removing a redundant conclusion. Signed-off-by: nilig <nili.ifergan@gmail.com>
Keep the warning readable in the narrative while preserving the requirement that peers agree on block sizing and hash seeding. Signed-off-by: nilig <nili.ifergan@gmail.com>
Tie the GLM break-even point to the measured transfer floor and prefill cost. Explain that deployment-specific CPU-tier and fabric capacity make calibration a prerequisite. Signed-off-by: nilig <nili.ifergan@gmail.com>
michalmalka
reviewed
Aug 2, 2026
|
|
||
| llm-d already improves reuse in two ways. Prefix-aware routing sends requests | ||
| to pods that hold their prefixes. KV offloading keeps copies in a larger CPU | ||
| tier, and a filesystem backend can extend that tier to shared storage for |
Contributor
Author
There was a problem hiding this comment.
Right, the backend takes any mounted filesystem. Changed to "local or shared storage".
michalmalka
reviewed
Aug 2, 2026
| ### 1. Price the Transfer Before Using It | ||
|
|
||
| Pulling is useful only when it costs less than recomputing. The crossover | ||
| depends on the model, KV representation, hardware, and network, so it must be |
There was a problem hiding this comment.
what's about the GPU load - does this being influenced as part of the decision?
Contributor
Author
There was a problem hiding this comment.
The pull-vs-recompute decision itself is a static token-lead threshold (minCachedTokenDelta); live GPU load is not an input to whether we pull. Load does shape both ends of the transfer, though. The destination goes through the regular load-aware placement scorers, and source selection is load-aware (llm-d/llm-d-router#2032): peers within one block of the best cached count are sampled with weight 1/(1 + waiting queue), so a deeply-queued producer is avoided while a strictly better-cached peer still wins. Added a sentence on the source side to the mechanism section.
The router repo glossary reserves "llm-d Router" for the routing system and "Endpoint Picker (EPP)" for the scoring component; the post used them inverted, naming the EPP for a system-level capability and the router for the p2p-source-producer threshold. P2P is also expanded on first prose use. Signed-off-by: nilig <nili.ifergan@gmail.com>
michalmalka
reviewed
Aug 2, 2026
| * **Agentic burst fan-out.** Fan a parent session's repository-scale context | ||
| out to several sub-agents at once, then measure tail latency, source load, | ||
| and how well concurrent pulls preserve the reuse advantage. | ||
|
|
There was a problem hiding this comment.
Maybe also Prefetch from CPU/remote CPU before the request arrives...
Contributor
Author
There was a problem hiding this comment.
Added a prefetch bullet to the list. Overlapping the transfer with idle time instead of the request's critical path is probably the biggest remaining win for the multi-turn case.
The compression pass dropped the per-experiment setup tables, so results no longer said what was deployed or which arm was a shipped guide configuration. Each experiment now carries a one-line setup note, and the agentic workload is identified as the agentic-serving guide's conversation-replay profile scaled for a six-GPU testbed. Signed-off-by: nilig <nili.ifergan@gmail.com>
Each experiment's setup moves into a collapsible block whose summary line carries the deployment and configuration names, so the attribution stays visible while the detail stays out of the reading flow. "Arm" is replaced with plain wording throughout. Signed-off-by: nilig <nili.ifergan@gmail.com>
The filesystem tier takes local as well as shared storage, the mechanism section covers load-aware source sampling among near-tied holders, and the future-work list gains a prefetch item. Signed-off-by: nilig <nili.ifergan@gmail.com>
The -88% annotation ran from the pull point to a floating label, reading as a stray vertical line. A double-headed arrow between the recompute and pull points at 48K shows what the percentage compares. Signed-off-by: nilig <nili.ifergan@gmail.com>
The intro leads with the concrete routing dilemma instead of a mechanism summary, and the roles paragraph states where the producer's cost lands so the reader does not have to infer it. Signed-off-by: nilig <nili.ifergan@gmail.com>
Replace the synthetic-prefix GLM load-first example with a replayed Claude Code session that forks 43 subagents over a shared 40K prefix: p90 branch-start TTFT 11.2s -> 1.6s on cold fleets, median unchanged, with the straggler band removed and only burst-head colds remaining. Add the fork strip-plot figure, a results table for the Llama-3.1-8B shared-prefix pool, align cross-references to the new example, and point What's-Next at restart/preemption recovery. Add Maroon as co-author. Signed-off-by: nilig <nili.ifergan@gmail.com>
Signed-off-by: nilig <nili.ifergan@gmail.com>
Signed-off-by: nilig <nili.ifergan@gmail.com>
Signed-off-by: nilig <nili.ifergan@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Blog post for P2P KV cache sharing, the companion to the well-lit path
guide (llm-d/llm-d#2067).
The post covers the mechanism (source selection in the EPP, sidecar
injection, the vLLM
OffloadingConnectorP2P tier) and the measuredresults: single-request pull-versus-recompute crossovers (gpt-oss and
GLM wide-EP), the document-Q&A headline, uniform-pool and hot-set
ladders, the wide-EP load-spill pair (-67% mean TTFT / 2.7x), and P/D
agentic multi-turn. All numbers match the guide's benchmark reports at
their current heads; figures are generated by
scripts/generate-p2p-blog-charts.pyfrom the same data.Opened as draft with
draft: truein the frontmatter: the router-sidefix the wide-EP section depends on is still in review
(llm-d/llm-d-router#2234), and date/tags are placeholders until a
publish date is set.
Ref #424