Add GLM-5.2 full-context H200 AgentX with SGLang PCP and HiCache / 新增 GLM-5.2 H200 全上下文 AgentX SGLang PCP 与 HiCache 配置

[JordanNanos]

Description

Add a three-node H200 AgentX qualification path for GLM-5.2 FP8 at its native 1,048,576-token context:

• run one eight-rank SGLang DSA prefill worker as effective TP1 x PCP8 with LayerSplit;
• transfer KV through Mooncake to one TP16 decode worker across two H200 nodes;
• enable prefill-side HiCache with direct I/O, layer-first layout, and write-back;
• retain decode-side RadixCache without decode HiCache, avoiding the long-prefix restore correctness failure found during bring-up;
• replay the unfiltered AgentX trace as seven isolated jobs at concurrency 1, 2, 3, 4, 5, 6, and 8, so one failed point cannot discard completed curve data;
• extend the H200 srt-slurm launcher with the checked-in AgentX recipe, persistent dataset/model caches, and serialized image import.

DCP is intentionally not enabled or labeled. SGLang DSA decode-context parallelism for GLM-5.2 is not merged, so TP16 provides active-KV and runtime HBM headroom without claiming unsupported DCP behavior. MTP is also deferred while the GLM-5.2 MTP plus HiCache correctness issue remains open.

Official sweep

The official full-sweep-fail-fast run passed on commit 5180d246: run 29717702856.

All seven c1/c2/c3/c4/c5/c6/c8 AgentX points completed successfully with zero error-dropped profiled requests. The observed throughput optimum is c3 at 19,626 total tok/s (817.77 tok/s/GPU); c2 is the latency-efficient point at 16,966 tok/s and 2.88s mean TTFT. Saturation begins at c4. See the full result table and bilingual analysis.

Validation

• python -m pytest utils/matrix_logic/ -v: 224 passed on 5180d246
• targeted matrix generation: exactly seven H200/Dynamo-SGLang/AgentX jobs at c1/c2/c3/c4/c5/c6/c8
• bash -n passes for the modified launcher and benchmark library
• recipe previously loaded successfully with NVIDIA srt-slurm v1.0.10 schema
• branch merged with current main; the Add GLM-5.2 full-context H200 AgentX with SGLang PCP and HiCache / 新增 GLM-5.2 H200 全上下文 AgentX SGLang PCP 与 HiCache 配置 #2275 changelog entry is appended at the new tail

Related Issue

None.

Type of Change

• Bug fix
• New feature
• Configuration change
• Documentation update
• Other

Checklist

• I have tested my changes locally
• Documentation updates are not required
• I have updated perf-changelog.yaml with this PR number
  • The new entry is appended at the end
• Before reuse merge, an authorized maintainer has commented /reuse-sweep-run only after a final green full sweep with evals passing

中文说明

为 GLM-5.2 FP8 新增三节点 H200 AgentX 资格验证路径，使用其原生 1,048,576 token 上下文：

• 单个八 rank SGLang DSA 预填充 worker 按有效 TP1 x PCP8 运行，并启用 LayerSplit；
• 通过 Mooncake 将 KV 传输到跨两台 H200 节点的单个 TP16 解码 worker；
• 预填充侧启用 HiCache，采用 direct I/O、layer-first 内存布局和 write-back 策略；
• 解码侧仅保留 RadixCache，不启用 HiCache，以规避 bring-up 阶段发现的长前缀恢复正确性故障；
• 使用并发 1、2、3、4、5、6、8 七个相互隔离的任务回放未截断的 AgentX 轨迹，避免单点失败导致已完成的曲线数据丢失；
• 扩展 H200 srt-slurm 启动器，支持仓库内 AgentX 配方、持久化数据集/模型缓存以及串行化镜像导入。

本 PR 不启用或标记 DCP。SGLang 针对 GLM-5.2 DSA 的解码上下文并行尚未合并，因此采用 TP16 为活跃 KV 和运行时显存预留空间，避免宣称尚未支持的 DCP 能力。GLM-5.2 的 MTP 与 HiCache 组合仍存在未解决的正确性问题，因此本轮也暂不启用 MTP。

官方扫描

官方 full-sweep-fail-fast 扫描已在提交 5180d246 上全部通过：运行 29717702856。

c1/c2/c3/c4/c5/c6/c8 七个 AgentX 扫描点均成功完成，所有已分析请求均无错误丢弃。已观测到的吞吐量最优点为 c3，总吞吐量 19,626 tok/s（817.77 tok/s/GPU）；c2 是延迟更优的运行点，总吞吐量 16,966 tok/s，平均 TTFT 为 2.88 秒。系统从 c4 开始进入饱和区。完整数据及中英文分析见结果评论。

验证

• python -m pytest utils/matrix_logic/ -v：在 5180d246 上 224 项通过
• 定向矩阵生成：准确生成 c1/c2/c3/c4/c5/c6/c8 七个 H200/Dynamo-SGLang/AgentX 任务
• 修改后的启动器和基准测试库均通过 bash -n
• 配方此前已通过 NVIDIA srt-slurm v1.0.10 schema 加载验证
• 分支已合并最新 main；Add GLM-5.2 full-context H200 AgentX with SGLang PCP and HiCache / 新增 GLM-5.2 H200 全上下文 AgentX SGLang PCP 与 HiCache 配置 #2275 的变更日志条目已追加到新的文件末尾

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[JordanNanos]

Unofficial run 29657445141 failed before Slurm submission: make setup refreshed the srt-slurm recipe tree after the local overlay, so CONFIG_FILE was missing. Commit b20b9b0c moves the overlay after setup. The SGLang image import completed successfully and is now cached; rerunning the same qualification.

中文：非正式运行 29657445141 在提交 Slurm 作业前失败：本地配方覆盖完成后，make setup 又刷新了 srt-slurm 配方目录，导致 CONFIG_FILE 丢失。提交 b20b9b0c 已将配方覆盖移动到 setup 之后。SGLang 镜像已成功导入并缓存，现重新运行同一资格验证。

[JordanNanos]

Unofficial run 2 (Actions run 29657842352) confirmed that the setup-order fix worked, then failed before Slurm submission because the launcher treated the srt-slurm destination (recipes/...) as the checked-in source path. Commit 7c138499 now maps it to benchmarks/multi_node/srt-slurm-recipes/..., overlays the normalized recipe after make setup, and preserves selector-aware paths. I am dispatching and monitoring run 3.\n\n中文：非正式运行 2（Actions 运行 29657842352）确认 setup 顺序修复已生效，但在提交 Slurm 作业前失败，原因是启动器误将 srt-slurm 目标路径（recipes/...）当作仓库内的源路径。提交 7c138499 已改为从 benchmarks/multi_node/srt-slurm-recipes/... 映射配方，在 make setup 后覆盖标准化路径，并兼容带选择器后缀的路径。现正触发并监控第 3 次运行。

[JordanNanos]

Unofficial run 3 (Actions run 29657920227) reached srtctl apply and validated the overlaid recipe, confirming the launcher fixes. Preflight then found that /models/GLM-5.2-FP8 is not staged on H200. Commit 433042d5 switches to srt-slurm v1.0.10 native hf:zai-org/GLM-5.2-FP8 loading and points prefill/decode at the persistent shared HF cache. Run 4 will use the feature branch as both the workflow ref and workload ref.\n\n中文：非正式运行 3（Actions 运行 29657920227）已进入 srtctl apply 并成功校验覆盖后的配方，说明启动器修复有效。随后预检发现 H200 上尚未预置 /models/GLM-5.2-FP8。提交 433042d5 已改用 srt-slurm v1.0.10 原生支持的 hf:zai-org/GLM-5.2-FP8 加载方式，并让预填充/解码节点使用持久化 HF 共享缓存。第 4 次运行会同时将工作流 ref 与负载 ref 指向功能分支。

[JordanNanos]

Unofficial run 4 (Actions run 29658013163) had correct feature-branch provenance, passed recipe/model preflight, and reached sbatch. Slurm rejected the GB300-derived cpus-per-task=144 request because the H200 nodes cannot satisfy it. Commit 89223d13 removes that override while retaining full-node memory allocation. Dispatching run 5 with both refs on the feature branch.\n\n中文：非正式运行 4（Actions 运行 29658013163）已具备正确的功能分支来源信息，通过配方与模型预检并进入 sbatch。由于 H200 节点无法满足源自 GB300 的 cpus-per-task=144 请求，Slurm 拒绝提交。提交 89223d13 已移除该覆盖配置，同时保留整节点内存分配。现正以功能分支作为双 ref 触发第 5 次运行。

[JordanNanos]

Unofficial run 5 (Actions run 29658079145, Slurm job 57417) successfully allocated four H200 nodes, started NATS/etcd/frontends, found the cached GLM-5.2 snapshot, and launched all three SGLang worker processes. Decode then exited because PD mode defaulted to chunk cache (disable_radix_cache), conflicting with HiCache. Commit d957349c enables SGLang’s disaggregation-decode-enable-radix-cache override and declares HF_HOME for the shared cache. Dispatching run 6.\n\n中文：非正式运行 5（Actions 运行 29658079145，Slurm 作业 57417）已成功分配 4 个 H200 节点，启动 NATS/etcd/前端，命中已缓存的 GLM-5.2 快照，并拉起全部 3 个 SGLang 工作进程。随后解码侧因 PD 模式默认使用 chunk cache（disable_radix_cache）而与 HiCache 冲突并退出。提交 d957349c 已启用 SGLang 的 disaggregation-decode-enable-radix-cache 覆盖配置，并为共享缓存补充 HF_HOME。现正触发第 6 次运行。

[JordanNanos]

R6 reached a healthy 1P/1D deployment and started AgentX, then the first warmup request crashed decode. The pinned SGLang build selected UnifiedRadixCache because SGLANG_ENABLE_UNIFIED_RADIX_TREE=1, but the disaggregated HiCache decode path calls query_storage_hit_length, which that class does not implement. Commit c48a286 removes the incompatible override from both workers so HiCache uses its supported radix-cache implementation. A corrected unofficial R7 will use the feature branch for both workflow refs.

中文：R6 已成功启动健康的 1P/1D 服务并进入 AgentX，但首个预热请求导致解码进程崩溃。由于设置了 SGLANG_ENABLE_UNIFIED_RADIX_TREE=1，当前固定版本的 SGLang 选择了 UnifiedRadixCache；而分离式 HiCache 解码路径会调用该类未实现的 query_storage_hit_length。提交 c48a286 已从预填充和解码 worker 中移除这一不兼容覆盖配置，使 HiCache 使用受支持的基数缓存实现。修正后的非正式 R7 将继续同时使用功能分支作为 workflow ref 和输入 ref。

[JordanNanos]

R7 confirmed the radix-cache fix: decode survived the first request and the run advanced into a long PCP8 prefill. The next failure occurred when prefill finished its layer-sharded cache work: direct HiCache attempted its default page_first_direct host layout and raised ValueError: Layer-sharded direct HiCache backup only supports layer_first layout. Commit e269af4 sets hicache-mem-layout: layer_first on the PCP prefill worker; decode keeps its supported default layout. R8 will again dispatch both refs from the feature branch.

中文：R7 已验证基数缓存修复有效：解码进程通过了首个请求，任务进入长上下文 PCP8 预填充阶段。随后在预填充完成分层分片缓存处理时出现新错误：direct HiCache 使用默认的 page_first_direct 主机内存布局，并报错 ValueError: Layer-sharded direct HiCache backup only supports layer_first layout。提交 e269af4 已为 PCP 预填充 worker 设置 hicache-mem-layout: layer_first，解码 worker 保持其受支持的默认布局。R8 仍将从功能分支同时派发两个 ref。

[JordanNanos]

R8 passed. The feature-branch workflow and workload completed successfully: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29660737166. The aggregate contains 62/62 successful profiled requests, zero error-dropped records, mean input length 430,096 tokens, p95 input length 497,847 tokens, 7,475.39 total tok/s (311.47 tok/s/GPU across 24 compute GPUs), mean TTFT 3.67 s, mean TPOT 19.50 ms, and a 99.46% theoretical cache-hit rate. This validates GLM-5.2 FP8 AgentX on H200 with PCP8 prefill, TP16 decode, Mooncake, and HiCache at the configured 1,048,576-token context limit.

中文：R8 已通过。功能分支上的 workflow 和 workload 均成功完成：https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29660737166。聚合结果包含 62/62 个成功的正式请求，错误丢弃记录为 0；平均输入长度为 430,096 tokens，P95 输入长度为 497,847 tokens；总吞吐量为 7,475.39 tok/s（24 张计算 GPU 上平均 311.47 tok/s/GPU），平均 TTFT 为 3.67 秒，平均 TPOT 为 19.50 毫秒，理论缓存命中率为 99.46%。该结果验证了 GLM-5.2 FP8 AgentX 在 H200 上采用 PCP8 预填充、TP16 解码、Mooncake 和 HiCache，并按 1,048,576-token 上下文上限配置运行。

[JordanNanos]

Expanded the H200 AgentX concurrency curve from [1] to seven points, split across two allocations:

• [1, 2, 3, 4]
• [5, 6, 8]

Each allocation stays at or below four one-hour points so benchmark execution plus trajectory draining remains within the recipe's 8-hour Slurm limit. All 224 matrix-logic tests pass.

Unofficial sweep: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29691471123

Both the workflow ref and workload ref are pinned to feat/glm52-h200-agentx-pcp-hicache at bc7bde76.

中文：已将 H200 AgentX 并发曲线从 [1] 扩展为七个数据点，并拆分至两个资源分配任务：

• [1, 2, 3, 4]
• [5, 6, 8]

每个任务最多包含四个一小时数据点，确保基准测试及轨迹排空过程能够在配方规定的 8 小时 Slurm 时限内完成。矩阵逻辑测试共 224 项，全部通过。

非正式扫描：https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29691471123

工作流 ref 与工作负载 ref 均固定为 feat/glm52-h200-agentx-pcp-hicache，提交为 bc7bde76。

[JordanNanos]

The first broad H200 sweep recovered valid c1 and c2 points, then exposed a decode-side HiCache concurrency bug: long-prefix restore failed on TP16 ranks and caused a Gloo collective-size mismatch. c3 and c5 both reproduced the failure.

Results recovered from run 29691471123:

• c1: 7,136.49 total tok/s (297.35 tok/s/GPU), 45.97 output tok/s, 3.63 s mean TTFT, 19.89 ms mean ITL, 60 successful requests, no recorded request errors.
• c2: 16,825.51 total tok/s (701.06 tok/s/GPU), 97.38 output tok/s, 2.62 s mean TTFT, 22.86 ms mean ITL, 266 successful requests, no recorded request errors.

Commit 07826595 now follows the repo's stable disaggregated DeepSeek-V4 pattern: HiCache remains enabled on PCP prefill, while TP16 decode keeps RadixCache without hierarchical-cache restore. Every concurrency point is isolated so one failure cannot mask later points. All 224 matrix tests pass.

Replacement unofficial sweep: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29698447013

Both workflow and workload refs are pinned to feat/glm52-h200-agentx-pcp-hicache.

中文：首轮 H200 扩展扫描获得了有效的 c1 和 c2 数据点，同时暴露了解码侧 HiCache 的并发缺陷：TP16 各 rank 在恢复长前缀时失败，并进一步触发 Gloo 集合通信消息长度不一致。c3 和 c5 均稳定复现该问题。

从运行 29691471123 恢复的结果：

• c1：总吞吐量 7,136.49 tok/s（297.35 tok/s/GPU），输出吞吐量 45.97 tok/s，平均 TTFT 3.63 秒，平均 ITL 19.89 毫秒；60 个请求成功，无已记录的请求错误。
• c2：总吞吐量 16,825.51 tok/s（701.06 tok/s/GPU），输出吞吐量 97.38 tok/s，平均 TTFT 2.62 秒，平均 ITL 22.86 毫秒；266 个请求成功，无已记录的请求错误。

提交 07826595 现已采用仓库中稳定的 DeepSeek-V4 分离式推理方案：PCP 预填充侧继续启用 HiCache，TP16 解码侧保留 RadixCache，但不执行分层缓存恢复。每个并发度使用独立资源分配任务，确保单点故障不会掩盖后续结果。矩阵测试共 224 项，全部通过。

替代非正式扫描：https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29698447013

工作流 ref 与工作负载 ref 均固定为 feat/glm52-h200-agentx-pcp-hicache。

[JordanNanos]

Replacement sweep R2 update: c1 completed successfully in https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29698447013.

• 63/63 profiled requests succeeded; 0 error-dropped records
• 7,552.69 total tok/s (314.70 tok/s/GPU across 24 compute GPUs)
• 47.07 output tok/s (1.96 tok/s/GPU)
• Mean TTFT 3.62 s; mean TPOT 19.28 ms; mean E2EL 55.42 s
• Mean input length 429,026 tokens; P95 497,735 tokens
• Theoretical cache-hit rate 99.46%

The remaining c2/c3/c4/c5/c6/c8 jobs are still running.

中文：替代扫描 R2 进展：c1 已在 https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29698447013 中成功完成。

• 63/63 个正式请求成功，错误丢弃记录为 0
• 总吞吐量 7,552.69 tok/s（24 张计算 GPU 上平均 314.70 tok/s/GPU）
• 输出吞吐量 47.07 tok/s（1.96 tok/s/GPU）
• 平均 TTFT 3.62 秒；平均 TPOT 19.28 毫秒；平均端到端延迟 55.42 秒
• 平均输入长度 429,026 tokens；P95 为 497,735 tokens
• 理论缓存命中率 99.46%

其余 c2/c3/c4/c5/c6/c8 任务仍在运行。

[JordanNanos]

Replacement sweep R2 update: c5 completed successfully—the same concurrency point that previously triggered the decode-side HiCache/Gloo failure.

• 189/189 profiled requests succeeded; 0 error-dropped records
• 12,745.95 total tok/s (531.08 tok/s/GPU across 24 compute GPUs)
• 77.92 output tok/s (3.25 tok/s/GPU)
• Mean TTFT 78.23 s; mean TPOT 23.80 ms; mean E2EL 112.76 s
• Mean input length 243,083 tokens; P95 564,346 tokens
• Theoretical cache-hit rate 98.06%

The engine is now stable at c5, although the sharp TTFT increase shows this topology is beyond its latency-efficient concurrency. c2/c3/c4/c6/c8 remain active.

中文：替代扫描 R2 进展：c5 已成功完成；该并发度正是此前触发解码侧 HiCache/Gloo 故障的数据点。

• 189/189 个正式请求成功，错误丢弃记录为 0
• 总吞吐量 12,745.95 tok/s（24 张计算 GPU 上平均 531.08 tok/s/GPU）
• 输出吞吐量 77.92 tok/s（3.25 tok/s/GPU）
• 平均 TTFT 78.23 秒；平均 TPOT 23.80 毫秒；平均端到端延迟 112.76 秒
• 平均输入长度 243,083 tokens；P95 为 564,346 tokens
• 理论缓存命中率 98.06%

引擎在 c5 已保持稳定，但 TTFT 大幅上升，说明该拓扑已超过延迟效率较高的并发区间。c2/c3/c4/c6/c8 仍在运行。

[JordanNanos]

Replacement sweep R2 update: c8 completed successfully.

• 372/372 profiled requests succeeded; 0 error-dropped records
• 11,953.69 total tok/s (498.07 tok/s/GPU across 24 compute GPUs)
• 79.00 output tok/s (3.29 tok/s/GPU)
• Mean TTFT 106.34 s; mean TPOT 24.26 ms; mean E2EL 123.72 s
• Theoretical cache-hit rate 97.66%

Throughput has fallen below c5 while TTFT exceeds 100 seconds, confirming that saturation occurs earlier in the curve. The corrected engine remains stable through the maximum tested concurrency. c2/c3/c4/c6 remain active.

中文：替代扫描 R2 进展：c8 已成功完成。

• 372/372 个正式请求成功，错误丢弃记录为 0
• 总吞吐量 11,953.69 tok/s（24 张计算 GPU 上平均 498.07 tok/s/GPU）
• 输出吞吐量 79.00 tok/s（3.29 tok/s/GPU）
• 平均 TTFT 106.34 秒；平均 TPOT 24.26 毫秒；平均端到端延迟 123.72 秒
• 理论缓存命中率 97.66%

总吞吐量已低于 c5，同时 TTFT 超过 100 秒，说明该拓扑在更低并发度即已达到饱和。修正后的引擎在最高测试并发度仍保持稳定。c2/c3/c4/c6 仍在运行。

[JordanNanos]

Replacement sweep R2 update: c4 completed successfully and is the highest-throughput point available so far.

• 408/408 profiled requests succeeded; 0 error-dropped records
• 18,647.29 total tok/s (776.97 tok/s/GPU across 24 compute GPUs)
• 117.47 output tok/s (4.89 tok/s/GPU)
• Mean TTFT 22.59 s; mean TPOT 24.64 ms; mean E2EL 47.39 s
• Theoretical cache-hit rate 97.76%

The current curve peaks at c4. c5 falls to 12,745.95 tok/s with 78.23 s mean TTFT, and c8 falls further to 11,953.69 tok/s with 106.34 s mean TTFT. c2/c3/c6 remain active.

中文：替代扫描 R2 进展：c4 已成功完成，目前是已获得数据点中总吞吐量最高的一点。

• 408/408 个正式请求成功，错误丢弃记录为 0
• 总吞吐量 18,647.29 tok/s（24 张计算 GPU 上平均 776.97 tok/s/GPU）
• 输出吞吐量 117.47 tok/s（4.89 tok/s/GPU）
• 平均 TTFT 22.59 秒；平均 TPOT 24.64 毫秒；平均端到端延迟 47.39 秒
• 理论缓存命中率 97.76%

当前曲线在 c4 达到峰值。c5 降至 12,745.95 tok/s，平均 TTFT 增至 78.23 秒；c8 进一步降至 11,953.69 tok/s，平均 TTFT 为 106.34 秒。c2/c3/c6 仍在运行。

[JordanNanos]

Replacement sweep R2 was cancelled at the workflow level at 23:26 UTC after c1/c4/c5/c8 completed successfully. The remaining c2/c3/c6 launch steps were cancelled within 51 seconds of one another; their cleanup and raw-log upload steps completed, but no usable benchmark artifacts were produced. This was not a model, HiCache, or benchmark-process failure.

Targeted feature-branch retries are now queued:

• c2: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29708342129
• c3: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29708350271
• c6: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29708357753

Each retry dispatches the workflow from feat/glm52-h200-agentx-pcp-hicache, tests the same ref, and generates only its missing concurrency point.

中文：替换扫描 R2 在 c1/c4/c5/c8 成功完成后，于 23:26 UTC 被工作流整体取消。剩余 c2/c3/c6 的启动步骤在 51 秒内先后被取消；清理和原始日志上传步骤均已完成，但没有生成可用的基准测试产物。这不是模型、HiCache 或基准测试进程本身的故障。

现已排队运行仅覆盖缺失点的 feature branch 定向重试：

• c2: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29708342129
• c3: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29708350271
• c6: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29708357753

每次重试均从 feat/glm52-h200-agentx-pcp-hicache 分支调度工作流，测试同一 ref，并且只生成对应的缺失并发点。

[JordanNanos]

Retry infrastructure update: all three targeted c2/c3/c6 workflows were cancelled before runner assignment during GitHub's active Actions incident. Each matrix job had runner_id=0, an empty runner name, and zero executed steps; therefore these failures provide no signal about GLM-5.2, SGLang, PCP, or HiCache.

GitHub's incident notice states that new workflows may fail to start and ongoing runs may fail:
https://www.githubstatus.com/incidents/8vfyvq16hzh9

The retries will remain paused until Actions is operational, then the same feature-branch-only c2/c3/c6 dispatches will be rerun.

中文：重试基础设施状态更新：在 GitHub Actions 事故期间，c2/c3/c6 三个定向重试工作流均在分配 runner 之前被取消。每个矩阵任务的 runner_id=0、runner 名称为空，并且没有执行任何步骤；因此这些失败不能用于判断 GLM-5.2、SGLang、PCP 或 HiCache 的运行情况。

GitHub 的事故公告明确说明，新工作流可能无法启动，正在运行的任务也可能失败：
https://www.githubstatus.com/incidents/8vfyvq16hzh9

在 Actions 恢复正常前暂停重试；恢复后将继续从同一 feature branch 重新调度仅包含 c2/c3/c6 的任务。

[claude]

Claude finished @JordanNanos's task in 5m 49s —— View job

---

Review

LGTM - no blocking issues found

• Gather context and diff against origin/main
• Review new srt-slurm recipe YAML
• Review configs/nvidia-master.yaml changes
• Review perf-changelog.yaml entry position
• Review runners/launch_h200-dgxc-slurm.sh

Verification details:

• configs/nvidia-master.yaml and perf-changelog.yaml are both updated, and the new changelog entry is correctly appended at the end of the file.
• The container image lmsysorg/sglang:nightly-dev-cu13-20260711-7de33ce8 is a public Docker Hub reference (same tag already used by existing entries), and the glm5.2 config-key prefix is already established on main.
• The new master-config entry matches the existing agentic-disagg structure (compared against dsv4-fp4-gb300-dynamo-sglang-agentic-disagg), pcp-size is a first-class matrix-logic field with test coverage, and the seven isolated concurrency points ([1, 2, 3, 4, 5, 6, 8]) are correctly split into separate search-space entries.
• The launcher changes are sound: the recipe overlay is selector-aware and applied after make setup, the enroot import -o $SQUASH_FILE docker://$IMAGE reproducibility pattern is present (flock-serialized), and the unconditional default_mounts block follows the existing launch_b300-nv.sh precedent with host paths already used by this launcher's single-node branch. WEKA_LOADER_OVERRIDE=semianalysis_cc_traces_weka_062126 is in the benchmark library's allowlist.

One non-blocking note: the recipe's health_check: max_attempts: 1440 never takes effect in CI — the launcher (lines 192–193) unconditionally rewrites it to max_attempts: 720. The bring-up runs all passed under the effective 2-hour window, so this only matters for standalone srtctl use or a cold HF cache.

I could not run python -m pytest utils/matrix_logic/ in this environment (command not permitted); relying on the reported 224-pass result and CI.

[JordanNanos]

Official full sweep started on commit 5180d246: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29717702856

• Trigger: PR full-sweep-fail-fast
• Provenance: feat/glm52-h200-agentx-pcp-hicache at 5180d246
• Changelog validation: passed
• Matrix: seven isolated H200 AgentX points at c1/c2/c3/c4/c5/c6/c8
• Current state: all seven jobs have assigned H200 runners and entered the multi-node launcher

中文：官方完整扫描已在提交 5180d246 上启动：https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29717702856

• 触发方式：PR full-sweep-fail-fast
• 来源：feat/glm52-h200-agentx-pcp-hicache 分支的 5180d246
• 变更日志验证：已通过
• 矩阵：c1/c2/c3/c4/c5/c6/c8 七个相互隔离的 H200 AgentX 扫描点
• 当前状态：七个任务均已分配 H200 runner，并进入多节点启动流程

[claude]

No bugs were found, but this is worth a human look: the PR modifies the shared H200 multinode launcher (launch_h200-dgxc-slurm.sh, used by 4 recipes on the h200-dgxc cluster) rather than being self-contained to the new GLM-5.2 recipe, and the official sweep is still incomplete (c2/c3/c6 pending after a GitHub Actions outage, per the timeline).

What was reviewed:

• New matrix entry glm5.2-fp8-h200-dynamo-sglang-agentic-pcp-hicache and its 7 isolated concurrency jobs, plus the checked-in recipe YAML's HiCache/PCP/decode settings.
• Launcher changes shared across H200 multinode recipes: CONFIG_FILE overlay path resolution, model-path hf: fallback, serialized squashfs import under flock, new unconditional default_mounts, and the health_check/tag rewrite now keyed off CONFIG_PATH instead of CONFIG_FILE.
• Checked the flock wait (1800s) against the on-demand import srun job's own 30-minute time limit, and the health_check override interaction with the recipe's 1440-attempt setting -- examined without confirming a bug, but flagged since this path is shared infrastructure.

Extended reasoning...

Overview

This PR adds a new three-node H200 AgentX qualification recipe for GLM-5.2 FP8 (SGLang DSA prefill with PCP8/LayerSplit, Mooncake KV transfer, TP16 decode, prefill-side HiCache) across four files: a new recipe YAML, a new nvidia-master.yaml matrix entry with 7 isolated concurrency jobs, a perf-changelog.yaml entry, and modifications to the shared runners/launch_h200-dgxc-slurm.sh launcher. Unlike most single-recipe additions, the launcher changes are not scoped to this recipe alone -- they touch code paths (CONFIG_FILE/CONFIG_PATH handling, squashfs import, srtslurm.yaml generation) that all H200 multinode recipes using cluster:h200-dgxc go through.

Security risks

None identified. This is CI/benchmark infrastructure -- no auth, crypto, or user-facing data paths. The shell interpolation in the launcher uses values from trusted matrix config, not external input.

Level of scrutiny

This warrants more than a mechanical-change level of review. The PR's own timeline documents eight-plus unofficial bring-up iterations fixing real correctness bugs (setup-order clobbering CONFIG_FILE, wrong recipe destination path, missing model staging, an incompatible Slurm cpus-per-task override, a radix-cache/HiCache incompatibility, and a HiCache memory-layout mismatch), which is a strong signal of real complexity in this integration. The shared-launcher edits (new default_mounts applied unconditionally to srtslurm.yaml, the health_check override precedence, and the flock-guarded on-demand squashfs import) affect other recipes on this cluster, not just this one. The bug-hunting system found no confirmable bugs, and its verifiers separately examined the flock-timeout margin and the health_check override precedence without confirming them as bugs -- but given the shared blast radius and the sweep still being incomplete (the PR's own checklist item requiring a green full sweep before reuse-merge is unchecked), a human maintainer familiar with the H200 cluster setup should confirm before merge.

Other factors

The PR author has been actively iterating in response to real CI failures, and matrix-logic tests (224) are reported passing. The PR's own checklist explicitly marks the "authorized maintainer has commented /reuse-sweep-run" item as unchecked, confirming the author does not consider this ready to merge yet -- consistent with deferring here rather than approving.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29717702856
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29717702856

[JordanNanos]

Official full sweep passed on commit 5180d246: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29717702856

All seven isolated GLM-5.2 H200 AgentX points completed successfully on 8 PCP-prefill + 16 TP-decode GPUs. Every point had zero error-dropped profiled requests.

Conc	Profiled	Total tok/s	Tok/s/GPU	Output tok/s	Mean TTFT	P95 TTFT	Mean TPOT	Mean E2EL	Theoretical cache hit
1	60	7,282.70	303.45	45.90	3.59s	4.44s	19.85ms	56.73s	99.459%
2	225	16,965.90	706.91	95.33	2.88s	5.00s	21.91ms	34.05s	98.828%
3	303	19,626.38	817.77	128.95	3.82s	10.14s	22.12ms	37.73s	98.444%
4	394	18,448.89	768.70	119.42	24.84s	108.52s	24.51ms	50.88s	98.001%
5	227	14,594.10	608.09	87.48	59.53s	131.75s	23.56ms	91.64s	98.786%
6	287	13,231.55	551.31	87.23	69.80s	151.43s	25.24ms	96.67s	97.851%
8	340	11,104.17	462.67	73.81	121.40s	232.12s	25.40ms	139.39s	97.499%

The throughput optimum is c3. c2 is the latency-efficient operating point. Saturation starts at c4: total throughput drops 6.0% from c3 while mean TTFT increases from 3.82s to 24.84s and P95 TTFT from 10.14s to 108.52s. Higher concurrency continues to reduce throughput and sharply increase queueing latency.

中文：官方完整扫描已在提交 5180d246 上全部通过：https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29717702856

七个相互隔离的 GLM-5.2 H200 AgentX 扫描点均成功完成，使用 8 张 GPU 执行 PCP 预填充、16 张 GPU 执行 TP 解码。所有扫描点的已分析请求均无错误丢弃。上表为完整结果。

吞吐量最优点为 c3，总吞吐量 19,626.38 tok/s（817.77 tok/s/GPU），输出吞吐量 128.95 tok/s。c2 是延迟更优的运行点，总吞吐量 16,965.90 tok/s，平均 TTFT 仅 2.88 秒。系统从 c4 开始进入饱和区：相比 c3，总吞吐量下降 6.0%，平均 TTFT 从 3.82 秒跃升至 24.84 秒，P95 TTFT 从 10.14 秒跃升至 108.52 秒；继续提高并发会进一步降低吞吐量并显著增加排队延迟。