Add GLM-5.2 full-context H200 AgentX with SGLang PCP and HiCache / 新增 GLM-5.2 H200 全上下文 AgentX SGLang PCP 与 HiCache 配置

[JordanNanos]

Description

Add a three-node H200 AgentX qualification path for GLM-5.2 FP8 at its native 1,048,576-token context:

• run one eight-rank SGLang DSA prefill worker as effective TP1 x PCP8 with LayerSplit;
• transfer KV through Mooncake to one TP16 decode worker across two H200 nodes;
• enable prefill-side HiCache with direct I/O, layer-first layout, and write-back;
• retain decode-side RadixCache without decode HiCache, avoiding the long-prefix restore correctness failure found during bring-up;
• replay the unfiltered AgentX trace as seven isolated jobs at concurrency 1, 2, 3, 4, 5, 6, and 8, so one failed point cannot discard completed curve data;
• extend the H200 srt-slurm launcher with the checked-in AgentX recipe, persistent dataset/model caches, and serialized image import.

DCP is intentionally not enabled or labeled. SGLang DSA decode-context parallelism for GLM-5.2 is not merged, so TP16 provides active-KV and runtime HBM headroom without claiming unsupported DCP behavior. MTP is also deferred while the GLM-5.2 MTP plus HiCache correctness issue remains open.

Bring-up evidence

Replacement run 29698447013 produced valid c1/c4/c5/c8 results before the workflow was externally cancelled. c4 was the observed knee at 18,647 total tok/s (777 tok/s/GPU). c2/c3/c6 are included in the official seven-point sweep requested on the current commit.

Validation

• python -m pytest utils/matrix_logic/ -v: 224 passed on 5180d246
• targeted matrix generation: exactly seven H200/Dynamo-SGLang/AgentX jobs at c1/c2/c3/c4/c5/c6/c8
• bash -n passes for the modified launcher and benchmark library
• recipe previously loaded successfully with NVIDIA srt-slurm v1.0.10 schema
• branch merged with current main; the Add GLM-5.2 full-context H200 AgentX with SGLang PCP and HiCache / 新增 GLM-5.2 H200 全上下文 AgentX SGLang PCP 与 HiCache 配置 #2275 changelog entry is appended at the new tail

Related Issue

None.

Type of Change

• Bug fix
• New feature
• Configuration change
• Documentation update
• Other

Checklist

• I have tested my changes locally
• Documentation updates are not required
• I have updated perf-changelog.yaml with this PR number
  • The new entry is appended at the end
• Before reuse merge, an authorized maintainer has commented /reuse-sweep-run only after a final green full sweep with evals passing

中文说明

为 GLM-5.2 FP8 新增三节点 H200 AgentX 资格验证路径，使用其原生 1,048,576 token 上下文：

• 单个八 rank SGLang DSA 预填充 worker 按有效 TP1 x PCP8 运行，并启用 LayerSplit；
• 通过 Mooncake 将 KV 传输到跨两台 H200 节点的单个 TP16 解码 worker；
• 预填充侧启用 HiCache，采用 direct I/O、layer-first 内存布局和 write-back 策略；
• 解码侧仅保留 RadixCache，不启用 HiCache，以规避 bring-up 阶段发现的长前缀恢复正确性故障；
• 使用并发 1、2、3、4、5、6、8 七个相互隔离的任务回放未截断的 AgentX 轨迹，避免单点失败导致已完成的曲线数据丢失；
• 扩展 H200 srt-slurm 启动器，支持仓库内 AgentX 配方、持久化数据集/模型缓存以及串行化镜像导入。

本 PR 不启用或标记 DCP。SGLang 针对 GLM-5.2 DSA 的解码上下文并行尚未合并，因此采用 TP16 为活跃 KV 和运行时显存预留空间，避免宣称尚未支持的 DCP 能力。GLM-5.2 的 MTP 与 HiCache 组合仍存在未解决的正确性问题，因此本轮也暂不启用 MTP。

Bring-up 结果

替换运行 29698447013 在工作流被外部取消前产出了有效的 c1/c4/c5/c8 结果。c4 为已观测到的性能拐点，总吞吐量为 18,647 tok/s（777 tok/s/GPU）。当前提交发起的官方七点扫描将补齐 c2/c3/c6。

验证

• python -m pytest utils/matrix_logic/ -v：在 5180d246 上 224 项通过
• 定向矩阵生成：准确生成 c1/c2/c3/c4/c5/c6/c8 七个 H200/Dynamo-SGLang/AgentX 任务
• 修改后的启动器和基准测试库均通过 bash -n
• 配方此前已通过 NVIDIA srt-slurm v1.0.10 schema 加载验证
• 分支已合并最新 main；Add GLM-5.2 full-context H200 AgentX with SGLang PCP and HiCache / 新增 GLM-5.2 H200 全上下文 AgentX SGLang PCP 与 HiCache 配置 #2275 的变更日志条目已追加到新的文件末尾

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[JordanNanos]

Unofficial run 29657445141 failed before Slurm submission: make setup refreshed the srt-slurm recipe tree after the local overlay, so CONFIG_FILE was missing. Commit b20b9b0c moves the overlay after setup. The SGLang image import completed successfully and is now cached; rerunning the same qualification.

中文：非正式运行 29657445141 在提交 Slurm 作业前失败：本地配方覆盖完成后，make setup 又刷新了 srt-slurm 配方目录，导致 CONFIG_FILE 丢失。提交 b20b9b0c 已将配方覆盖移动到 setup 之后。SGLang 镜像已成功导入并缓存，现重新运行同一资格验证。

[JordanNanos]

Unofficial run 2 (Actions run 29657842352) confirmed that the setup-order fix worked, then failed before Slurm submission because the launcher treated the srt-slurm destination (recipes/...) as the checked-in source path. Commit 7c138499 now maps it to benchmarks/multi_node/srt-slurm-recipes/..., overlays the normalized recipe after make setup, and preserves selector-aware paths. I am dispatching and monitoring run 3.\n\n中文：非正式运行 2（Actions 运行 29657842352）确认 setup 顺序修复已生效，但在提交 Slurm 作业前失败，原因是启动器误将 srt-slurm 目标路径（recipes/...）当作仓库内的源路径。提交 7c138499 已改为从 benchmarks/multi_node/srt-slurm-recipes/... 映射配方，在 make setup 后覆盖标准化路径，并兼容带选择器后缀的路径。现正触发并监控第 3 次运行。

[JordanNanos]

Unofficial run 3 (Actions run 29657920227) reached srtctl apply and validated the overlaid recipe, confirming the launcher fixes. Preflight then found that /models/GLM-5.2-FP8 is not staged on H200. Commit 433042d5 switches to srt-slurm v1.0.10 native hf:zai-org/GLM-5.2-FP8 loading and points prefill/decode at the persistent shared HF cache. Run 4 will use the feature branch as both the workflow ref and workload ref.\n\n中文：非正式运行 3（Actions 运行 29657920227）已进入 srtctl apply 并成功校验覆盖后的配方，说明启动器修复有效。随后预检发现 H200 上尚未预置 /models/GLM-5.2-FP8。提交 433042d5 已改用 srt-slurm v1.0.10 原生支持的 hf:zai-org/GLM-5.2-FP8 加载方式，并让预填充/解码节点使用持久化 HF 共享缓存。第 4 次运行会同时将工作流 ref 与负载 ref 指向功能分支。

[JordanNanos]

Unofficial run 4 (Actions run 29658013163) had correct feature-branch provenance, passed recipe/model preflight, and reached sbatch. Slurm rejected the GB300-derived cpus-per-task=144 request because the H200 nodes cannot satisfy it. Commit 89223d13 removes that override while retaining full-node memory allocation. Dispatching run 5 with both refs on the feature branch.\n\n中文：非正式运行 4（Actions 运行 29658013163）已具备正确的功能分支来源信息，通过配方与模型预检并进入 sbatch。由于 H200 节点无法满足源自 GB300 的 cpus-per-task=144 请求，Slurm 拒绝提交。提交 89223d13 已移除该覆盖配置，同时保留整节点内存分配。现正以功能分支作为双 ref 触发第 5 次运行。

[JordanNanos]

Unofficial run 5 (Actions run 29658079145, Slurm job 57417) successfully allocated four H200 nodes, started NATS/etcd/frontends, found the cached GLM-5.2 snapshot, and launched all three SGLang worker processes. Decode then exited because PD mode defaulted to chunk cache (disable_radix_cache), conflicting with HiCache. Commit d957349c enables SGLang’s disaggregation-decode-enable-radix-cache override and declares HF_HOME for the shared cache. Dispatching run 6.\n\n中文：非正式运行 5（Actions 运行 29658079145，Slurm 作业 57417）已成功分配 4 个 H200 节点，启动 NATS/etcd/前端，命中已缓存的 GLM-5.2 快照，并拉起全部 3 个 SGLang 工作进程。随后解码侧因 PD 模式默认使用 chunk cache（disable_radix_cache）而与 HiCache 冲突并退出。提交 d957349c 已启用 SGLang 的 disaggregation-decode-enable-radix-cache 覆盖配置，并为共享缓存补充 HF_HOME。现正触发第 6 次运行。

[JordanNanos]

R6 reached a healthy 1P/1D deployment and started AgentX, then the first warmup request crashed decode. The pinned SGLang build selected UnifiedRadixCache because SGLANG_ENABLE_UNIFIED_RADIX_TREE=1, but the disaggregated HiCache decode path calls query_storage_hit_length, which that class does not implement. Commit c48a286 removes the incompatible override from both workers so HiCache uses its supported radix-cache implementation. A corrected unofficial R7 will use the feature branch for both workflow refs.

中文：R6 已成功启动健康的 1P/1D 服务并进入 AgentX，但首个预热请求导致解码进程崩溃。由于设置了 SGLANG_ENABLE_UNIFIED_RADIX_TREE=1，当前固定版本的 SGLang 选择了 UnifiedRadixCache；而分离式 HiCache 解码路径会调用该类未实现的 query_storage_hit_length。提交 c48a286 已从预填充和解码 worker 中移除这一不兼容覆盖配置，使 HiCache 使用受支持的基数缓存实现。修正后的非正式 R7 将继续同时使用功能分支作为 workflow ref 和输入 ref。

[JordanNanos]

R7 confirmed the radix-cache fix: decode survived the first request and the run advanced into a long PCP8 prefill. The next failure occurred when prefill finished its layer-sharded cache work: direct HiCache attempted its default page_first_direct host layout and raised ValueError: Layer-sharded direct HiCache backup only supports layer_first layout. Commit e269af4 sets hicache-mem-layout: layer_first on the PCP prefill worker; decode keeps its supported default layout. R8 will again dispatch both refs from the feature branch.

中文：R7 已验证基数缓存修复有效：解码进程通过了首个请求，任务进入长上下文 PCP8 预填充阶段。随后在预填充完成分层分片缓存处理时出现新错误：direct HiCache 使用默认的 page_first_direct 主机内存布局，并报错 ValueError: Layer-sharded direct HiCache backup only supports layer_first layout。提交 e269af4 已为 PCP 预填充 worker 设置 hicache-mem-layout: layer_first，解码 worker 保持其受支持的默认布局。R8 仍将从功能分支同时派发两个 ref。

[JordanNanos]

R8 passed. The feature-branch workflow and workload completed successfully: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29660737166. The aggregate contains 62/62 successful profiled requests, zero error-dropped records, mean input length 430,096 tokens, p95 input length 497,847 tokens, 7,475.39 total tok/s (311.47 tok/s/GPU across 24 compute GPUs), mean TTFT 3.67 s, mean TPOT 19.50 ms, and a 99.46% theoretical cache-hit rate. This validates GLM-5.2 FP8 AgentX on H200 with PCP8 prefill, TP16 decode, Mooncake, and HiCache at the configured 1,048,576-token context limit.

中文：R8 已通过。功能分支上的 workflow 和 workload 均成功完成：https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29660737166。聚合结果包含 62/62 个成功的正式请求，错误丢弃记录为 0；平均输入长度为 430,096 tokens，P95 输入长度为 497,847 tokens；总吞吐量为 7,475.39 tok/s（24 张计算 GPU 上平均 311.47 tok/s/GPU），平均 TTFT 为 3.67 秒，平均 TPOT 为 19.50 毫秒，理论缓存命中率为 99.46%。该结果验证了 GLM-5.2 FP8 AgentX 在 H200 上采用 PCP8 预填充、TP16 解码、Mooncake 和 HiCache，并按 1,048,576-token 上下文上限配置运行。

[JordanNanos]

Expanded the H200 AgentX concurrency curve from [1] to seven points, split across two allocations:

• [1, 2, 3, 4]
• [5, 6, 8]

Each allocation stays at or below four one-hour points so benchmark execution plus trajectory draining remains within the recipe's 8-hour Slurm limit. All 224 matrix-logic tests pass.

Unofficial sweep: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29691471123

Both the workflow ref and workload ref are pinned to feat/glm52-h200-agentx-pcp-hicache at bc7bde76.

中文：已将 H200 AgentX 并发曲线从 [1] 扩展为七个数据点，并拆分至两个资源分配任务：

• [1, 2, 3, 4]
• [5, 6, 8]

每个任务最多包含四个一小时数据点，确保基准测试及轨迹排空过程能够在配方规定的 8 小时 Slurm 时限内完成。矩阵逻辑测试共 224 项，全部通过。

非正式扫描：https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29691471123

工作流 ref 与工作负载 ref 均固定为 feat/glm52-h200-agentx-pcp-hicache，提交为 bc7bde76。

[JordanNanos]

The first broad H200 sweep recovered valid c1 and c2 points, then exposed a decode-side HiCache concurrency bug: long-prefix restore failed on TP16 ranks and caused a Gloo collective-size mismatch. c3 and c5 both reproduced the failure.

Results recovered from run 29691471123:

• c1: 7,136.49 total tok/s (297.35 tok/s/GPU), 45.97 output tok/s, 3.63 s mean TTFT, 19.89 ms mean ITL, 60 successful requests, no recorded request errors.
• c2: 16,825.51 total tok/s (701.06 tok/s/GPU), 97.38 output tok/s, 2.62 s mean TTFT, 22.86 ms mean ITL, 266 successful requests, no recorded request errors.

Commit 07826595 now follows the repo's stable disaggregated DeepSeek-V4 pattern: HiCache remains enabled on PCP prefill, while TP16 decode keeps RadixCache without hierarchical-cache restore. Every concurrency point is isolated so one failure cannot mask later points. All 224 matrix tests pass.

Replacement unofficial sweep: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29698447013

Both workflow and workload refs are pinned to feat/glm52-h200-agentx-pcp-hicache.

中文：首轮 H200 扩展扫描获得了有效的 c1 和 c2 数据点，同时暴露了解码侧 HiCache 的并发缺陷：TP16 各 rank 在恢复长前缀时失败，并进一步触发 Gloo 集合通信消息长度不一致。c3 和 c5 均稳定复现该问题。

从运行 29691471123 恢复的结果：

• c1：总吞吐量 7,136.49 tok/s（297.35 tok/s/GPU），输出吞吐量 45.97 tok/s，平均 TTFT 3.63 秒，平均 ITL 19.89 毫秒；60 个请求成功，无已记录的请求错误。
• c2：总吞吐量 16,825.51 tok/s（701.06 tok/s/GPU），输出吞吐量 97.38 tok/s，平均 TTFT 2.62 秒，平均 ITL 22.86 毫秒；266 个请求成功，无已记录的请求错误。

提交 07826595 现已采用仓库中稳定的 DeepSeek-V4 分离式推理方案：PCP 预填充侧继续启用 HiCache，TP16 解码侧保留 RadixCache，但不执行分层缓存恢复。每个并发度使用独立资源分配任务，确保单点故障不会掩盖后续结果。矩阵测试共 224 项，全部通过。

替代非正式扫描：https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29698447013

工作流 ref 与工作负载 ref 均固定为 feat/glm52-h200-agentx-pcp-hicache。

[JordanNanos]

Replacement sweep R2 update: c1 completed successfully in https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29698447013.

• 63/63 profiled requests succeeded; 0 error-dropped records
• 7,552.69 total tok/s (314.70 tok/s/GPU across 24 compute GPUs)
• 47.07 output tok/s (1.96 tok/s/GPU)
• Mean TTFT 3.62 s; mean TPOT 19.28 ms; mean E2EL 55.42 s
• Mean input length 429,026 tokens; P95 497,735 tokens
• Theoretical cache-hit rate 99.46%

The remaining c2/c3/c4/c5/c6/c8 jobs are still running.

中文：替代扫描 R2 进展：c1 已在 https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29698447013 中成功完成。

• 63/63 个正式请求成功，错误丢弃记录为 0
• 总吞吐量 7,552.69 tok/s（24 张计算 GPU 上平均 314.70 tok/s/GPU）
• 输出吞吐量 47.07 tok/s（1.96 tok/s/GPU）
• 平均 TTFT 3.62 秒；平均 TPOT 19.28 毫秒；平均端到端延迟 55.42 秒
• 平均输入长度 429,026 tokens；P95 为 497,735 tokens
• 理论缓存命中率 99.46%

其余 c2/c3/c4/c5/c6/c8 任务仍在运行。

[JordanNanos]

Replacement sweep R2 update: c5 completed successfully—the same concurrency point that previously triggered the decode-side HiCache/Gloo failure.

• 189/189 profiled requests succeeded; 0 error-dropped records
• 12,745.95 total tok/s (531.08 tok/s/GPU across 24 compute GPUs)
• 77.92 output tok/s (3.25 tok/s/GPU)
• Mean TTFT 78.23 s; mean TPOT 23.80 ms; mean E2EL 112.76 s
• Mean input length 243,083 tokens; P95 564,346 tokens
• Theoretical cache-hit rate 98.06%

The engine is now stable at c5, although the sharp TTFT increase shows this topology is beyond its latency-efficient concurrency. c2/c3/c4/c6/c8 remain active.

中文：替代扫描 R2 进展：c5 已成功完成；该并发度正是此前触发解码侧 HiCache/Gloo 故障的数据点。

• 189/189 个正式请求成功，错误丢弃记录为 0
• 总吞吐量 12,745.95 tok/s（24 张计算 GPU 上平均 531.08 tok/s/GPU）
• 输出吞吐量 77.92 tok/s（3.25 tok/s/GPU）
• 平均 TTFT 78.23 秒；平均 TPOT 23.80 毫秒；平均端到端延迟 112.76 秒
• 平均输入长度 243,083 tokens；P95 为 564,346 tokens
• 理论缓存命中率 98.06%

引擎在 c5 已保持稳定，但 TTFT 大幅上升，说明该拓扑已超过延迟效率较高的并发区间。c2/c3/c4/c6/c8 仍在运行。

[JordanNanos]

Replacement sweep R2 update: c8 completed successfully.

• 372/372 profiled requests succeeded; 0 error-dropped records
• 11,953.69 total tok/s (498.07 tok/s/GPU across 24 compute GPUs)
• 79.00 output tok/s (3.29 tok/s/GPU)
• Mean TTFT 106.34 s; mean TPOT 24.26 ms; mean E2EL 123.72 s
• Theoretical cache-hit rate 97.66%

Throughput has fallen below c5 while TTFT exceeds 100 seconds, confirming that saturation occurs earlier in the curve. The corrected engine remains stable through the maximum tested concurrency. c2/c3/c4/c6 remain active.

中文：替代扫描 R2 进展：c8 已成功完成。

• 372/372 个正式请求成功，错误丢弃记录为 0
• 总吞吐量 11,953.69 tok/s（24 张计算 GPU 上平均 498.07 tok/s/GPU）
• 输出吞吐量 79.00 tok/s（3.29 tok/s/GPU）
• 平均 TTFT 106.34 秒；平均 TPOT 24.26 毫秒；平均端到端延迟 123.72 秒
• 理论缓存命中率 97.66%

总吞吐量已低于 c5，同时 TTFT 超过 100 秒，说明该拓扑在更低并发度即已达到饱和。修正后的引擎在最高测试并发度仍保持稳定。c2/c3/c4/c6 仍在运行。

[JordanNanos]

Replacement sweep R2 update: c4 completed successfully and is the highest-throughput point available so far.

• 408/408 profiled requests succeeded; 0 error-dropped records
• 18,647.29 total tok/s (776.97 tok/s/GPU across 24 compute GPUs)
• 117.47 output tok/s (4.89 tok/s/GPU)
• Mean TTFT 22.59 s; mean TPOT 24.64 ms; mean E2EL 47.39 s
• Theoretical cache-hit rate 97.76%

The current curve peaks at c4. c5 falls to 12,745.95 tok/s with 78.23 s mean TTFT, and c8 falls further to 11,953.69 tok/s with 106.34 s mean TTFT. c2/c3/c6 remain active.

中文：替代扫描 R2 进展：c4 已成功完成，目前是已获得数据点中总吞吐量最高的一点。

• 408/408 个正式请求成功，错误丢弃记录为 0
• 总吞吐量 18,647.29 tok/s（24 张计算 GPU 上平均 776.97 tok/s/GPU）
• 输出吞吐量 117.47 tok/s（4.89 tok/s/GPU）
• 平均 TTFT 22.59 秒；平均 TPOT 24.64 毫秒；平均端到端延迟 47.39 秒
• 理论缓存命中率 97.76%

当前曲线在 c4 达到峰值。c5 降至 12,745.95 tok/s，平均 TTFT 增至 78.23 秒；c8 进一步降至 11,953.69 tok/s，平均 TTFT 为 106.34 秒。c2/c3/c6 仍在运行。

[JordanNanos]

Replacement sweep R2 was cancelled at the workflow level at 23:26 UTC after c1/c4/c5/c8 completed successfully. The remaining c2/c3/c6 launch steps were cancelled within 51 seconds of one another; their cleanup and raw-log upload steps completed, but no usable benchmark artifacts were produced. This was not a model, HiCache, or benchmark-process failure.

Targeted feature-branch retries are now queued:

• c2: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29708342129
• c3: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29708350271
• c6: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29708357753

Each retry dispatches the workflow from feat/glm52-h200-agentx-pcp-hicache, tests the same ref, and generates only its missing concurrency point.

中文：替换扫描 R2 在 c1/c4/c5/c8 成功完成后，于 23:26 UTC 被工作流整体取消。剩余 c2/c3/c6 的启动步骤在 51 秒内先后被取消；清理和原始日志上传步骤均已完成，但没有生成可用的基准测试产物。这不是模型、HiCache 或基准测试进程本身的故障。

现已排队运行仅覆盖缺失点的 feature branch 定向重试：

• c2: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29708342129
• c3: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29708350271
• c6: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29708357753

每次重试均从 feat/glm52-h200-agentx-pcp-hicache 分支调度工作流，测试同一 ref，并且只生成对应的缺失并发点。

[JordanNanos]

Retry infrastructure update: all three targeted c2/c3/c6 workflows were cancelled before runner assignment during GitHub's active Actions incident. Each matrix job had runner_id=0, an empty runner name, and zero executed steps; therefore these failures provide no signal about GLM-5.2, SGLang, PCP, or HiCache.

GitHub's incident notice states that new workflows may fail to start and ongoing runs may fail:
https://www.githubstatus.com/incidents/8vfyvq16hzh9

The retries will remain paused until Actions is operational, then the same feature-branch-only c2/c3/c6 dispatches will be rerun.

中文：重试基础设施状态更新：在 GitHub Actions 事故期间，c2/c3/c6 三个定向重试工作流均在分配 runner 之前被取消。每个矩阵任务的 runner_id=0、runner 名称为空，并且没有执行任何步骤；因此这些失败不能用于判断 GLM-5.2、SGLang、PCP 或 HiCache 的运行情况。

GitHub 的事故公告明确说明，新工作流可能无法启动，正在运行的任务也可能失败：
https://www.githubstatus.com/incidents/8vfyvq16hzh9

在 Actions 恢复正常前暂停重试；恢复后将继续从同一 feature branch 重新调度仅包含 c2/c3/c6 的任务。

[claude]

Claude finished @JordanNanos's task in 5m 49s —— View job

---

Review

LGTM - no blocking issues found

• Gather context and diff against origin/main
• Review new srt-slurm recipe YAML
• Review configs/nvidia-master.yaml changes
• Review perf-changelog.yaml entry position
• Review runners/launch_h200-dgxc-slurm.sh

Verification details:

• configs/nvidia-master.yaml and perf-changelog.yaml are both updated, and the new changelog entry is correctly appended at the end of the file.
• The container image lmsysorg/sglang:nightly-dev-cu13-20260711-7de33ce8 is a public Docker Hub reference (same tag already used by existing entries), and the glm5.2 config-key prefix is already established on main.
• The new master-config entry matches the existing agentic-disagg structure (compared against dsv4-fp4-gb300-dynamo-sglang-agentic-disagg), pcp-size is a first-class matrix-logic field with test coverage, and the seven isolated concurrency points ([1, 2, 3, 4, 5, 6, 8]) are correctly split into separate search-space entries.
• The launcher changes are sound: the recipe overlay is selector-aware and applied after make setup, the enroot import -o $SQUASH_FILE docker://$IMAGE reproducibility pattern is present (flock-serialized), and the unconditional default_mounts block follows the existing launch_b300-nv.sh precedent with host paths already used by this launcher's single-node branch. WEKA_LOADER_OVERRIDE=semianalysis_cc_traces_weka_062126 is in the benchmark library's allowlist.

One non-blocking note: the recipe's health_check: max_attempts: 1440 never takes effect in CI — the launcher (lines 192–193) unconditionally rewrites it to max_attempts: 720. The bring-up runs all passed under the effective 2-hour window, so this only matters for standalone srtctl use or a cold HF cache.

I could not run python -m pytest utils/matrix_logic/ in this environment (command not permitted); relying on the reported 224-pass result and CI.

[JordanNanos]

Official full sweep started on commit 5180d246: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29717702856

• Trigger: PR full-sweep-fail-fast
• Provenance: feat/glm52-h200-agentx-pcp-hicache at 5180d246
• Changelog validation: passed
• Matrix: seven isolated H200 AgentX points at c1/c2/c3/c4/c5/c6/c8
• Current state: all seven jobs have assigned H200 runners and entered the multi-node launcher

中文：官方完整扫描已在提交 5180d246 上启动：https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29717702856

• 触发方式：PR full-sweep-fail-fast
• 来源：feat/glm52-h200-agentx-pcp-hicache 分支的 5180d246
• 变更日志验证：已通过
• 矩阵：c1/c2/c3/c4/c5/c6/c8 七个相互隔离的 H200 AgentX 扫描点
• 当前状态：七个任务均已分配 H200 runner，并进入多节点启动流程

[claude]

No bugs were found, but this is worth a human look: the PR modifies the shared H200 multinode launcher (launch_h200-dgxc-slurm.sh, used by 4 recipes on the h200-dgxc cluster) rather than being self-contained to the new GLM-5.2 recipe, and the official sweep is still incomplete (c2/c3/c6 pending after a GitHub Actions outage, per the timeline).

What was reviewed:

• New matrix entry glm5.2-fp8-h200-dynamo-sglang-agentic-pcp-hicache and its 7 isolated concurrency jobs, plus the checked-in recipe YAML's HiCache/PCP/decode settings.
• Launcher changes shared across H200 multinode recipes: CONFIG_FILE overlay path resolution, model-path hf: fallback, serialized squashfs import under flock, new unconditional default_mounts, and the health_check/tag rewrite now keyed off CONFIG_PATH instead of CONFIG_FILE.
• Checked the flock wait (1800s) against the on-demand import srun job's own 30-minute time limit, and the health_check override interaction with the recipe's 1440-attempt setting -- examined without confirming a bug, but flagged since this path is shared infrastructure.

Extended reasoning...

Overview

This PR adds a new three-node H200 AgentX qualification recipe for GLM-5.2 FP8 (SGLang DSA prefill with PCP8/LayerSplit, Mooncake KV transfer, TP16 decode, prefill-side HiCache) across four files: a new recipe YAML, a new nvidia-master.yaml matrix entry with 7 isolated concurrency jobs, a perf-changelog.yaml entry, and modifications to the shared runners/launch_h200-dgxc-slurm.sh launcher. Unlike most single-recipe additions, the launcher changes are not scoped to this recipe alone -- they touch code paths (CONFIG_FILE/CONFIG_PATH handling, squashfs import, srtslurm.yaml generation) that all H200 multinode recipes using cluster:h200-dgxc go through.

Security risks

None identified. This is CI/benchmark infrastructure -- no auth, crypto, or user-facing data paths. The shell interpolation in the launcher uses values from trusted matrix config, not external input.

Level of scrutiny

This warrants more than a mechanical-change level of review. The PR's own timeline documents eight-plus unofficial bring-up iterations fixing real correctness bugs (setup-order clobbering CONFIG_FILE, wrong recipe destination path, missing model staging, an incompatible Slurm cpus-per-task override, a radix-cache/HiCache incompatibility, and a HiCache memory-layout mismatch), which is a strong signal of real complexity in this integration. The shared-launcher edits (new default_mounts applied unconditionally to srtslurm.yaml, the health_check override precedence, and the flock-guarded on-demand squashfs import) affect other recipes on this cluster, not just this one. The bug-hunting system found no confirmable bugs, and its verifiers separately examined the flock-timeout margin and the health_check override precedence without confirming them as bugs -- but given the shared blast radius and the sweep still being incomplete (the PR's own checklist item requiring a green full sweep before reuse-merge is unchecked), a human maintainer familiar with the H200 cluster setup should confirm before merge.

Other factors

The PR author has been actively iterating in response to real CI failures, and matrix-logic tests (224) are reported passing. The PR's own checklist explicitly marks the "authorized maintainer has commented /reuse-sweep-run" item as unchecked, confirming the author does not consider this ready to merge yet -- consistent with deferring here rather than approving.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29717702856
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29717702856