feat(agentic): add GLM-5.2 MI325X full-context sweeps / 新增 GLM-5.2 MI325X 全上下文 AgentX 扫描#2273
feat(agentic): add GLM-5.2 MI325X full-context sweeps / 新增 GLM-5.2 MI325X 全上下文 AgentX 扫描#2273JordanNanos wants to merge 35 commits into
Conversation
Add a concurrency-1 TP8 SGLang AgentX configuration that explicitly allocates the native 1,048,576-token BF16 KV pool on MI325X and replays the unfiltered corpus. 中文：新增 GLM-5.2 MI325X TP8 SGLang AgentX 并发 1 验证配置，显式分配原生 1,048,576-token BF16 KV 缓存池，并回放未过滤的完整语料。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
3 similar comments
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Deferring — the inline finding (wrong --dsa-prefill/decode-backend flag name, should be --nsa-*) breaks server startup for this recipe's sole job, so a human should review before merge.
Extended reasoning...
This PR adds a single new benchmark recipe (config + shell script + changelog entry) for GLM-5.2 FP8 full-context AgentX qualification on MI325X. It follows the same structural pattern as many existing agentic recipes in the repo, and the changes to configs/amd-master.yaml and perf-changelog.yaml are simple, additive, and low risk in isolation.
However, the bug hunting system found a concrete, high-confidence defect in the new shell script: it passes --dsa-prefill-backend/--dsa-decode-backend to sglang.launch_server, but every other GLM-5.x recipe in this repo (fp4/fp8 variants across b200, b300, mi355x, mi325x, and multi-node slurm YAMLs) uses --nsa-prefill-backend/--nsa-decode-backend for the identical TileLang backend. I independently confirmed this via a repo-wide grep — there is no other --dsa-* usage anywhere. Since sglang parses args with argparse, this typo would cause the server process to exit immediately, so the job (the entire point of this PR) would fail on first run.
There are no security-sensitive concerns here (no auth, crypto, or permission logic touched), but this is exactly the kind of functional bug that warrants a human look before merge rather than an automated approval, since the PR as-is would not achieve its stated purpose.
| --dsa-prefill-backend tilelang | ||
| --dsa-decode-backend tilelang |
There was a problem hiding this comment.
🔴 This recipe launches sglang with --dsa-prefill-backend tilelang / --dsa-decode-backend tilelang (lines 68-69), but SGLang's actual CLI flag for this sparse-attention backend is --nsa-* (Native Sparse Attention), not --dsa-* — every other GLM-5.x sglang recipe in the repo uses --nsa-prefill-backend/--nsa-decode-backend for the identical TileLang backend. Since sglang.launch_server uses argparse, this unrecognized flag will make the server process exit immediately on startup, causing wait_for_server_ready to time out and the whole qualification job to fail.
Extended reasoning...
The bug: benchmarks/single_node/agentic/glm5.2_fp8_mi325x.sh builds SGLANG_CMD with:
--dsa-prefill-backend tilelang
--dsa-decode-backend tilelang
SGLang does not expose a --dsa-prefill-backend/--dsa-decode-backend CLI flag anywhere in this repo's usage. The correct flag family, used consistently across every other GLM-5.x sglang recipe (glm5_fp8_mi325x.sh, glm5_fp8_mi355x.sh, the _mtp variants, glm5.1_fp4_mi355x.sh, and the multi-node GLM-5.1 slurm YAMLs on gb300), is --nsa-prefill-backend / --nsa-decode-backend (NSA = Native Sparse Attention, SGLang's actual server-arg name for this sparse-attention backend selector).
Why the author made this mistake: GLM-5.2 (like GLM-5/5.1) is internally architected as GlmMoeDsaForCausalLM — 'DSA' is the informal architecture/model-class name used in ZAI's docs and in this script's own comment ('the DSA TileLang backends'). The author appears to have named the CLI flag after the architecture rather than checking SGLang's actual exposed argument, which remains nsa regardless of the underlying model naming.
Why nothing catches this before merge: The PR's validation section only runs bash -n (a syntax check) and a local matrix-generation dry run (pytest utils/matrix_logic/). Neither exercises the actual sglang server process, so an argparse-level 'unrecognized arguments' failure is invisible until the real GPU job runs.
Step-by-step proof of impact:
- The generated job invokes this script, which builds and execs
SGLANG_CMD, including--dsa-prefill-backend tilelang --dsa-decode-backend tilelang. python3 -m sglang.launch_serverparses argv withargparse.--dsa-prefill-backendis not a registered option and is not a prefix of any registered option (in particular, it is not a prefix of--nsa-prefill-backend, since they differ at the very first differentiating character group), so argparse cannot fuzzy-match it.- argparse aborts immediately with
error: unrecognized arguments: --dsa-prefill-backend tilelang --dsa-decode-backend tilelangand a non-zero exit code, before the server ever binds to--port. - The script backgrounds the process (
"${SGLANG_CMD[@]}" > "$SERVER_LOG" 2>&1 &) and then callswait_for_server_ready --port "$PORT" ... --server-pid "$SERVER_PID", which polls for the port to come up. Since the process already exited, this polling times out (or detects the dead PID) and the script fails. - The entire GLM-5.2 MI325X full-context qualification job — the sole purpose of this PR — fails on every run, before a single token is ever served.
The fix is a one-line-times-two rename: --dsa-prefill-backend → --nsa-prefill-backend and --dsa-decode-backend → --nsa-decode-backend, matching every sibling GLM-5.x sglang recipe in the repo (e.g. benchmarks/single_node/fixed_seq_len/glm5_fp8_mi325x.sh uses this exact pair for the same MI325X SKU and the same TileLang backend).
Avoid the gfx942 sgl-kernel JIT failure caused by its CUDA cooperative_groups.h include by selecting the Torch DSA top-k fallback and disabling fused top-k. 中文：MI30X 镜像的 sgl-kernel JIT 在 gfx942 编译时错误引用 CUDA cooperative_groups.h；改用 Torch DSA top-k 回退实现并关闭融合 top-k，避免启动失败。
Disable the independently gated topk_v2 CUDA-graph planner so the configured Torch DSA top-k fallback is honored on gfx942. 中文：在 MI325X 上禁用独立控制的 topk_v2 CUDA Graph 规划内核，确保 gfx942 使用已配置的 Torch DSA top-k 回退路径。
|
The unofficial MI325X full-context qualification passed: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29654139122
This validates the native 1M server/KV operating point on MI325X. The one-hour trace did not include a single 1M-token request, so it is not an empirical 1M single-request replay. 中文：MI325X 非正式全上下文验证已通过：https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29654139122
该结果验证了 MI325X 上原生 1M 上下文的服务端与 KV 工作点。本次一小时 trace 未包含单个 1M-token 请求，因此不属于 1M 单请求的实测回放。 |
Expand the full-context AgentX configuration to sample concurrency densely around the expected BF16 KV-pool capacity knee.\n\n中文：扩展 GLM-5.2 MI325X 全上下文 AgentX 配置，在预期的 BF16 KV 缓存池容量拐点附近进行密集并发采样。
Add full-context AgentX 1P1D, 1P2D, and 2P1D TP8 topology arms, plus MI325X Slurm launch and artifact plumbing. Keep EP/DP attention disabled on the Broadcom bnxt_re fleet. 中文：新增 GLM-5.2 MI325X 全上下文 AgentX 分离式推理扫描，覆盖 TP8 的 1P1D、1P2D 和 2P1D 拓扑，并补齐 MI325X Slurm 启动与产物收集流程。Broadcom bnxt_re 集群暂不启用 EP/DP attention。
JordanNanos
changed the title
中文：将 main 合并到 GLM-5.2 MI325X 扫描分支，并按追加规则重新放置性能变更日志条目。
Serialize missing-model downloads into the MI325X shared NFS Hugging Face cache before submitting disaggregated Slurm jobs. 中文：在提交 MI325X 分离式 Slurm 任务前，将缺失的 GLM-5.2 模型串行下载到共享 NFS Hugging Face 缓存，避免多个拓扑重复下载。
Use privileged directory creation only when the MI325X NFS cache is absent, then assign ownership to the runner before downloading model files. 中文：仅在 MI325X NFS 缓存目录缺失时使用特权创建目录，并在下载模型文件前立即将目录所有权交给 runner 用户。
Place the shared Hugging Face cache under the common runner-home filesystem already used for Slurm logs, avoiding the root-squashed data export. 中文：将共享 Hugging Face 缓存放到 Slurm 日志已验证可见的 runner 公共主目录文件系统中，绕过启用 root-squash 的数据挂载。
Propagate the multi-node container srun exit code and relay the preserved Slurm stderr through the GitHub job log. 中文：向上传递多节点容器 srun 的退出码，并将保留的 Slurm stderr 输出到 GitHub 任务日志，便于准确定位分布式服务启动失败。
Stage a per-run repository copy and Slurm logs under the runner-common filesystem proven visible on all compute nodes, then copy artifacts back and clean it. 中文：在所有计算节点均可见的 runner 公共文件系统中暂存每次运行的仓库副本与 Slurm 日志，完成后回传产物并清理，避免节点无法访问 runner 专属工作目录。
Allow NFS metadata to settle after a fast batch exit, then relay stderr and extended accounting before cleaning the per-run directory. 中文：Slurm 批任务快速退出后等待 NFS 元数据同步，再输出 stderr 与扩展计费状态，最后清理本次运行目录，避免丢失真实错误。
Run distributed container tasks with a non-login host shell and report the resolved Docker image before launch.\n\n中文：在 Slurm 分布式容器任务中使用非登录宿主机 shell，避免加载陈旧的 runner profile；启动前输出最终解析的 Docker 镜像。
Emit the resolved Docker argv for MI325X distributed launch failures so malformed arguments can be identified from Slurm stderr.\n\n中文：在 MI325X 分布式启动失败时输出最终 Docker 参数，便于从 Slurm stderr 中定位格式错误的参数。
Pass space-bearing metadata through Docker environment inheritance so JSON remains a single value in distributed Slurm launches. Remove the temporary argv trace.\n\n中文：通过 Docker 环境变量继承传递包含空格的元数据，确保 JSON 在 Slurm 分布式启动中保持为单一参数；同时移除临时参数跟踪。
Pass runner-selected RDMA settings into distributed SGLang containers, pre-pull the image on every allocated node before the creation barrier, and persist server logs in the shared benchmark directory.\n\n中文：将 runner 选择的 RDMA 配置传入 SGLang 分布式容器，在容器创建屏障前为所有分配节点预拉取镜像，并将服务端日志持久化到共享基准测试目录。
When a distributed SGLang process dies during a blocking startup wait, print its local prefill or decode log before tearing down the Slurm step.\n\n中文：分布式 SGLang 进程在启动等待阶段退出时，先输出对应节点的预填充或解码日志，再终止 Slurm 任务步骤。
Replace the unsupported DSA backend options with the NSA prefill/decode option names exposed by the MI325X disaggregated SGLang image.\n\n中文：将不受支持的 DSA 后端参数替换为 MI325X 分离式 SGLang 镜像实际提供的 NSA 预填充/解码参数名。
Reap failed SGLang server processes, report their return code, and allow process-substitution logs to flush before printing the local failure log.\n\n中文：回收失败的 SGLang 服务进程并输出返回码，同时等待进程替换日志完成刷新后再打印本地故障日志。
Mirror the working MI325X single-node top-k fallbacks in the disaggregated launcher and enable informative startup logging for GLM-5.2 worker initialization.\n\n中文：在分离式启动器中复用 MI325X 单节点已验证的 top-k 回退配置，并为 GLM-5.2 worker 初始化启用更详细的启动日志。
Use the same ROCm 7.2 SGLang 0.5.15 image as the successful single-node run. This published image includes MoRI and Broadcom userspace support while adding current GLM-5.2 runtime support. 中文：GLM-5.2 分离式推理改用当前版本的 SGLang 使用已通过单节点测试的 ROCm 7.2 SGLang 0.5.15 镜像。该公开镜像同时包含 MoRI、Broadcom 用户态支持以及当前 GLM-5.2 运行时支持。
Match the successful MI325X single-node recipe by selecting TileLang DSA attention and the Torch top-k backend. This avoids the sgl-kernel CUDA-only top-k path on gfx942. 中文：GLM-5.2 分离式推理改用可移植的 DSA top-k 后端 与已通过测试的 MI325X 单节点方案保持一致，DSA attention 使用 TileLang，top-k 使用 Torch 后端，避免 gfx942 上 sgl-kernel 的 CUDA 专用 top-k 路径。
Mirror GLM-5.2 scheduler exceptions to synchronous stderr before SGLang tears down its process tree, so MI325X Slurm logs preserve the actual initialization failure. 中文：输出 SGLang scheduler 初始化错误 在 SGLang 清理进程树之前，将 GLM-5.2 scheduler 异常同步写入 stderr，使 MI325X Slurm 日志能够保留真实的初始化失败原因。
Wrap the GLM-5.2 scheduler multiprocessing target so failures before SGLang enters its internal exception handler are written synchronously to stderr. 中文：追踪 scheduler 进程启动失败 为 GLM-5.2 scheduler 的 multiprocessing 入口增加外层异常输出，使 SGLang 内部异常处理器启动前的失败也能同步写入 stderr。
Patch the stable multiprocessing target assignment used by the published ROCm image so pre-handler scheduler failures are captured synchronously. 中文：适配 SGLang scheduler 入口布局 针对公开 ROCm 镜像中稳定的 multiprocessing target 赋值位置应用补丁，从而同步捕获内部异常处理器启动前的 scheduler 失败。
Use whitespace-tolerant matching for the scheduler multiprocessing target in the published ROCm image so the diagnostic wrapper is applied reliably. 中文：灵活匹配 scheduler 进程入口 对公开 ROCm 镜像中的 scheduler multiprocessing 入口使用可容忍空白差异的匹配方式，确保诊断包装器可靠生效。
Correct the Python raw regexes used to instrument the SGLang scheduler process so the diagnostic wrapper is actually installed.\n\n中文：修正用于插桩 SGLang 调度器进程的 Python 原始正则表达式，确保诊断包装器能够正确安装。
Mirror uncaught launch exceptions directly to stderr and enable Python fatal-signal traces for the GLM-5.2 disaggregated startup investigation.\n\n中文：将未捕获的启动异常直接同步输出到 stderr，并启用 Python 致命信号追踪，用于排查 GLM-5.2 分离式推理启动失败。
Install sentencepiece and tiktoken for GLM-5.2 disaggregated workers, which otherwise fail before model loading while constructing the tokenizer. Remove the temporary traceback instrumentation now that the root cause is identified.\n\n中文：为 GLM-5.2 分离式推理工作进程安装 sentencepiece 和 tiktoken，避免在模型加载前构建分词器时失败。根因已确认，同时移除临时异常追踪插桩。
Require the MI325X multi-node cache snapshot to contain its config, weight index, and tokenizer files before reuse. An incomplete GLM-5.2 snapshot is now resumed through snapshot_download instead of reaching SGLang without tokenizer.json. Remove the unnecessary tokenizer dependency installation.\n\n中文：MI325X 多节点共享缓存快照只有在配置、权重索引和分词器文件齐全时才会复用。若 GLM-5.2 快照不完整，将通过 snapshot_download 续传，避免缺少 tokenizer.json 时直接启动 SGLang；同时移除不必要的分词器依赖安装。
Point MI325X multi-node GLM-5.2 jobs at the exact node-local Hugging Face snapshot already validated by the successful single-node sweep on nodes 017 and 018. This avoids reconstructing the 753 GB checkpoint in the controller-visible cache.\n\n中文：MI325X 多节点 GLM-5.2 任务改为复用节点 017 和 018 上已通过单节点扫描验证的 Hugging Face 精确快照，避免在控制器可见缓存中重新构建 753 GB 检查点。
中文：将 GLM-5.2 MI325X 分离式推理的服务同步等待时间延长到一小时，并将该配置传入各 Slurm 容器，以适配多节点并发加载模型时更长的启动时间。
中文：同步最新 main，并按追加式变更日志规则将本 PR 的 GLM-5.2 MI325X 基准测试条目重新追加到文件末尾。
中文：合并本 PR 的 GLM-5.2 MI325X 变更日志条目，确保正式扫描仅执行一次 7 点单节点曲线和一次 12 点多节点曲线，避免重复占用 GPU。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29717743712 |
1 participant
Summary
[1, 2, 3, 4, 5, 6, 8]after the concurrency-1 qualification used about 35% of the KV pool.1P1D,1P2D, and2P1Dtopology arms, with at most four concurrency points per server allocation.bnxt_refleet.Validation
bash -npasses for the new and modified AMD multi-node launch scripts.uv run --with 'pydantic>=2' --with pyyaml --with pytest pytest utils/matrix_logic/ -q: 224 passed.utils/validate_perf_changelog.py --base-ref origin/main --head-ref HEAD: passed.main, so its dashboard provenance label is wrong. Future unofficial dispatches will set both workflow and input refs to this feature branch.中文说明
[1, 2, 3, 4, 5, 6, 8]。1P1D、1P2D和2P1D拓扑，每次服务端分配最多连续测试 4 个并发点。bnxt_re集群上，MoRI all-to-all 路径仍不稳定，因此多节点扫描暂不启用 EP 和 DP attention。验证
bash -n语法检查。uv run --with 'pydantic>=2' --with pyyaml --with pytest pytest utils/matrix_logic/ -q：224 项通过。utils/validate_perf_changelog.py --base-ref origin/main --head-ref HEAD：通过。main调度，导致 dashboard 来源标签错误；后续非正式调度会同时将 workflow ref 和输入 ref 指向本功能分支。