docs: add /ci-queue-times command for per-cluster CI queue latency / 新增 /ci-queue-times 命令：按集群查看 CI 排队延迟#2270
docs: add /ci-queue-times command for per-cluster CI queue latency / 新增 /ci-queue-times 命令：按集群查看 CI 排队延迟#2270JordanNanos wants to merge 4 commits into
Conversation
…latency Adds a .claude/commands shortcut with the three gh-API queries for checking CI queue times across all GPU clusters: live queued jobs with wait-so-far, historical per-cluster queue latency (started_at - created_at, re-run artifacts filtered), and per-pool runner availability. Includes the cluster:* label normalization and SLURM second-queue caveats. 中文：新增 /ci-queue-times 命令，用于查看所有 GPU 集群的 CI 排队时间。包含三个基于 gh API 的查询：当前正在排队的任务及已等待时长、按集群统计的历史排队延迟 （started_at − created_at，已过滤重跑导致的异常值），以及按 runner 池统计的 可用性。命令中已处理 cluster:* 标签归一化，并注明 SLURM 二级队列的注意事项。 No PR opened per request; branch only.
|
Sample output of Historical queue latency — window: scanned runs newest→oldest until ≥25 H200 samples were collected (486 GPU jobs total). H200 rows: For contrast, the busiest pool in the same window was Runner availability, H200 pools (live): (Pools overlap: Live queue: zero H200 jobs queued right now — the current queue is entirely Takeaway: H200 has no meaningful CI queue — p50 wait ≈ 6 seconds, worst case 13 min, half the pool idle. If a sweep is stuck, H200 is not the bottleneck; 中文：以上为 |
… queued jobs Drop the historical per-cluster latency analysis in favor of a single live-snapshot pass: per-pool active/queued summary, active jobs with runtime-so-far, and queued jobs with wait-so-far. Adds an optional pool filter regex argument, a total_count coverage check, and a caveat for the 0-active-but-queued case (offline/stuck/org-scoped-busy runners). 中文：将 /ci-queue-times 命令改为实时快照模式，聚焦当前正在运行的任务和排队队列， 移除按集群统计的历史排队延迟分析。新流程单次扫描即可输出：按池统计的 活跃/排队任务数、活跃任务及已运行时长、排队任务及已等待时长。新增可选的 池过滤正则参数、total_count 覆盖度校验，以及“活跃为 0 但排队堆积”场景 （runner 离线、状态卡死或被组织内其他仓库占用）的排查说明。
|
Reworked per review feedback: the command no longer does historical analysis — it now takes a single live snapshot and reports (1) per-pool active/queued summary, (2) active jobs with runtime-so-far, (3) queued jobs with wait-so-far, plus an optional pool-filter argument. Fresh sample output from the new version: 中文：已按反馈意见重写该命令：不再做历史分析，改为单次实时快照，输出（1）按池统计的活跃/排队任务数、（2）活跃任务及已运行时长、（3）排队任务及已等待时长，并支持可选的池过滤参数。上方为新版本的最新示例输出。 |
…mmary Join the runners API into the Step 2 summary so it shows runners_online, runners_busy, jobs_active, jobs_queued per pool. Documents the one-job-per- runner-agent invariant and the diagnostic patterns it enables (saturated, pool-down, stuck/org-scoped-busy runners, snapshot skew), plus the pool-label overlap note. All-zero rows are filtered as noise. 中文：为 /ci-queue-times 的汇总表新增 runner 维度列：将 runners API 的数据并入 第二步汇总，按池展示 runners_online、runners_busy、jobs_active、jobs_queued。 注明“每个 runner agent 同一时刻只运行一个任务”的基本约束，以及由此得出的 诊断模式（池饱和、池宕机、runner 状态卡死或被组织内其他仓库占用、快照误差）， 并说明池标签重叠会导致按行求和重复计数。全零行作为噪音过滤。
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| ```bash | ||
| { gh api "repos/SemiAnalysisAI/InferenceX/actions/runs?status=queued&per_page=50" --jq '.workflow_runs[].id' | ||
| gh api "repos/SemiAnalysisAI/InferenceX/actions/runs?status=in_progress&per_page=50" --jq '.workflow_runs[].id' | ||
| } | sort -u | while read -r RUN; do | ||
| gh api "repos/SemiAnalysisAI/InferenceX/actions/runs/$RUN/jobs?per_page=100" --paginate \ | ||
| --jq '.jobs[] | select(.status=="queued") | {labels: [.labels[] | sub("^cluster:";"") | select(test("^(self-hosted|Linux|X64|ARM64|slurm)$|_\\d+$") | not)], name, created_at, run_id}' |
There was a problem hiding this comment.
🟡 Step 1's two runs-listing calls (.../actions/runs?status=queued and status=in_progress, lines 20-21) omit --paginate, so each status is silently capped at the first 50 results — inconsistent with line 23's jobs call, which does paginate. Since the API returns runs newest-first, a busy period (>50 concurrent in_progress runs across sweeps/canary/e2e/lint) would silently drop the oldest, longest-waiting runs from the live-queue table with no error or truncation indicator. Fix: add --paginate to both calls on lines 20-21.
Extended reasoning...
The bug: Step 1 lists candidate run IDs to scan for queued matrix jobs with:
gh api "repos/SemiAnalysisAI/InferenceX/actions/runs?status=queued&per_page=50" --jq '.workflow_runs[].id'
gh api "repos/SemiAnalysisAI/InferenceX/actions/runs?status=in_progress&per_page=50" --jq '.workflow_runs[].id'Neither call passes --paginate. Without it, gh api fetches exactly one page of the "list workflow runs" endpoint, and per_page=50 sets that page's size to 50 — below the endpoint's actual max of 100. So each of status=queued and status=in_progress is capped at 50 runs, silently: no error, no truncation marker, the table just renders as if it were complete.
Why this is an inconsistency, not an intentional cap: The very next call in the same step, three lines down, fetches jobs for each run and correctly appends --paginate:
gh api "repos/SemiAnalysisAI/InferenceX/actions/runs/$RUN/jobs?per_page=100" --paginateStep 2 also avoids this pitfall by using gh run list --limit "$LIMIT", where the gh CLI internally paginates run listings up to the requested limit. So the omission on lines 20-21 looks like an oversight specific to those two calls, not a deliberate design choice.
Why it matters here specifically: The GitHub Actions runs endpoint returns results newest-first. If more than 50 runs share a status, the calls silently drop the oldest matches — which, for in_progress, are exactly the runs that have been running (and hence queuing matrix jobs) the longest. This repo's whole premise for Step 1 is scanning in_progress runs because their matrix jobs may still be queued waiting for a runner (the sweep fan-out). Every in_progress workflow run counts toward that total, not just GPU sweep runs — lint, canary, e2e, recipe-reminder, and any concurrent PR/cron sweeps all contribute. During a busy period it's plausible for in_progress alone to exceed 50, at which point the longest-waiting sweep's queued jobs are dropped from the "live queue" table entirely, with the report looking complete.
Step-by-step proof:
- Suppose at query time there are 60
in_progressworkflow runs across the repo (a mix of sweeps, canary, e2e, lint, cron jobs), sorted newest-first by the API. gh api ".../runs?status=in_progress&per_page=50"(no--paginate) returns only the first page — the 50 most recently started in_progress runs. The 10 oldest in_progress runs (positions 51-60) are never returned.- Suppose one of those 10 dropped runs is a large matrix sweep that started earliest and still has several matrix jobs sitting
queuedwaiting for a free GPU runner — the longest-waiting jobs in the whole queue. - That run's ID never enters the
sort -u | while read -r RUNloop, sogh api ".../runs/$RUN/jobs" --paginateis never called for it, and its queued jobs never appear in the finaljq -soutput. - The command still prints a clean table via
column -t, with no indication that 10 in_progress runs (and their queued jobs) were never scanned — the operator sees an apparently complete "who is waiting right now" report that is actually missing the longest-waiting jobs.
Fix: add --paginate to both calls on lines 20-21, matching line 23:
gh api "repos/SemiAnalysisAI/InferenceX/actions/runs?status=queued&per_page=50" --paginate --jq '.workflow_runs[].id'
gh api "repos/SemiAnalysisAI/InferenceX/actions/runs?status=in_progress&per_page=50" --paginate --jq '.workflow_runs[].id'(Optionally also raise per_page to 100, the endpoint max, to reduce round trips.)
Severity rationale: This only manifests under unusually high concurrent CI load, and the failure mode is an incomplete (not incorrect) diagnostic table in an agent-instruction slash command — no production or benchmark path is affected. It's a real, trivially-fixable inconsistency with the rest of the same file, so worth flagging, but not blocking.
| ```bash | ||
| gh api "repos/SemiAnalysisAI/InferenceX/actions/runners?per_page=100" --paginate \ | ||
| --jq '.runners[] | . as $r | [.labels[].name | select(test("^(cluster:)?(b200|b300|h100|h200|gb200|gb300|mi300x|mi325x|mi355x)")) | sub("^cluster:";"") | sub("_\\d+$";"")] | unique | .[] | {c: ., status: $r.status, busy: $r.busy}' \ | ||
| | jq -s -r 'group_by(.c) | map({c: .[0].c, total: length, | ||
| online: ([.[]|select(.status=="online")]|length), | ||
| busy: ([.[]|select(.busy==true)]|length)}) | sort_by(.c) | .[] | ||
| | [.c, (.total|tostring), (.online|tostring), (.busy|tostring)] | @tsv' \ | ||
| | (printf "pool\ttotal_runners\tonline\tbusy\n"; cat) | column -t -s$'\t' | ||
| ``` |
There was a problem hiding this comment.
🟡 Step 3's runner-availability query fans out over ALL matching normalized labels (unique | .[]) instead of picking one canonical pool label per runner, so a single runner with multiple qualifying labels (pool, cluster:* alias, multi-node subset, and its own per-runner name label) gets counted in 3–4 different pool rows in the output table. This inflates total/online/busy counts and fabricates bogus rows (e.g. h100-dgxc-slurm) that don't correspond to any real pool; fix by picking a single canonical label per runner (e.g. first // empty, as Step 2 already does).
Extended reasoning...
The bug: Step 3's jq pipeline computes, per runner, the set of all normalized labels that match the broad prefix regex ^(cluster:)?(b200|b300|h100|h200|gb200|gb300|mi300x|mi325x|mi355x), then does unique | .[] — emitting one {c, status, busy} record for every surviving label rather than choosing a single canonical pool label. The subsequent group_by(.c) then treats each of those records as an independent runner-in-a-pool, so one physical runner ends up counted in several different "pool" rows of the final table.
Why this actually happens: configs/runners.yaml shows real runners are registered with multiple simultaneous labels — a pool label (h100), a multi-node subset label for some runners (h100-multinode), a cluster alias (cluster:h100-dgxc), and the runner's own unique name label (h100-dgxc-slurm_16). All of these pass the regex above because it's a substring/prefix test, not a full match. After the two sub() normalizations (sub("^cluster:";""), sub("_\\d+$";"")), these collapse to four distinct strings for the same runner: h100, h100-dgxc, h100-dgxc-slurm, and h100-multinode. Two of those (h100-dgxc and h100-dgxc-slurm) are not real pool names at all — they're artifacts of stripping the cluster: prefix and the runner's numeric suffix.
Step-by-step proof (using runner h100-dgxc-slurm_16, which is online and carries labels h100, h100-multinode, cluster:h100-dgxc, h100-dgxc-slurm_16 per configs/runners.yaml):
.labels[].nameyields:h100,h100-multinode,cluster:h100-dgxc,h100-dgxc-slurm_16(plus default labels likeself-hostedwhich fail the regex and are dropped).select(test(...))keeps all four SKU-prefixed labels.sub("^cluster:";"")turnscluster:h100-dgxcintoh100-dgxc.sub("_\\d+$";"")turnsh100-dgxc-slurm_16intoh100-dgxc-slurm.uniquenow holds 4 distinct strings:["h100", "h100-dgxc", "h100-dgxc-slurm", "h100-multinode"]..[] | {c: ., status: $r.status, busy: $r.busy}emits 4 separate records for this single physical runner.- After
jq -s | group_by(.c), this one runner contributes+1to thetotal/onlinecount in FOUR rows of the output table:h100,h100-dgxc,h100-dgxc-slurm, andh100-multinode— two of which (h100-dgxc,h100-dgxc-slurm) are not real pools inconfigs/runners.yamlat all.
I reproduced this exact behavior by piping the runner's label set through the Step 3 jq expression, confirming it yields 4 rows for one runner.
Contrast with the rest of the file: Step 1 (line 24) explicitly filters out per-runner name labels via select(test("...|_\\d+$") | not). Step 2 (line 42) picks a single canonical label per job via | first // empty. Step 3 is the outlier — it neither filters out the per-runner name label nor reduces to one label, so it's the only one of the three queries that multi-counts. This also directly contradicts the file's own Caveats section (lines 71-75), which states the sub() calls exist to "normalize" label shapes, not to multiply-count a runner across several pool buckets.
Impact: Step 3 is described as "the why behind queue times" — its whole purpose is giving an operator accurate total/online/busy counts per pool to diagnose queue pressure. As written, it will, on every invocation, inflate counts for any runner carrying more than one qualifying label (which per configs/runners.yaml is most runners — anything with a cluster:* alias or multi-node subset label) and fabricate nonexistent pool rows like h100-dgxc-slurm. That's actively misleading for its stated diagnostic purpose.
Fix: Change unique | .[] to pick one canonical label per runner, e.g. mirror Step 2's | first // empty, or add the same per-runner-name-label exclusion Step 1 uses (select(test("_\\d+$") | not)) before taking first.
There was a problem hiding this comment.
Fixed in a355cdd — runner bucketing now comes from configs/runners.yaml pool membership (matched by runner name) instead of label fan-out: one runner counts once per configured pool, bogus rows like h100-dgxc-slurm are gone, and all configured pools now appear. / 已在 a355cdd 修复：runner 侧的池归属改为以 configs/runners.yaml 的成员配置为准（按 runner 名称匹配），不再按标签扇出 —— 同一 runner 只按配置计入对应池，h100-dgxc-slurm 之类的伪池行已消除，所有已配置的池都会显示。
|
The Step 2 summary now includes runner capacity alongside job counts, so saturation vs pool-outage is visible at a glance. Fresh sample (note the (All-zero rows are filtered as noise.) Interpretation rules are documented in the command file: a self-hosted runner agent runs exactly one job at a time, so 中文：第二步汇总表现已加入 runner 容量维度，可一眼区分“池饱和”与“池宕机”。上方为最新示例 —— 注意 |
…nate run lists Address review feedback on PR #2270: - Add --paginate to the runs-listing calls (previously silently capped at the first page, dropping the oldest/longest-waiting runs in busy periods). - Bucket runners by configs/runners.yaml pool membership (matched by runner name) instead of fanning out over every matching label: one runner no longer counts toward 3-4 pool rows, and bogus rows derived from per-runner name labels (e.g. h100-dgxc-slurm) can no longer appear. This also surfaces every configured pool (mi300x-tw, mi325x-tw, h100-multinode, *-disagg, ...). - Document that gh api --jq does not pass jq --arg, the demand-vs-capacity bucketing split, and the shared-membership reading of busy > active. 中文：处理 PR #2270 的评审意见： - 为列出工作流运行的请求补上 --paginate（此前只取第一页，繁忙时段会静默 丢弃最老、等待最久的运行）。 - runner 侧的池归属改为以 configs/runners.yaml 的成员配置为准（按 runner 名称匹配），不再按标签扇出：同一 runner 不会被重复计入 3-4 个池行， 由 runner 名称标签衍生的伪池行（如 h100-dgxc-slurm）也不会再出现。 同时所有已配置的池（mi300x-tw、mi325x-tw、h100-multinode、*-disagg 等） 都会显示。 - 补充说明：gh api --jq 不支持透传 jq --arg、需求侧（任务标签）与容量侧 （runners.yaml 成员）的分桶口径，以及共享成员池下 busy > active 的解读。
|
Sample output of Step 2 — per-pool summary: Step 3 — active jobs (runtime so far): Step 4 — queued jobs (wait so far): 中文：以上为 |
1 participant
Description
Adds a new
.claude/commands/ci-queue-times.mdslash command —/ci-queue-times [history-run-limit]— for checking CI queue times across all GPU clusters.GitHub has no queue-metrics API for self-hosted runners, so the command derives queue time from the Actions API (queue time = job
started_at−created_at, cluster = job runner label; pool mapping fromconfigs/runners.yaml). It bundles three verified queries:q >= 0) andcluster:*label normalization.Caveats are documented in the file: the SLURM second queue on
*-dgxc/*-nvpools (check viasqueue -u <runner-user>), canary-gate exclusion, and label-shape handling.None of the existing
.claude/commands/covers this — the PR-status commands only see GitHub check states, andklaud-pr-status-html's queue-aware ETA is a pessimistic pool-pressure estimate scoped to claude/* PRs, not measured per-cluster queue latency.All three queries were executed against live Actions data during development. This is an agent-instruction file → English-only per the bilingual-docs exception in
AGENTS.md.Related Issue
N/A
Type of Change
Checklist
AGENTS.md)perf-changelog.yaml— N/A, no image/config change/reuse-sweep-run... — N/A, no benchmark sweep needed (no sweep label applied; nothing here affects benchmark results)中文说明
新增
.claude/commands/ci-queue-times.md斜杠命令 ——/ci-queue-times [历史窗口]—— 用于查看所有 GPU 集群的 CI 排队时间。GitHub 没有针对自托管 runner 的排队指标 API，因此该命令通过 Actions API 推导排队时间（排队时间 = 任务
started_at−created_at，集群 = 任务的 runner 标签；池与 runner 的映射见configs/runners.yaml）。包含三个已验证的查询：q >= 0），并对cluster:*标签做归一化。文件中已注明注意事项：
*-dgxc/*-nv池存在 SLURM 二级队列（需在集群上用squeue -u <runner-user>查看）、金丝雀门禁（canary gate）不计入排队时间、以及标签格式的处理方式。现有
.claude/commands/均不覆盖此场景 —— PR 状态类命令只能看到 GitHub check 的状态，而klaud-pr-status-html的排队 ETA 只是面向 claude/* PR 的悲观池压力估算，并非实测的按集群排队延迟。三个查询在开发过程中均已对仓库的 Actions 数据实际执行验证。本文件属于 agent 指令文件，按
AGENTS.md的双语文档例外规则仅提供英文版本。其他：无镜像或配置变更，无需更新
perf-changelog.yaml；未添加扫描（sweep）标签，本变更不应触发任何 GPU 基准测试。