fix(ci): raise AVM check-circuit per-tx timeout to 120s - #24952
Conversation
|
Superseded by #24959, which makes the same Five open PRs plus ~50 stale branches all propose this same one-line change and none has ever landed. Consolidating on #24959 — safe to close this one. Created by claudebox · group: |
|
Closing in favour of #24959, which carries the same fix rebased onto current This is a consolidation, not a rejection — the analysis and the auto-dispatch-loop writeup here were correct. Six identical open drafts were competing for the same review, and each was expiring to the stale-draft closer before anyone looked. Please review and merge #24959 to break the loop. Created by claudebox · group: |
What
Raise the per-tx timeout for the AVM
check-circuitcommands inyarn-project/end-to-end/bootstrap.shfrom30sto120s, and refresh the stale warning comment.Root cause (identical to every prior occurrence)
The AVM Circuit Inputs Collection and Check workflow failed on the
nextmerge queue (run 30045391060, commit6295a38). Theavm-check-circuitjob exited124— a wall-clock timeout, not a correctness failure. From the CI dashboard log, every dumped tx checked in 2–3s except one:avm_check_circuit_cmdsapplies a uniformTIMEOUT=30s(defaultCPUS=2/MEM=8g) to every tx. Themultiple_blobsBatchCalltx emits a full-size public log to span >1 blob, producing by far the largest AVM circuit in the suite (~700k rows). Simulation + trace-gen (~6s) + checking that circuit (~24s) on 2 CPUs lands right at the 30s ceiling with zero headroom, so it intermittently trips under merge-queue load. No OOM (dmesg clean), no assertion. The triggering commit (#24934) only forward-portscli/bot/node-keystoreand deletesTHREAT_MODEL.md— nothing AVM-related, so this is not a regression. The existing in-codeWARNINGcomment anticipated exactly this.Fix
Raise the ceiling to
120s(~4× the largest observed check). Small txs are unaffected — the timeout is a maximum and they still finish in 2–3s.check_circuitis bounded, deterministic O(rows) work, so a larger ceiling cannot let a hang run forever.CPUS/MEMleft at defaults (peak mem was ~2.1 GiB; bumping CPUs would oversubscribe the 64-core runner against the 16-wayparallelize); the refreshed comment points maintainers at those knobs if a future tx needs them.This exact failure has been dispatched to claudebox dozens of times; ~30
cb/avm-check-circuit-timeout*branches exist. Prior fix PRs were never rejected on merit — they were auto-closed as stale claudebox drafts after 5 days unmerged, sonextstill carriesTIMEOUT=30sand the job keeps going red and re-dispatching:This PR needs to be merged (by a maintainer, or landed via claudebox merge) to break the cycle. Pinging the AVM / #team-bonobos owners.
Testing
Config-only change to a CI timeout; no code paths change. The fix is exercised by this PR's own
avm-circuit-inputsrun (theci-draftlabel runs CI on the draft) — the previously-timing-outmultiple_blobscheck now has headroom to complete.Created by claudebox · group:
slackbot