fix(ci): give AVM check-circuit a workable timeout and CPU budget

[AztecBot]

Re-evidenced against run 30400452994 (2026-07-28 21:53 UTC, 63aa610) — the fourth red run of this workflow in the last five. This PR consolidates the duplicate PRs opened for earlier occurrences. It is mergeable onto current next with no rebase needed. One manual step is required to land it — see the last section.

Problem

avm-check-circuit keeps going red on next. Exit code 124: a wall-clock timeout, not a circuit-check failure.

Every dumped input but one checks in 1–4s. Exactly one blows its budget, and it is the same one every time:

FAILED (ee905c17dc98846d): bb-avm avm_check_circuit -v --avm-inputs
  .../single-node/block-building/multiple_blobs/avm-circuit-inputs-tx-0x1406da30....bin (31s) (code: 124)
parallel: This job failed:
run_test_cmd '...:ISOLATE=1:TIMEOUT=30s:NAME=avm_cc_single-node_block-building_multiple_blobs_0x1406da30 ...'

Root cause

avm_check_circuit_cmds gives every dumped input the same flat budget — TIMEOUT=30s plus the harness defaults CPUS=2 / MEM=8g — but the inputs are wildly uneven. Almost all check in 1–4s. The heaviest is the multiple_blobs tx, which by design emits a full-size public log and enough side effects to fill more than one blob; it produces a ~700k-row trace.

The circuit is valid — the process is killed at the wall, not failing a check. Tonight's test log is the clearest instance yet: the check completed successfully and printed its full stats block, and the timeout wrapper fired in the same second, before the process could exit.

21:53:00 Simulating...
21:53:07 Running check (with skippable) circuit over 700560 rows. (mem: 2159.71 MiB)
21:53:30 circuit check: success                       <- valid, no assertion, no OOM
21:53:30 proving/check_circuit_ms: 19750
21:53:30 tracegen/all_ms: 4899
21:53:30 simulation/all_ms: 1729
21:53:30 timeout: sending signal TERM to command 'bash'

The measured stages sum to ~26.4s of a 30s budget, and total wall clock lands on 30.0s. There is no headroom left at all — the run is decided by ordinary spot-instance variance. Because parallelize runs with --halt now,fail=1 (ci3/parallelize), that one input aborts the whole run and turns the check red.

Last five runs on next:

Run	Result	multiple_blobs heavy tx
30310279188	red	killed at 30s
30327434657	red	killed at 30s
30377443468	red	killed at 30s
30386036639	green	31s wall — squeaked in under the 30s inner cap
30400452994	red	check succeeded, then killed at 30s

The trace did not grow and the AVM did not change. The row count is identical — 700560 — in the green run and in every red one; only the wall clock moves. The green→red boundary contains no barretenberg/ change, and tonight's head is an accumulated-ports commit that touches nothing in the AVM.

The original WARNING comment called this exact failure mode in advance: "theoretically, transactions could need more CPU and MEM than we allocate by default. In that case, they might start timing out."

Fix

-local prefix="$hash:ISOLATE=1:TIMEOUT=30s"
+local prefix="$hash:ISOLATE=1:CPUS=4:MEM=16g:TIMEOUT=2m"

• TIMEOUT=2m gives ~4x headroom over the worst observed run instead of 1.0x. The fast inputs still finish in 1–4s, so this costs nothing in the happy path. The timeout is a hang guard, not a performance assertion — parallelize separately warns on any test exceeding 5 minutes, so a real blowup still gets flagged.
• CPUS=4 / MEM=16g (up from the defaults 2 / 8g). avm_check_circuit is CPU bound and the run was leaving half the box idle: this file calls parallelize 16, so at 2 CPUs each it used only 32 of the 64 vCPUs the CI instance reports. 16 × 4 = 64 restores total quota == nproc, the same ratio parallelize assumes by default (jobs = num_cpus / 2 at CPUS=2), and 64 vCPU is the floor of the spot fleet's instance pools.
• On memory, one correction to this PR's earlier text: the headroom is thinner than previously stated. Earlier occurrences were killed mid-check at ~2.1 GiB; tonight's run got all the way through and peaked at 4.82 GiB against the 8g cap (60%). Still not the binding constraint, but MEM=16g is now doing real work rather than being purely cosmetic. 16 × 16g sits well inside the 493 GiB the failing box reported.

Testing

I could not run the failing command in this container to show red-then-green, and I am stating that plainly rather than implying a local repro: no barretenberg build, no AWS credentials for the input tarball in s3://aztec-ci-artifacts/build-cache/, and the dumped inputs are regenerated per run.

Verified directly against this branch in this session:

• bash -O extglob -O globstar -n yarn-project/end-to-end/bootstrap.sh passes on the branch's version of the file.
• Feeding the emitted prefix through ci3/source_test_params resolves it to ISOLATE=1 CPUS=4 MEM=16g TIMEOUT=2m with test_cmd intact — the harness reads the new fields as intended.
• timeout -v 2m is accepted by the coreutils build in the CI image's lineage.
• Re-read ci3/parallelize to confirm the fail-fast (--halt now,fail=1) and 5-minute slow-test warning claims above.

Real verification is the next run of this workflow on next.

This failure has recurred for months and the fix has never been reviewed

Worth a maintainer's attention more than the one-line diff:

• git log --all -S'TIMEOUT=30s' -- yarn-project/end-to-end/bootstrap.sh returns 69 commits, and there are now 71 cb/*avm* branches on the remote. Nearly all are bot attempts at this same fix since the line was introduced in #18747.
• On next, the only commit that has ever touched that line is test(avm): e2e tests can optionally dump avm circuit inputs to files #18747 itself. No bump has ever landed.
• Earlier attempts were not rejected on the merits — they were auto-closed unreviewed. #23743 and #23803 proposed the identical change and were closed with "Automatically closing this stale claudebox draft PR (no updates for 5+ days)."
• No human has commented on this PR or any of its predecessors. There is no recorded objection to the change — only an absence of review.

Each failure spawns a draft PR, the draft expires before anyone looks, and the next failure spawns another. Merging one of these ends the loop.

The one manual step: the stale-closer targets drafts, so this PR needs to be marked "Ready for review" or it will expire the same way its predecessors did. Claudebox has no tool that can un-draft a PR, so a human has to click it.

Note for reviewers

Targeting next directly rather than merge-train/spartan, where yarn-project/** normally routes, because the check is red on next now and this only touches a CI resource budget.

Separately, and deliberately not bundled here: bb sizes its thread pool from min(32, std::thread::hardware_concurrency()) (barretenberg/cpp/src/barretenberg/common/thread.cpp:17), which is not cgroup-quota aware. Tonight's log again shows num threads: 32 inside a container capped at --cpus=2 — 32 threads contending for a 2-CPU quota, which invites CFS throttling stalls and is a plausible source of the run-to-run jitter on identical work. docker_isolate already passes CPUS into the container and bb already honours a HARDWARE_CONCURRENCY env var, so aligning the two would be a small change. I left it out because I cannot measure the effect in this environment, and an unverified performance change does not belong in a CI unblock.

---

Updated by claudebox for run 30400452994

---

Created by claudebox · group: slackbot

[AztecBot]

Current failure matches this PR's root cause.

Run 30327434657 failed in avm-check-circuit on next at 63e8535. The failed child log is 1019f83651cb82e4: the single-node/block-building/multiple_blobs input simulated and generated its trace, then reached Running check (with skippable) circuit over 700560 rows and was killed by the fixed TIMEOUT=30s wrapper. There was no circuit assertion or OOM in the log.

I rechecked the branch locally:

• bash -O extglob -O globstar -n yarn-project/end-to-end/bootstrap.sh
• ci3/source_test_params parses the new generated prefix as ISOLATE=1 CPUS=4 MEM=16g TIMEOUT=2m

So today's red run is another occurrence of the same too-tight per-input CI budget addressed here.

---

Created by claudebox · group: slackbot

[AztecBot]

Tonight's failure is another occurrence of this PR's root cause, and it is the cleanest proof yet that the circuit is fine and only the budget is wrong.

Run 30400452994 failed in avm-check-circuit on next at 63aa610. Failed child log: ee905c17dc98846d, the single-node/block-building/multiple_blobs input again. In this run the check finished successfully and printed its whole stats block, and the timeout wrapper fired in the same second:

21:53:07 Running check (with skippable) circuit over 700560 rows. (mem: 2159.71 MiB)
21:53:30 circuit check: success (mem: 4820.82 MiB)
21:53:30 proving/check_circuit_ms: 19750
21:53:30 timeout: sending signal TERM to command 'bash'

No assertion, no OOM. Measured stages sum to ~26.4s (simulation 1.73s + tracegen 4.90s + check 19.75s) inside a 30s cap, and total wall clock landed on 30.0s — zero headroom, so spot-instance variance decides red vs. green. Four of the last five runs of this workflow have been red on this one input; the one green run took 31s wall.

One correction to the PR body, now applied: earlier occurrences were killed mid-check at ~2.1 GiB peak, so this PR previously said memory was nowhere near the cap. Tonight's run got all the way through and peaked at 4.82 GiB against the 8g cap (60%). Memory still isn't the binding constraint, but it has less slack than stated, which makes the MEM=16g half of this change less cosmetic than I claimed.

Re-verified on this branch in this session: bash -n on the modified bootstrap.sh passes; ci3/source_test_params resolves the generated prefix to ISOLATE=1 CPUS=4 MEM=16g TIMEOUT=2m; ci3/parallelize does run --halt now,fail=1, so this single input aborts the whole run.

The branch still merges onto current next with no rebase — yarn-project/end-to-end/bootstrap.sh has not changed since this PR's base. It remains a draft, which is what has killed every predecessor: the stale-closer targets drafts. Marking it ready for review is the one step claudebox cannot do.

---

Created by claudebox · group: slackbot