improvement(ci): bound docker layer caches and right-size ten runners - #7210
Conversation
The Blacksmith sticky disks backing our docker layer caches had no eviction policy. setup-docker-builder skips pruning entirely unless max-cache-size-mb is set, and BuildKit's own GC is time-based only (8 days unused), so on a repo that builds this often nothing ever aged out: app.Dockerfile/linux-amd64 hit 351 GB within a day of being created, and realtime — an image under 300 MB — sat at 249 GB. Layer caches alone were 920 GB across ten disks. Cap them per image via a cache_mb matrix field, alongside the bs_runner field that already encodes per-image sizing. The app image keeps 100 GB (several generations over its working set of ~34 layers plus monorepo apt/bun cache mounts); everything else takes the 25 GB default, which is still 4x the tightest working set in the matrix (pii, whose spaCy models are ~2.2 GB). The fallback lives in the composite action rather than an input default, because an unset matrix key arrives as the empty string and would bypass a default — silently restoring unbounded growth on any row that forgot the field. Runner sizes follow measured CPU and memory percentiles over 30 days: - CodeQL splits per language. javascript-typescript peaks at 19.5 GB so it stays on 8 vCPU; actions peaks at 1.3 GB and averages 22% CPU over a 39s median run, and drops to 4 vCPU. - The pii and realtime image builds drop to 2 vCPU. Both already ran on 8 vCPU earlier in the window, so the 8->4 step is measured, not modelled: realtime went 52s -> 51s and pii 24s -> 27s. - Five desktop release jobs drop to 2 vCPU. They peak under 0.4 GB and finish in 4-13s. None of these sit on their group's critical path — each has 190-275s of slack behind an app build that dominates it — so wall-clock is unchanged. Those five desktop jobs also hardcoded a Blacksmith label with no CI_PROVIDER fallback, against the invariant stated at the top of ci.yml. In GitHub break-glass mode they would have sat in `queued` forever; they now fall back like every other job. Left alone deliberately: the 16 vCPU app builds (memory-bound, and 16 vCPU measured 2.1x faster and 6% cheaper than 8 vCPU), Lint and Test (CPU-bound, 62% of the run above 80%), and the push-path Build App (build-amd64 has no needs:, so it is what stops a migration applying for a build that cannot ship).
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryThe PR bounds Blacksmith Docker layer caches and reduces runner capacity for measured low-resource CI jobs while preserving larger runners for memory-intensive work.
Confidence Score: 5/5The PR appears safe to merge, with no concrete changed-code defect established. The matrix values resolve to bounded cache sizes, the new runner expressions follow established provider-selection conventions, and the changed runner labels are already used elsewhere in the repository.
|
| Filename | Overview |
|---|---|
| .github/actions/docker-build/action.yml | Adds an optional cache-size input and supplies a bounded 25 GB fallback to the Blacksmith builder setup. |
| .github/workflows/ci.yml | Configures a 100 GB app cache override, passes cache limits into all image builds, and right-sizes several image and desktop orchestration runners. |
| .github/workflows/codeql.yml | Replaces the fixed Blacksmith runner with a valid include matrix that retains the larger tier for JavaScript/TypeScript and uses a smaller tier for Actions analysis. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
Matrix[Image matrix] --> Cache{cache_mb set?}
Cache -->|App| Large[100 GB cache cap]
Cache -->|Other images| Default[25 GB fallback]
Large --> Builder[Blacksmith Docker builder]
Default --> Builder
Provider{CI_PROVIDER}
Provider -->|Unset or blacksmith| Sized[Per-job Blacksmith runner]
Provider -->|Other value| GitHub[GitHub-hosted fallback]
Sized --> Jobs[Build and CI jobs]
GitHub --> Jobs
Reviews (1): Last reviewed commit: "improvement(ci): bound docker layer cach..." | Re-trigger Greptile
There was a problem hiding this comment.
1 issue found across 3 files
Confidence score: 3/5
- In
.github/actions/docker-build/action.yml, the pinned builder revision does not readmax-cache-size-mb, so both the matrix value and25600fallback are ignored and the intended cache-size control is not applied. Pin a builder revision that supports this input or perform the pruning through a supported mechanism.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name=".github/actions/docker-build/action.yml">
<violation number="1" location=".github/actions/docker-build/action.yml:69">
P1: With the pinned builder revision, this `with` key is not a supported or read input, so the matrix value and `25600` fallback are ignored. Pin a builder revision that implements `max-cache-size-mb`, or perform the prune in this workflow, before relying on the cache cap.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
| uses: useblacksmith/setup-docker-builder@a5256a73e30f09e37e3eceb8ca36043d17621d24 # v2 | ||
| with: | ||
| cache-key: ${{ steps.cache-key.outputs.value }} | ||
| max-cache-size-mb: ${{ inputs.max-cache-size-mb || '25600' }} |
There was a problem hiding this comment.
P1: With the pinned builder revision, this with key is not a supported or read input, so the matrix value and 25600 fallback are ignored. Pin a builder revision that implements max-cache-size-mb, or perform the prune in this workflow, before relying on the cache cap.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At .github/actions/docker-build/action.yml, line 69:
<comment>With the pinned builder revision, this `with` key is not a supported or read input, so the matrix value and `25600` fallback are ignored. Pin a builder revision that implements `max-cache-size-mb`, or perform the prune in this workflow, before relying on the cache cap.</comment>
<file context>
@@ -42,11 +49,24 @@ runs:
uses: useblacksmith/setup-docker-builder@a5256a73e30f09e37e3eceb8ca36043d17621d24 # v2
with:
cache-key: ${{ steps.cache-key.outputs.value }}
+ max-cache-size-mb: ${{ inputs.max-cache-size-mb || '25600' }}
- name: Build and push (Blacksmith)
</file context>
Summary
Our Blacksmith docker layer caches had no eviction policy at all.
setup-docker-builderskips pruning entirely unlessmax-cache-size-mbis set, and BuildKit's own GC is time-based only (8 days unused), so on a repo that builds this often nothing ever ages out. Measured over the last 30 days:app.Dockerfile/linux-amd64— 351 GB, created a day earlierrealtime.Dockerfile/linux-amd64— 249 GB, for an image under 300 MBSticky disk storage is billed per GB-month, and it was still growing.
What changed
cache_mbmatrix field, sitting alongside thebs_runnerfield that already encodes per-image sizing. App keeps 100 GB (several generations over its working set: ~34 layers plus monorepo apt/bun cache mounts); everything else takes the 25 GB default, still ~4x the tightest working set in the matrix (pii, whose spaCy models are ~2.2 GB). The fallback lives in the composite action rather than an inputdefault:— an unset matrix key arrives as'', which counts as "provided", so adefault:would never apply and a row that forgot the field would silently go back to unbounded growth.javascript-typescriptpeaks at 19.5 GB (p95 over 3090 runs) and would OOM below the 8 vCPU tier, so it stays.actionspeaks at 1.3 GB and averages 22% CPU over a 39s median run — it drops to 4 vCPU.piiandrealtimeimage builds drop to 2 vCPU. Both ran on 8 vCPU earlier in the window, so the 8→4 step is measured rather than modelled: realtime went 52s → 51s, pii 24s → 27s.CI_PROVIDERfallback they were missing, against the invariant stated at the top ofci.yml. In GitHub break-glass mode they would have sat inqueuedforever.Why this can't slow CI
Every downsized job sits off its parallel group's critical path, which is set by an app build in each case:
Left alone deliberately
build-amd64has noneeds:, so this job is what stops a DB migration applying for a build that cannot ship. It is not redundant.next build's type-check pass. Looks duplicated againstturbo run type-check, butapps/sim/tsconfig.jsonexcludes.next, so only thenext buildpass validates Next's generated route types.Type of Change
Testing
actionlintreports strictly fewer findings than before (the five hardcoded labels are now inside expressions) and zero new ones. Verified all 14 matrix rows resolve to a bounded cache value and that no job can reach the unpruned path. Job names are unchanged, and no ruleset requires status checks.bun run lint,check-block-registry, and all 37check:auditspass.Checklist