Skip to content

improvement(ci): bound docker layer caches and right-size ten runners - #7210

Merged
waleedlatif1 merged 1 commit into
stagingfrom
ci/rightsize-runners-and-cap-cache-storage
Aug 28, 2026
Merged

improvement(ci): bound docker layer caches and right-size ten runners#7210
waleedlatif1 merged 1 commit into
stagingfrom
ci/rightsize-runners-and-cap-cache-storage

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Summary

Our Blacksmith docker layer caches had no eviction policy at all. setup-docker-builder skips pruning entirely unless max-cache-size-mb is set, and BuildKit's own GC is time-based only (8 days unused), so on a repo that builds this often nothing ever ages out. Measured over the last 30 days:

  • app.Dockerfile/linux-amd64351 GB, created a day earlier
  • realtime.Dockerfile/linux-amd64249 GB, for an image under 300 MB
  • layer caches in total — 920 GB across 10 disks

Sticky disk storage is billed per GB-month, and it was still growing.

What changed

  • Cap the layer cache per image via a cache_mb matrix field, sitting alongside the bs_runner field that already encodes per-image sizing. App keeps 100 GB (several generations over its working set: ~34 layers plus monorepo apt/bun cache mounts); everything else takes the 25 GB default, still ~4x the tightest working set in the matrix (pii, whose spaCy models are ~2.2 GB). The fallback lives in the composite action rather than an input default: — an unset matrix key arrives as '', which counts as "provided", so a default: would never apply and a row that forgot the field would silently go back to unbounded growth.
  • CodeQL sizes per language. javascript-typescript peaks at 19.5 GB (p95 over 3090 runs) and would OOM below the 8 vCPU tier, so it stays. actions peaks at 1.3 GB and averages 22% CPU over a 39s median run — it drops to 4 vCPU.
  • pii and realtime image builds drop to 2 vCPU. Both ran on 8 vCPU earlier in the window, so the 8→4 step is measured rather than modelled: realtime went 52s → 51s, pii 24s → 27s.
  • Five desktop release jobs drop to 2 vCPU. They peak under 0.4 GB and finish in 4–13s.
  • Those same five jobs get the CI_PROVIDER fallback they were missing, against the invariant stated at the top of ci.yml. In GitHub break-glass mode they would have sat in queued forever.

Why this can't slow CI

Every downsized job sits off its parallel group's critical path, which is set by an app build in each case:

Group Critical path (p50) Job downsized Its p50 Slack
CodeQL js-ts 255s actions 39s 216s
Build AMD64 app 229s realtime / pii 39s / 28s ~190s
Build Dev ECR app 302s realtime / pii 51s / 27s ~250s

Left alone deliberately

  • The 16 vCPU app builds. Memory-bound, not oversized — 16 vCPU measured 2.13x faster and 6% cheaper than 8 vCPU (488s×8 vs 229s×16 vCPU-s). The gain came from memory headroom, so it does not repeat at 32 vCPU.
  • Lint and Test. Genuinely CPU-bound — 62% of the run sits above 80% CPU.
  • Build App on push. build-amd64 has no needs:, so this job is what stops a DB migration applying for a build that cannot ship. It is not redundant.
  • next build's type-check pass. Looks duplicated against turbo run type-check, but apps/sim/tsconfig.json excludes .next, so only the next build pass validates Next's generated route types.

Type of Change

  • Improvement

Testing

actionlint reports strictly fewer findings than before (the five hardcoded labels are now inside expressions) and zero new ones. Verified all 14 matrix rows resolve to a bounded cache value and that no job can reach the unpruned path. Job names are unchanged, and no ruleset requires status checks. bun run lint, check-block-registry, and all 37 check:audits pass.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

The Blacksmith sticky disks backing our docker layer caches had no eviction
policy. setup-docker-builder skips pruning entirely unless max-cache-size-mb
is set, and BuildKit's own GC is time-based only (8 days unused), so on a repo
that builds this often nothing ever aged out: app.Dockerfile/linux-amd64 hit
351 GB within a day of being created, and realtime — an image under 300 MB —
sat at 249 GB. Layer caches alone were 920 GB across ten disks.

Cap them per image via a cache_mb matrix field, alongside the bs_runner field
that already encodes per-image sizing. The app image keeps 100 GB (several
generations over its working set of ~34 layers plus monorepo apt/bun cache
mounts); everything else takes the 25 GB default, which is still 4x the
tightest working set in the matrix (pii, whose spaCy models are ~2.2 GB).
The fallback lives in the composite action rather than an input default,
because an unset matrix key arrives as the empty string and would bypass a
default — silently restoring unbounded growth on any row that forgot the field.

Runner sizes follow measured CPU and memory percentiles over 30 days:

- CodeQL splits per language. javascript-typescript peaks at 19.5 GB so it
  stays on 8 vCPU; actions peaks at 1.3 GB and averages 22% CPU over a 39s
  median run, and drops to 4 vCPU.
- The pii and realtime image builds drop to 2 vCPU. Both already ran on 8 vCPU
  earlier in the window, so the 8->4 step is measured, not modelled: realtime
  went 52s -> 51s and pii 24s -> 27s.
- Five desktop release jobs drop to 2 vCPU. They peak under 0.4 GB and finish
  in 4-13s.

None of these sit on their group's critical path — each has 190-275s of slack
behind an app build that dominates it — so wall-clock is unchanged.

Those five desktop jobs also hardcoded a Blacksmith label with no CI_PROVIDER
fallback, against the invariant stated at the top of ci.yml. In GitHub
break-glass mode they would have sat in `queued` forever; they now fall back
like every other job.

Left alone deliberately: the 16 vCPU app builds (memory-bound, and 16 vCPU
measured 2.1x faster and 6% cheaper than 8 vCPU), Lint and Test (CPU-bound,
62% of the run above 80%), and the push-path Build App (build-amd64 has no
needs:, so it is what stops a migration applying for a build that cannot ship).
@vercel

vercel Bot commented Aug 28, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Ready Ready Preview Aug 28, 2026 8:52am

Request Review

@greptile-apps

greptile-apps Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR bounds Blacksmith Docker layer caches and reduces runner capacity for measured low-resource CI jobs while preserving larger runners for memory-intensive work.

  • Adds per-image Docker cache limits with a 25 GB fallback and a 100 GB app override.
  • Moves lightweight image, desktop orchestration, and CodeQL Actions jobs to smaller Blacksmith runners.
  • Adds GitHub-hosted fallback selection to five desktop-related jobs.

Confidence Score: 5/5

The PR appears safe to merge, with no concrete changed-code defect established.

The matrix values resolve to bounded cache sizes, the new runner expressions follow established provider-selection conventions, and the changed runner labels are already used elsewhere in the repository.

Important Files Changed

Filename Overview
.github/actions/docker-build/action.yml Adds an optional cache-size input and supplies a bounded 25 GB fallback to the Blacksmith builder setup.
.github/workflows/ci.yml Configures a 100 GB app cache override, passes cache limits into all image builds, and right-sizes several image and desktop orchestration runners.
.github/workflows/codeql.yml Replaces the fixed Blacksmith runner with a valid include matrix that retains the larger tier for JavaScript/TypeScript and uses a smaller tier for Actions analysis.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  Matrix[Image matrix] --> Cache{cache_mb set?}
  Cache -->|App| Large[100 GB cache cap]
  Cache -->|Other images| Default[25 GB fallback]
  Large --> Builder[Blacksmith Docker builder]
  Default --> Builder
  Provider{CI_PROVIDER}
  Provider -->|Unset or blacksmith| Sized[Per-job Blacksmith runner]
  Provider -->|Other value| GitHub[GitHub-hosted fallback]
  Sized --> Jobs[Build and CI jobs]
  GitHub --> Jobs
Loading

Reviews (1): Last reviewed commit: "improvement(ci): bound docker layer cach..." | Re-trigger Greptile

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 3 files

Confidence score: 3/5

  • In .github/actions/docker-build/action.yml, the pinned builder revision does not read max-cache-size-mb, so both the matrix value and 25600 fallback are ignored and the intended cache-size control is not applied. Pin a builder revision that supports this input or perform the pruning through a supported mechanism.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".github/actions/docker-build/action.yml">

<violation number="1" location=".github/actions/docker-build/action.yml:69">
P1: With the pinned builder revision, this `with` key is not a supported or read input, so the matrix value and `25600` fallback are ignored. Pin a builder revision that implements `max-cache-size-mb`, or perform the prune in this workflow, before relying on the cache cap.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic

uses: useblacksmith/setup-docker-builder@a5256a73e30f09e37e3eceb8ca36043d17621d24 # v2
with:
cache-key: ${{ steps.cache-key.outputs.value }}
max-cache-size-mb: ${{ inputs.max-cache-size-mb || '25600' }}

@cubic-dev-ai cubic-dev-ai Bot Aug 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: With the pinned builder revision, this with key is not a supported or read input, so the matrix value and 25600 fallback are ignored. Pin a builder revision that implements max-cache-size-mb, or perform the prune in this workflow, before relying on the cache cap.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At .github/actions/docker-build/action.yml, line 69:

<comment>With the pinned builder revision, this `with` key is not a supported or read input, so the matrix value and `25600` fallback are ignored. Pin a builder revision that implements `max-cache-size-mb`, or perform the prune in this workflow, before relying on the cache cap.</comment>

<file context>
@@ -42,11 +49,24 @@ runs:
       uses: useblacksmith/setup-docker-builder@a5256a73e30f09e37e3eceb8ca36043d17621d24 # v2
       with:
         cache-key: ${{ steps.cache-key.outputs.value }}
+        max-cache-size-mb: ${{ inputs.max-cache-size-mb || '25600' }}
 
     - name: Build and push (Blacksmith)
</file context>
Fix with cubic

@waleedlatif1
waleedlatif1 merged commit f775fe0 into staging Aug 28, 2026
30 checks passed
@waleedlatif1
waleedlatif1 deleted the ci/rightsize-runners-and-cap-cache-storage branch August 28, 2026 09:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant