Skip to content

ci: persist ccache across ephemeral runner pods via actions/cache - #7809

Open
MrLi000001 wants to merge 5 commits into
deepmodeling:developfrom
MrLi000001:ci/ccache-actions-cache
Open

ci: persist ccache across ephemeral runner pods via actions/cache#7809
MrLi000001 wants to merge 5 commits into
deepmodeling:developfrom
MrLi000001:ci/ccache-actions-cache

Conversation

@MrLi000001

Copy link
Copy Markdown

Reminder

  • I have read AGENTS.md and docs/developers_guide/agent_governance.md.
  • I have linked an issue or explained why this PR does not need one.
  • I have added adequate unit tests and/or case tests, or explained why not.
  • I have listed the exact verification commands run and their results.
  • I have described user-visible behavior changes, including INPUT parameter changes.
  • I have explained core-module impact for ESolver, HSolver, ElecState, Hamilt, Operator, Psi, or other source/ changes.
  • I have requested any needed governance exception below.

Linked Issue

This PR does not need an issue: it is a CI-infrastructure-only change. The motivation is
backed by 1683 workflow runs (2026-04 to 2026-08): ARC runner pods rebuild from an empty
ccache on every job (build median 2021s / ~34min) while VM runners hit their warm local
cache (median 155s) — a 13x gap that is growing as the pod share rises (16% in June to
39% in August). The pod cache is lost because ephemeral pods are recreated per job and
/tmp/ccache lives in the pod's temporary storage.

Unit Tests and/or Case Tests for my changes

  • Commands run:
    • Local YAML parse of both modified workflow files:
      python -c "import yaml; yaml.safe_load(open('.github/workflows/test.yml')); yaml.safe_load(open('.github/workflows/cuda.yml'))"
    • Local zstd pack/unpack roundtrip on the ccache directory (tar + zstd -T0).
  • Result summary: YAML validation passed; pack/unpack roundtrip preserves ccache -s stats.
  • Checks not run, with reason: no local full build — this is a workflow-only change and the
    GitHub Actions run itself is the verification; expected CI behavior (documented below)
    will be confirmed by the run logs after merge.

What's changed?

  • Persist the ccache directory of test.yml / cuda.yml through the GitHub Actions Cache
    Service so ephemeral ARC runner pods no longer start every job with a cold cache.
  • Restore the cache only on ARC pods (if: startsWith(runner.name, 'gpu-runner-')); VM
    runners keep their warm local cache untouched and only upload it as the cloud seed.
  • Pack ccache as a zstd tarball before upload (~4.3G -> ~600M) to stay within the 10GB free
    cache quota; unpack after restore with ownership fix.
  • Cache key: ccache-<toolchain>-${{ github.sha }} with
    restore-keys: ccache-<toolchain>- — exact key for re-runs, prefix fallback for new
    commits/PRs. runner.name is intentionally absent from the key because pod names are
    random per job (would always miss) and VM/pod caches should be shared.
  • Set CCACHE_BASEDIR: ${{ github.workspace }} so cached objects remain portable across
    checkout paths.
  • Print RUNNER_NAME / runner type in the logs to allow timing statistics grouped by
    runner type (pod vs vm).
  • Expected CI behavior: run Merge recent abacus changes to deepmodeling #1 after merge = seed (restore miss, full build, upload);
    subsequent runs = restore hit, pod build step from ~2021s down to the minute scale.

Governance Notes

  • INPUT/docs changes: none.
  • Core module impact: none — workflow-only change.
  • Exceptions requested: none.

@MrLi000001
MrLi000001 force-pushed the ci/ccache-actions-cache branch from 8bd3dac to d99f43f Compare August 12, 2026 17:57
# Conflicts:
#	.github/workflows/cuda.yml
@MrLi000001 MrLi000001 closed this Aug 13, 2026
@MrLi000001 MrLi000001 reopened this Aug 13, 2026
- Unpack: replace cache-hit=='true' gate with a file-existence check.
  actions/cache outputs cache-hit=false on restore-keys prefix hits even
  though the cache IS downloaded, so the old condition skipped unpacking
  and every build ran fully cold despite downloading 1.2GB.
- Restore: match all ephemeral ARC runners by excluding VM name prefix
  (iZ0x) instead of only gpu-runner-*, which missed cpu-runner-* pods.
@MrLi000001 MrLi000001 closed this Aug 13, 2026
@MrLi000001 MrLi000001 reopened this Aug 13, 2026
@MrLi000001 MrLi000001 closed this Aug 13, 2026
@MrLi000001 MrLi000001 reopened this Aug 13, 2026
@MrLi000001 MrLi000001 closed this Aug 13, 2026
@MrLi000001 MrLi000001 reopened this Aug 13, 2026
@MrLi000001 MrLi000001 closed this Aug 13, 2026
@MrLi000001 MrLi000001 reopened this Aug 13, 2026
@MrLi000001 MrLi000001 closed this Aug 13, 2026
@MrLi000001 MrLi000001 reopened this Aug 13, 2026
@MrLi000001 MrLi000001 closed this Aug 13, 2026
@MrLi000001 MrLi000001 reopened this Aug 13, 2026
@MrLi000001 MrLi000001 closed this Aug 13, 2026
@MrLi000001 MrLi000001 reopened this Aug 13, 2026
Restore ccache unconditionally so every runner (VM/pod) attempts a
restore; the previous ${{ !startsWith(runner.name, 'iZ0x') }} guard
was not evaluated reliably and the guard provides no benefit (a VM
restore hit simply refreshes local cache from the cloud).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant