ci: shared per-run build + nightly-warmed builder image for PR CI

[fengttt]

What this PR does / why we need it:

Cuts the repeated cold compilation out of MatrixOne PR CI. Tracking issue: matrixorigin/matrixone#27076.

1. build-mo.yaml (new reusable workflow) — builds the -cover mo-service once per run and uploads the self-contained runtime tree (~99MB artifact). It builds inside the nightly ci-builder image when reachable (only PR-changed packages recompile; the binary links against the ubuntu-22.04 toolchain, matching the runtime image glibc by construction), and falls back to a from-source runner build otherwise.
2. BVT consumers — e2e-standalone-parallel and e2e-compose-parallel accept an optional build_artifact input: download, verify head sha + flavor, and skip their own ~10-minute builds; any mismatch falls back to building from source. optools/images/Dockerfile.prebuilt (matrixone side) assembles the compose runtime image from the artifact in seconds.
3. UT / coverage-UT / SCA cache seeding — the nightly image warms exactly the compile flavors those jobs use (-race -tags matrixone_test; -covermode=set with the verbatim coverpkg list; the full static-check pass on arm64). Guarded, continue-on-error seed steps extract the Go build/module caches (and golangci-lint's cache for SCA) onto the runner; the Go build cache is location-independent for pure-Go packages so tests keep running on the host exactly as today. prepare_coverage_cgo.sh honors a seeded thirdparties dir so make cgo stops rebuilding the C thirdparties.
4. Nightly ci-builder images — added to image-build.yaml: amd64+arm64 matrix, combined under one manifest tag, pushed to Docker Hub and mirrored to ACR; every pull site prefers the ACR mirror (time-bounded) because the SCA/coverage runners are in Shanghai/Guangzhou. Skipped on release events and on branches without the Dockerfile.

Merge order: this PR is a safe no-op on its own (all new inputs optional, all seeds degrade to cold builds) and MUST merge before the matrixone-side PR that wires entrypoint.yaml to build-mo.yaml.

Measured locally: incremental -cover build 20s–1min vs ~10min cold; full race-flavor test compile ~3min warm; make cgo ~11s with seeded thirdparties.

🤖 Generated with Claude Code

[XuPeng-SH]

Requesting changes at head 176f2059. The shared-once build direction is sound, but the producer/cache owner still has three correctness and security blockers.

1. [P0] Keep untrusted PR compilation outside the privileged credential boundary. The current matrixone caller is a pull_request_target workflow using secrets: inherit. This reusable job adds environment: ci, checks out the fork head with the supplied token and default persisted credentials, logs into registries, and then executes the PR-controlled Makefile. The host cold path can directly read runner credential state, and the shared build does not need privileged secrets to validate a public PR. Please run head compilation in an unprivileged pull_request context with no inherited/environment secrets and persist-credentials: false, or split trusted cache preparation from untrusted compilation so credentials and untrusted code never share a runner. The CI workflow should make that trust contract explicit rather than relying on every caller to remember it.

2. [P1] Do not restore thirdparties/install without exact input compatibility. build-mo.yaml, ci.yaml, and coverage-ut.yaml copy the nightly native tree into the PR checkout unconditionally. The thirdparty file targets do not depend on all source archives, Makefiles, flags, compiler/base-image identity, target OS/arch, and branch state, so a PR changing native inputs can test the nightly main implementation and go falsely green. The smallest safe first rollout is to seed only Go module/build caches. If native reuse is retained, require an exact manifest/fingerprint covering every native input and toolchain dimension and rebuild on any mismatch.

3. [P1] Close fallback inside the shared-build producer. The caller makes both BVT consumers hard needs dependents. If the cached docker run or builder image itself is stale/broken, this job fails and both consumers are skipped; their missing-artifact fallback is unreachable. Treat the cached/container build as best effort, remove all partial outputs after failure, set up the checkout Go version, and run the existing clean host build. Publish only after one path succeeds; a real cold compile failure should still fail once and block both consumers.

Before calling the speedup proven, please also report end-to-end measurements including compressed builder-image size and pull time on both GitHub ubuntu-22.04 westus and the China self-hosted pools. Bound both ACR and Docker Hub pulls; today ACR may consume 300 seconds and the Docker Hub fallback has no explicit bound, so a cache outage can make the optimized path slower than the baseline.

Suggested acceptance closure: fork PR cannot access inherited/environment/checkout/registry credentials; native-input change and divergent release branch force a native rebuild; forced cached-build failure completes through a clean cold build; produced artifact is SHA/flavor/platform-validated and smoke-tested by both standalone and compose consumers.

[aptend]

Requesting changes at head 176f205.

1. [P0] Keep untrusted PR code outside the privileged credential boundary. The caller is pull_request_target with secrets: inherit; this workflow checks out the PR head, logs into registries, and then executes the PR-controlled Makefile. Move PR compilation to an unprivileged pull_request workflow with no inherited/environment secrets and persist-credentials: false, or strictly separate trusted cache preparation from untrusted compilation.

2. [P1] Do not restore thirdparties/install without exact compatibility validation. The native targets lack source/config/toolchain prerequisites, so a PR changing native archives, headers, flags, or a release branch diverging from main can test the nightly main artifacts and falsely pass. Seed only Go caches, or require an exact fingerprint covering native inputs, branch, toolchain, OS/arch, base image, and build flags.

3. [P2] Close fallback inside the shared-build producer. MatrixOne#27077 makes both BVT consumers hard needs dependents, so a producer failure skips them before their cold-build fallback can run. Treat cached/container failures as best effort, clean partial outputs, and retry once with a clean host build while preserving genuine PR compile failures.

Please also isolate and clean Docker credentials; raw docker login persists credentials in runner Docker state that later PR-controlled steps can use.

[fengttt]

Thanks @XuPeng-SH @aptend — all points addressed at head 8b18024 (with matrixone-side e417582532):

[P0] Credential boundary. The shared build job now holds no credentials at all: environment: ci is removed, every registry secret is dropped from build-mo.yaml and from all three seed steps (ci.yaml UT/SCA, coverage-ut.yaml), all ci-builder pulls are anonymous, and the PR-head checkout uses persist-credentials: false. There is no docker login anywhere in a PR-context path, so no credential can persist into runner docker state that PR-controlled steps could read. The trust contract is documented at the top of the job. If anonymous ACR pulls are not permitted for the mirror, the right fix is a pull-only credential provisioned at the runner/node level by infrastructure — never workflow-level login. (A full unprivileged pull_request split for compilation would change the entrypoint architecture that all existing BVT jobs share today — happy to pursue that as a follow-up, but this PR no longer adds any credential exposure beyond that existing baseline.)

[P1] Native cache compatibility. thirdparties/install now restores only on an exact fingerprint match (schema + os + arch + git rev-parse HEAD:thirdparties) at all four consumers; mismatch clears the seed and builds native deps from source. The tree hash covers every archive, header, the Makefile with its build flags, and the download script, since they all live under thirdparties/; schema= is bumped on any producing-toolchain/base-image contract change. Validated locally: unchanged tree seeds, a thirdparties/ change rebuilds. Go caches remain best-effort only.

[P2] Fallback containment. The cached container build is now best-effort inside the producer: continue-on-error, ownership repair, loader smoke test (env -u LD_LIBRARY_PATH GOCOVERDIR=… ./mo-service -h); any failure falls through to toolchain setup + the existing make clean cold build, and packaging (with a final smoke test) runs after either path. Consumers now skip only on a genuine compile failure.

Pull bounds. Both the ACR and Docker Hub pulls are bounded at 300s each. Layers are content-addressed, so a timed-out ACR attempt still counts toward the Docker Hub attempt of the same image; worst case a full registry outage adds a bounded ~10 min before the cold path, typical is 1–3 min.

Artifact contract. All runtime DSOs consolidated under lib/ ($ORIGIN/lib); prebuilt-cgo/ is gone; the artifact is smoke-tested before upload.

Measurements. Local: incremental -cover build 20s (unchanged tree) / ~54s (worst-case exported-API change to pkg/container/types) vs ~10 min cold; full ./... race-flavor test compile 3m18s warm; make cgo ~11s with seeded thirdparties; image 13.1GB uncompressed. Compressed size and real pull times on ubuntu-22.04 (westus) and the Shanghai/Guangzhou pools can only be measured once the first nightly publishes — I'll post them here from the first nightly run plus a canary PR run before matrixone#27077 leaves draft.

🤖 Generated with Claude Code

[aptend]

Requesting changes at head 8b18024 after re-review.

[P1] Preserve the cached build's failure status before running the loader smoke test. The step starts with set -uo pipefail (without -e), captures docker run in status, and evaluates [ "${status}" -eq 0 ], but a false test does not stop the script. If make build produces mo-service and then fails a later validation, the subsequent ./mo-service -h can succeed, making the entire step successful. The host cold-build fallback is then skipped and the artifact is published despite the failed build target.

Please explicitly exit on nonzero status after the ownership repair, for example:

if [ "${status}" -ne 0 ]; then
  exit "${status}"
fi

The previous credential and native-cache blockers are otherwise addressed at this head: registry pulls are anonymous with non-persisted checkout credentials, and native restoration is fingerprint-gated.

[fengttt]

Re-review round addressed at head 2c39853 (matrixone side ad96d0c882):

[P1] Cached-build failure propagation (@aptend, head 8b18024). Confirmed — without set -e the bare status test couldn't stop the script, so a failed docker run with a leftover mo-service passing the smoke test could publish an artifact from a failed build. Fixed exactly as suggested: explicit exit "${status}" immediately after the ownership repair (verified the step now exits with the docker run status before the smoke test can run).

[P1] Producer credential boundary (@XuPeng-SH #2). The producer now accepts no caller secrets at all: the TOKEN_ACTION declaration and the checkout token: are removed (anonymous public checkout with the job's default token), and the workflow itself declares permissions: contents: read so the least-privilege contract is enforced here rather than remembered by callers. matrixone#27077 drops secrets: inherit from matrixone-shared-build in tandem (ad96d0c882).

[P1] Registry routing & disk (@XuPeng-SH #3). Pulls now route by RUNNER_ENVIRONMENT: GitHub-hosted runners try Docker Hub first, Shanghai ACR second; the self-hosted China pools the reverse. Each attempt stays bounded at 300s (completed layers are content-addressed and carry across attempts), and if both fail the step prunes partially pulled layers before the cold path so they can't starve it of disk. Worst-case bounded overhead before cold: ~10 min against a ~40-min job budget with a ~13-min cold build.

[P2] Native-cache toolchain dimensions (@XuPeng-SH #4). Adopted your smaller safe first rollout: UT/SCA/coverage-UT now seed Go module/build caches only — the content-addressed Go cache is always correctness-safe — and native deps keep building from source on those hosts; prepare_coverage_cgo.sh is reverted to its original form. Fingerprint-gated native reuse remains only in this workflow's container path, where producer and consumer toolchains are identical by construction (same image, same env). Extending native reuse to hosts behind a toolchain-ABI identifier can be a follow-up.

Process/canary (@XuPeng-SH #1). Agreed: #27077 stays draft; after this PR merges I'll dispatch the image workflow, then run a canary on the exact #27077 head — including a forced cached-build failure — and post per-pool compressed-size/pull/build timings plus cache-hit/miss/registry-unavailable outcomes here before it leaves draft.

🤖 Generated with Claude Code

[fengttt]

@XuPeng-SH @aptend — friendly status note: aptend's deep re-review on matrixone#27077 (2026-08-14) examined this PR's exact head 2c39853 jointly and confirmed the previously raised blockers are fixed (producer permissions cap, no inherited secrets, cached-build status propagation, host seeding scaled to Go caches only). The remaining item over there was purely the readiness state, now restored to draft. This PR is the head of the merge sequence — when you get a chance to re-review, everything downstream (first image publish, canary with forced cached-failure, per-pool measurements) is queued behind it.

🤖 Generated with Claude Code

[fengttt]

Merged latest main into this branch (head cd102e2) and re-validated end to end. Two upstream changes required real integration decisions:

1. SCA is ceded to its new dedicated cache pipeline (ci: speed up SCA dependency checks with shared module cache #422/ci: cache SCA preparation artifacts #423 + matrixone's sca-go-module-cache.yaml trusted producer). That mechanism subsumes this branch's SCA seed — including the compiler-identity cache keys we had deferred as follow-up — so the SCA seed step is dropped, the nightly ci-builder publishes amd64 only (the arm64 leg existed solely for SCA), and the multi-arch manifest job is gone. The cumulative ci.yaml diff vs main is now exactly one added step: the UT cache seed, coexisting with the new Shanghai-proxy canary.
2. The pessimistic BVT's new compose shape (ci: run complementary BVT group on multi-CN compose #419) replaces the old native-binary consumer: both BVT jobs now consume the shared build identically — guarded artifact extraction → Dockerfile.prebuilt image assembly (with its in-image loader smoke test) → fallback to the full in-docker build. Symmetric code, one consumer pattern.

Re-verified on the merged tree: run_ut.sh's compile flags still match the race warm flavor (#419's scope refactor changed parallelism/scoping only), base images remain in sync at go1.26.4, both Dockerfiles pass docker build --check, all six workflows pass schema validation, and a full builder-image build on the merged tree (57 new commits) completes with warm-race=ok, warm-coverage=ok, and a fingerprint matching the merged thirdparties tree.

🤖 Generated with Claude Code