ci: build TKE images from shared ci-builder

[daviszhen]

What this PR does

• Publishes the nightly ci-builder image with immutable tags and an optional Tencent TCR mirror for Shanghai TKE runners.
• Reuses the prewarmed builder in merge-trigger-tke.yaml to compile MatrixOne with the Go/native caches, then assembles the runtime image with Dockerfile.prebuilt.
• Verifies the shared native fingerprint before restoring thirdparties/install; mismatches and builder outages retain the source Dockerfile fallback.
• Keeps the existing build-mo.yaml path compatible with MatrixOne revisions that do not yet contain the fingerprint helper.

The companion MatrixOne change adds optools/images/ci-builder-fingerprint.sh and updates Dockerfile.ci-builder. The CI workflow remains safe to merge first because it falls back to the legacy schema until that helper is present.

Validation

• Parsed all modified workflows with PyYAML.
• Shell-checked every run block after replacing GitHub expressions.
• git diff --check passed.
• Docker Buildx --check passed for the builder and runtime Dockerfiles.

This is intended to stop every TKE build from re-downloading stable Go/native dependencies while preserving correctness and an explicit cold-build fallback.

[daviszhen]

merge-trigger-tke-basic-image-plan.md

[loveRhythm1990]

Reviewed together with matrixorigin/matrixone#27330. The fingerprint-gated restore is sound and the fallback structure is right, but two issues in merge-trigger-tke.yaml need fixing before this lands — one makes the optimisation a no-op, the other can publish a broken runtime image.

1. docker run is missing -i, so the cached build never runs (blocking)

docker run --rm \
  ... \
  "${{ steps.builder.outputs.ref }}" bash -s <<'CONTAINER_SCRIPT'

Without -i the container's stdin is not attached, so bash -s hits EOF immediately, does nothing, and exits 0. status is 0, the script falls through, and then

env -u LD_LIBRARY_PATH GOCOVERDIR="$(mktemp -d)" ./mo-service -h >/dev/null

fails because no binary was produced. The step fails, continue-on-error swallows it, and the source Dockerfile fallback runs. Net effect: every TKE build still compiles from source and this PR delivers no speedup — it just adds a container round-trip first.

The existing build-mo.yaml path uses bash -c '...', which does not depend on stdin, which is why that one works. Either add -i or switch this to bash -c.

2. set +e still in effect during artifact staging — can push an image missing .so files (correctness)

set -u +e -o pipefail is set before docker run, and set -e is never restored after the status check. Everything after it runs with errexit off, including:

cp ./lib/*.so ./cgo/*.so "$GITHUB_WORKSPACE/mo-runtime/lib/"

If either glob fails to match or the copy errors, it is silently ignored. The step's exit status is then decided by the final BUILD_INFO heredoc, which always succeeds — so steps.cached.outcome == 'success', and Build and Push Image from CI builder assembles a runtime image from a mo-runtime/ directory with no shared libraries and pushes it to the BVT consumers.

The mo-service -h loader smoke test catches most breakage, but it runs before the staging copies, so it cannot catch this. Please re-enable set -e immediately after the status check:

if [ "${status}" -ne 0 ]; then
  exit "${status}"
fi
set -e

This is the only finding across either PR that can produce a wrong artifact rather than just a slow one.

3. github.ref == 'refs/heads/main' gate in image-build.yaml

image-build.yaml is a workflow_call workflow, so github.ref resolves to the caller's ref. MatrixOne's image-build.yml triggers on schedule (which runs on the default branch), so the nightly path is fine. But the same workflow also has workflow_dispatch: a manual run from any non-main branch will now silently skip builder publication, on top of the existing gate skip. If that is intentional — builder images should only come from trusted main — please emit a ::notice:: on the skip path, otherwise "why did the builder image stop updating" will be painful to diagnose later.

4. legacy fingerprint acceptance

Correct and not a staleness risk: the schema-1 tree hash is actually stricter for thirdparties than the new content hash, and the compiler is whatever the image itself ships, so it is consistent by construction. But there is no trigger for removing it. Please file a follow-up to delete the legacy branches in both build-mo.yaml and merge-trigger-tke.yaml once the first schema-2 nightly image exists.

Merge order

Both sides are bidirectionally compatible so either order is safe. Suggest landing matrixorigin/matrixone#27330 first and letting one nightly publish a schema-2 image, then landing this — otherwise CI takes the fallback path throughout and neither the cache hit rate nor the fingerprint determinism concerns raised on that PR get exercised.

The sed that quotes GOPROXY in the fallback path is the right compatibility shim for the chained http://...|https://goproxy.cn|direct proxy against older MatrixOne checkouts; it can be dropped once #27330 has landed everywhere.

[daviszhen]

Addressed the review findings in 95d9633.

• merge-trigger-tke.yaml now passes -i to docker run, so the bash -s build script receives the heredoc and actually runs.
• Host-side set -e is restored immediately after the cached build status check, so staging failures cannot publish an incomplete runtime image.
• The ci-builder job now runs the cheap gate on non-main refs and emits an explicit notice instead of silently skipping; publication remains restricted to refs/heads/main.
• Fingerprint logs now show both computed and cached values, and rollout comments track the schema-3 contract.

The schema-1 compatibility removal is tracked in CI#427. Please re-review the updated PR.

[loveRhythm1990]

Re-reviewed at 95d9633 ("ci: fix cached TKE build execution"). Both issues from my earlier review are fixed — docker run now has -i, and set -e is restored after the status check. Thanks.

New findings below: 1 blocker (P0) and 3 P1s.

---

P0 — Non-main branches lose their entire post-merge TKE regression, with no fallback

Verified against the API:

branch	merge-trigger-tke routes to	Dockerfile.prebuilt	ci-builder-fingerprint.sh
main	CI@main	present	present
4.2-dev / 4.1-dev / 4.0-dev / 2.2-dev	CI@main	404	404
3.0-dev	CI@release/3.0-dev (unaffected)	404	404

MatrixOne's merge-trigger-tke.yaml routes with if: github.base_ref != '3.0-dev' → @main, so every 4.x / 2.x branch picks up this PR, but none of those checkouts contain optools/images/Dockerfile.prebuilt.

Failure chain for a merge into e.g. 4.2-dev:

1. Resolve CI builder image pulls the main-built ci-builder from TCR → ref is non-empty.
2. Build MatrixOne with CI builder takes the legacy-fingerprint branch (the missing helper is handled), mismatches the schema-3 fingerprint, rebuilds native deps from source, and make build succeeds → cached.outcome == 'success'.
3. Build and Push Image from CI builder points file-path at optools/images/Dockerfile.prebuilt, which does not exist on that branch → the step fails hard.
4. That step has no continue-on-error, and the source fallback is gated on steps.cached.outcome != 'success', which is false here → the fallback is skipped.
5. docker_image_build fails → setup_mo_test_env's needs.docker_image_build.result == 'success' gate is false → BVT, SSB/TPCH, sysbench and TPCC never run.

So the post-merge regression is silently disabled on those branches.

Suggested fix — add a presence gate in Resolve CI builder image so the cached path only engages on checkouts that can actually complete it:

if [ ! -f "$GITHUB_WORKSPACE/matrixone/optools/images/Dockerfile.prebuilt" ]; then
  echo "::notice::Dockerfile.prebuilt not present in this checkout; using the source Dockerfile"
  echo "ref=" >> "$GITHUB_OUTPUT"
  echo "tag=" >> "$GITHUB_OUTPUT"
  exit 0
fi

Alternatively, be conservative for the first rollout and gate the whole cached path on if: github.base_ref == 'main'. That also resolves P1-3 below.

---

P1-1 — The prebuilt and source paths share one registry cache ref and clobber each other

Both Build and Push Image from CI builder and Build and Push Source Image Fallback write to:

cache-to: type=registry,ref=${{ steps.prep.outputs.docker_image }}:cache-${{ github.base_ref }},mode=max

• The cached path builds Dockerfile.prebuilt — four trivial COPY layers, essentially worthless to cache.
• The fallback path builds the full Dockerfile — ONNX download, C/C++ compilation, full Go build. Very expensive to cache.

With mode=max, the former overwrites the latter. Every successful cached run wipes the source-build layer cache, so when the fallback is actually needed (builder outage, registry unreachable) it starts completely cold — the slowest possible moment to lose the cache.

Suggest giving the prebuilt path its own ref (:cache-prebuilt-${{ github.base_ref }}), or simply dropping cache-to for it.

---

P1-2 — Go build-cache flavor mismatch: most of the intended speedup does not materialise

optools/images/Dockerfile.ci-builder:43 warms the cache with:

RUN make build GOBUILD_OPT=-cover

but this PR runs a plain make build (no -cover) inside the container. Coverage instrumentation changes the build ID of nearly every package, so /root/.cache/go-build misses across essentially the whole dependency graph for this consumer.

What this PR actually saves is: the Go module downloads (GOMODCACHE), thirdparties/install when the fingerprint matches, and the base image layers. Go compilation time is essentially unchanged — and that is the dominant cost of a TKE build.

This is worth aligning before merge, because as written the result does not match the PR description ("stop every TKE build from re-downloading stable Go/native dependencies"): the downloads are indeed avoided, the compiles are not. The fix is on the MatrixOne side — warm a plain flavor as well:

RUN make build GOBUILD_OPT=-cover
RUN make build            # flavor consumed by merge-trigger-tke and the product image

The cost is a larger builder image and a longer nightly build.

---

P1-3 — Without TCR secrets provisioned, this is a net loss

tcr_enabled depends on the optional TCR_USERNAME / TCR_TOKEN. If those are not yet configured on the MatrixOne repo:

• Publish Tencent CI-builder mirror is skipped silently, so ccr.ccs.tencentyun.com/matrixone-dev/matrixone:ci-builder-amd64 never exists;
• every TKE merge then burns the full timeout 300 docker pull before falling back to the source path — five minutes added to every post-merge run, with zero benefit.

Please make "secrets provisioned and one nightly has successfully pushed to TCR" an explicit merge precondition, and/or lower that first-pull timeout to 60–90s so the failure mode is cheap.

---

P2 (non-blocking)

• docker image inspect --format '{{index .RepoDigests 0}}': on a long-lived dind runner the same image ID can carry digests for several repositories, so index 0 is not guaranteed to be the TCR one. Filtering by repository name would be more robust.
• The schema-1 legacy acceptance branch is safe today (the old tree hash is stricter for thirdparties, and the toolchain comes from the image itself), but it has no removal trigger. Worth a follow-up issue to drop it once every active branch carries ci-builder-fingerprint.sh.

[daviszhen]

Follow-up: 456aff6 raises the nightly ci-builder timeout to 180 minutes so both the coverage and plain warm builds can complete. The companion MatrixOne PR #27388 supplies the plain make build layer consumed by the TKE workflow. Latest CI head is 456aff6; please re-review.

[daviszhen]

Final CI head is 69822c4. Added an explicit warning when TCR credentials are absent, while keeping the 90-second pull timeout and source fallback. Workflow YAML and all 42 embedded shell blocks parse successfully; the validation check is green. Please re-review CI #426 with MatrixOne companion PR #27388.

[aptend]

Reviewed against current MatrixOne main and companion matrixorigin/matrixone#27388. The current main already contains the prebuilt runtime Dockerfile, schema-3 native fingerprint contract, reusable workflow callers, and safe source fallback expected by this change. Builder lookup is digest-pinned after pull, native restoration is fingerprint-gated, and cached-build/staging failures fail closed or fall back to the source path. The remaining plain-build cache warm is explicitly supplied by #27388; without it the impact is reduced cache hit rate rather than correctness risk. No blocking findings.