Skip to content

STAC-25457 Publish and sign the agent and cluster-agent images from GitHub Actions #454

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
LouisParkin wants to merge 3 commits into
base: STAC-25142-agent-lint-unit
Choose a base branch
from
Open

STAC-25457 Publish and sign the agent and cluster-agent images from GitHub Actions #454

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
98efcef
STAC-25457: publish and sign the agent and cluster-agent images
LouisParkin Aug 5, 2026
93709cd
STAC-25457 Gate agent image publication on scans, deb branding, and p…
LouisParkin Aug 6, 2026
61e2bce
STAC-25457 Manage the agent image CVE backlog with time-limited excep…
LouisParkin Aug 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
120 changes: 117 additions & 3 deletions .github/workflows/build-binaries.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,7 @@ jobs:
runs-on: ${{ matrix.runner }}
timeout-minutes: 45
env:
LOCAL_IMAGE: stackstate-cluster-agent:ci-${{ matrix.arch }}
LOCAL_IMAGE: quay.io/stackstate/stackstate-k8s-cluster-agent:ci-${{ matrix.arch }}
steps:
- name: Check out repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
Expand All @@ -209,8 +209,6 @@ jobs:
REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
run: |
set -eo pipefail
# Dockerfiles/cluster-agent pulls its ubuntu builder stage through the
# proxy; the BCI stages come from registry.suse.com and need no auth.
printf '%s' "${REGISTRY_PASSWORD}" | docker login -u "${REGISTRY_USER}" --password-stdin "${REGISTRY_HOST}"

- name: Build cluster-agent image
Expand Down Expand Up @@ -239,6 +237,120 @@ jobs:
--entrypoint /opt/stackstate-agent/bin/stackstate-cluster-agent/stackstate-cluster-agent \
"${LOCAL_IMAGE}" version

- name: Scan cluster-agent image (Trivy and Grype vulnerabilities, VEX-aware, plus Trivy secrets)
uses: StackVista/image-pipeline/.github/actions/scan-image@6284a6fc006a7cc46a7f00d02c50d5f21b117b63
with:
image: ${{ env.LOCAL_IMAGE }}
mode: gate
severity: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
with-grype: true
exceptions-path: exceptions
upload-sarif: false
sarif-category: stackstate-k8s-cluster-agent-${{ matrix.arch }}

publish-cluster-agent-image:
name: Publish and sign cluster-agent image (${{ matrix.arch }})
if: github.event_name == 'push'
needs: build-cluster-agent-image
strategy:
fail-fast: false
matrix:
include:
- arch: amd64
runner: docker-public
- arch: arm64
runner: arm64-xlarge-public
runs-on: ${{ matrix.runner }}
timeout-minutes: 45
permissions:
contents: read
id-token: write
env:
IMAGE: quay.io/stackstate/stackstate-k8s-cluster-agent
steps:
- name: Check out repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false

- name: Download cluster-agent binary
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
with:
name: cluster-agent-binary-${{ matrix.arch }}

- name: Stage the cluster-agent binary in the image build context
run: |
set -eo pipefail
chmod +x bin/stackstate-cluster-agent/stackstate-cluster-agent
cp -r bin/stackstate-cluster-agent Dockerfiles/cluster-agent/

- name: Resolve image tag
id: image
env:
SOURCE_SHA: ${{ github.sha }}
run: |
set -euo pipefail
echo "tag=$(printf '%s' "${SOURCE_SHA}" | cut -c1-8)" >> "${GITHUB_OUTPUT}"

- name: Resolve canonical OCI labels
id: oci
uses: StackVista/image-pipeline/.github/actions/apply-oci-labels@6284a6fc006a7cc46a7f00d02c50d5f21b117b63
with:
image-name: stackstate-k8s-cluster-agent
tag: ${{ steps.image.outputs.tag }}
title: SUSE Observability Cluster Agent
description: Cluster-level agent collecting Kubernetes topology and cluster checks for SUSE Observability.
component: stackstate-k8s-cluster-agent
dockerfile: Dockerfiles/cluster-agent/Dockerfile
base-name: registry.suse.com/bci/bci-micro:latest
registry-credentials: |
[{"registry": "${{ vars.REGISTRY_HOST }}", "username": "${{ vars.REGISTRY_USER }}", "password": "${{ secrets.REGISTRY_PASSWORD }}"}]

- name: Build, publish, and sign architecture image
uses: StackVista/image-pipeline/.github/actions/push-single-arch@6284a6fc006a7cc46a7f00d02c50d5f21b117b63
with:
image: ${{ env.IMAGE }}
tag: ${{ steps.image.outputs.tag }}
arch: ${{ matrix.arch }}
docker-context: Dockerfiles/cluster-agent
dockerfile: Dockerfiles/cluster-agent/Dockerfile
labels: |
${{ steps.oci.outputs.labels }}
org.opencontainers.image.revision=${{ github.sha }}
source-registry-credentials: |
[{"registry": "${{ vars.REGISTRY_HOST }}", "username": "${{ vars.REGISTRY_USER }}", "password": "${{ secrets.REGISTRY_PASSWORD }}"}]
target-registry: quay.io
target-registry-user: ${{ vars.QUAY_USER }}
target-registry-password: ${{ secrets.QUAY_PASSWORD }}

merge-cluster-agent-manifest:
name: Publish and sign multi-architecture cluster-agent image
if: github.event_name == 'push'
needs: publish-cluster-agent-image
runs-on: docker-public
timeout-minutes: 30
permissions:
contents: read
id-token: write
steps:
- name: Resolve image tag
id: image
env:
SOURCE_SHA: ${{ github.sha }}
run: |
set -euo pipefail
echo "tag=$(printf '%s' "${SOURCE_SHA}" | cut -c1-8)" >> "${GITHUB_OUTPUT}"

- name: Merge and sign multi-architecture manifest
uses: StackVista/image-pipeline/.github/actions/merge-multiarch@6284a6fc006a7cc46a7f00d02c50d5f21b117b63
with:
image: quay.io/stackstate/stackstate-k8s-cluster-agent
tag: ${{ steps.image.outputs.tag }}
arches: amd64,arm64
target-registry: quay.io
target-registry-user: ${{ vars.QUAY_USER }}
target-registry-password: ${{ secrets.QUAY_PASSWORD }}

cerberus-notify:
name: Report failure to Slack (Cerberus)
needs:
Expand All @@ -247,6 +359,8 @@ jobs:
- build-agent
- build-cluster-agent
- build-cluster-agent-image
- publish-cluster-agent-image
- merge-cluster-agent-manifest
if: >-
always()
&& github.event_name == 'push'
Expand Down
129 changes: 126 additions & 3 deletions .github/workflows/build-deb.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -236,7 +236,7 @@ jobs:
timeout-minutes: 60
env:
ARCH: ${{ matrix.arch }}
LOCAL_IMAGE: stackstate-agent:ci-${{ matrix.arch }}
LOCAL_IMAGE: quay.io/stackstate/stackstate-k8s-agent:ci-${{ matrix.arch }}
steps:
- name: Check out repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
Expand All @@ -255,8 +255,6 @@ jobs:
REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
run: |
set -eo pipefail
# Dockerfiles/agent pulls its ubuntu extract stage through the proxy;
# the BCI stages come from registry.suse.com and need no auth.
printf '%s' "${REGISTRY_PASSWORD}" | docker login -u "${REGISTRY_USER}" --password-stdin "${REGISTRY_HOST}"

- name: Build agent image
Expand All @@ -278,6 +276,129 @@ jobs:
set -eo pipefail
docker run --rm --entrypoint /opt/stackstate-agent/bin/agent/agent "${LOCAL_IMAGE}" version

- name: Scan agent image (Trivy and Grype vulnerabilities, VEX-aware, plus Trivy secrets)
uses: StackVista/image-pipeline/.github/actions/scan-image@6284a6fc006a7cc46a7f00d02c50d5f21b117b63
with:
image: ${{ env.LOCAL_IMAGE }}
mode: gate
severity: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
with-grype: true
exceptions-path: exceptions
upload-sarif: false
sarif-category: stackstate-k8s-agent-${{ matrix.arch }}

publish-agent-image:
name: Publish and sign agent image (${{ matrix.arch }})
if: github.event_name == 'push'
needs:
- build-agent-image
- test-deb-renaming
strategy:
fail-fast: false
matrix:
include:
- arch: amd64
runner: docker-public
- arch: arm64
runner: arm64-xlarge-public
runs-on: ${{ matrix.runner }}
timeout-minutes: 60
permissions:
contents: read
id-token: write
env:
ARCH: ${{ matrix.arch }}
IMAGE: quay.io/stackstate/stackstate-k8s-agent
steps:
- name: Check out repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false

- name: Download DEB package
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
with:
name: deb-package-${{ matrix.arch }}

- name: Stage the DEB in the image build context
run: |
set -eo pipefail
shopt -s nullglob
debs=(outcomes/pkg/stackstate-agent_*_"${ARCH}".deb)
if [ "${#debs[@]}" -ne 1 ]; then
echo "Expected exactly one stackstate-agent_*_${ARCH}.deb, found ${#debs[@]}: ${debs[*]}" >&2
exit 1
fi
cp "${debs[0]}" Dockerfiles/agent/

- name: Resolve image tag
id: image
env:
SOURCE_SHA: ${{ github.sha }}
run: |
set -euo pipefail
echo "tag=$(printf '%s' "${SOURCE_SHA}" | cut -c1-8)" >> "${GITHUB_OUTPUT}"

- name: Resolve canonical OCI labels
id: oci
uses: StackVista/image-pipeline/.github/actions/apply-oci-labels@6284a6fc006a7cc46a7f00d02c50d5f21b117b63
with:
image-name: stackstate-k8s-agent
tag: ${{ steps.image.outputs.tag }}
title: SUSE Observability Agent
description: Node agent collecting metrics, logs, traces and topology for SUSE Observability.
component: stackstate-k8s-agent
dockerfile: Dockerfiles/agent/Dockerfile
base-name: registry.suse.com/bci/bci-micro:latest

@LouisLotter LouisLotter Aug 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Pull BCI bases through the SUSE Private Registry proxy. These jobs run on self-hosted docker-public and arm64-xlarge-public, but both Dockerfiles pull BCI stages directly from registry.suse.com. Parameterize those bases and pass registry.tooling.stackstate.io/suse/... to both builds; source-registry-credentials does not rewrite the FROM references.

@LouisLotter LouisLotter Aug 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@LouisParkin some of this feedback is recurring now. Something I find useful is to tell the agent to not only fix feedback like this but also understand why this mistake happened and write down guidance so it does not make the same mistake again.

@LouisParkin LouisParkin Aug 7, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Both Dockerfiles now parameterize the base registry and default it to the proxy:

Dockerfiles/agent/Dockerfile

ARG BASE_IMAGE_REGISTRY=registry.tooling.stackstate.io/docker/library
ARG BCI_IMAGE_REGISTRY=registry.tooling.stackstate.io/suse/bci
...
FROM ${BCI_IMAGE_REGISTRY}/bci-base:latest AS software
FROM ${BCI_IMAGE_REGISTRY}/bci-micro:latest

Dockerfiles/cluster-agent/Dockerfile gets the same treatment, including the ubuntu:20.04 builder stage.

Your diagnosis was exactly right: source-registry-credentials authenticates but does not rewrite FROM, so the proxying has to happen where the base is named. Defaulting in the Dockerfile rather than passing --build-arg from the workflow is deliberate — these Dockerfiles are built from more places than this workflow (the GitLab jobs during the overlap period, local helper scripts), and only the default reaches all of them. It also matches the BASE_IMAGE_REGISTRY convention already in the files.

One deliberate exception: apply-oci-labels' base-name stays on registry.suse.com/bci/bci-micro:latest. That feeds org.opencontainers.image.base.name, which VEX data and base-image tracking key on upstream identity, so it should record where the base really comes from even though the bytes arrive via the proxy.

@LouisParkin LouisParkin Aug 7, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point, and taken. The recurrence was the signal — the same class of mistake kept reappearing because the reasoning behind the fix lived only in review threads.

The guidance is now written down in the mission-control wiki, in wiki/concepts/gitlab-to-github-ci-porting.md, as a section titled "The registry proxy is a Dockerfile concern, not a credentials concern". It records the trap in the form that actually causes it:

Credentials only authenticate; they do not rewrite FROM. A Dockerfile line that says FROM registry.suse.com/bci/bci-micro:latest pulls from registry.suse.com no matter what you are logged in to. The mistake is invisible in the workflow file, because the workflow file is not where the base image is named.

...along with the rule (ARG before the first FROM, defaulted to the proxy), why the default belongs in the Dockerfile rather than in a workflow --build-arg, and two caveats that are easy to get wrong in the opposite direction:

  • only repoint self-hosted jobs, since GitHub-hosted runners cost us no NAT egress and routing them through our registry adds outbound load-balancer traffic instead of removing it
  • keep apply-oci-labels' base-name on the upstream name, because VEX and base-image tracking key on upstream identity

The same document now also carries the scan-gate section and the house rule on not commenting YAML, both of which came out of this review round.

registry-credentials: |
[{"registry": "${{ vars.REGISTRY_HOST }}", "username": "${{ vars.REGISTRY_USER }}", "password": "${{ secrets.REGISTRY_PASSWORD }}"}]

- name: Build, publish, and sign architecture image
uses: StackVista/image-pipeline/.github/actions/push-single-arch@6284a6fc006a7cc46a7f00d02c50d5f21b117b63

@LouisLotter LouisLotter Aug 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Gate publication on image scanning. push-single-arch builds, pushes, attests and signs but does not run scan-image. Both customer images can therefore be published without the mandatory VEX-aware Trivy/Grype vulnerability gate and separate fail-closed secret scan. Please scan the local amd64/arm64 images in mode: gate and make both publish jobs depend on successful scans; the cluster-agent path has the same gap.

@LouisParkin LouisParkin Aug 7, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 4d9feabdbd. Agreed, and this was a real hole rather than a cosmetic one.

Publication is now gated on the scan through the job graph. scan-image runs in mode: gate in the build job for both images, and the publish job depends on it, so a failing gate stops publication:

  • agent — build-agent-image scans (build-deb.yml:280); publish-agent-image has needs: [build-agent-image, test-deb-renaming]
  • cluster-agent — build-cluster-agent-image scans (build-binaries.yml:241); publish-cluster-agent-image has needs: build-cluster-agent-image

You were right that the cluster-agent path had the same gap; both are covered now.

Two things were needed to make the gate actually bite, and the second was not obvious:

  1. LOCAL_IMAGE had to become registry-qualified (quay.io/stackstate/stackstate-k8s-agent:ci-<arch>). The evaluator keys exceptions on NormaliseImage(image), which strips only the tag/digest and not the registry or namespace, so under the old bare stackstate-agent:ci-<arch> name nothing matched.
  2. exceptions-path: exceptions, with 26 exception files carrying per-CVE remediation state.

I checked this against the real evaluator rather than by eye — built evaluator/ from the pinned SHA, synthesised Grype reports and ran the gate; all 26 matched, exit 0. A control run under the old bare image name returned all 24 findings as unmanaged, which is what proved the rename was mandatory.

Confirmed on CI (amd64):

total in-scope findings:  24
suppressed by exception:  24
unmanaged:                0

The secret scan runs as a separate fail-closed invocation, unfiltered by severity, per the AGENTS.md rule.

with:
image: ${{ env.IMAGE }}
tag: ${{ steps.image.outputs.tag }}
arch: ${{ matrix.arch }}
docker-context: Dockerfiles/agent
dockerfile: Dockerfiles/agent/Dockerfile
build-args: ARCH=${{ matrix.arch }}
labels: |
${{ steps.oci.outputs.labels }}
org.opencontainers.image.revision=${{ github.sha }}
source-registry-credentials: |
[{"registry": "${{ vars.REGISTRY_HOST }}", "username": "${{ vars.REGISTRY_USER }}", "password": "${{ secrets.REGISTRY_PASSWORD }}"}]
target-registry: quay.io
target-registry-user: ${{ vars.QUAY_USER }}
target-registry-password: ${{ secrets.QUAY_PASSWORD }}

merge-agent-manifest:
name: Publish and sign multi-architecture agent image
if: github.event_name == 'push'
needs: publish-agent-image
runs-on: docker-public
timeout-minutes: 30
permissions:
contents: read
id-token: write
steps:
- name: Resolve image tag
id: image
env:
SOURCE_SHA: ${{ github.sha }}
run: |
set -euo pipefail
echo "tag=$(printf '%s' "${SOURCE_SHA}" | cut -c1-8)" >> "${GITHUB_OUTPUT}"

- name: Merge and sign multi-architecture manifest
uses: StackVista/image-pipeline/.github/actions/merge-multiarch@6284a6fc006a7cc46a7f00d02c50d5f21b117b63
with:
image: quay.io/stackstate/stackstate-k8s-agent
tag: ${{ steps.image.outputs.tag }}
arches: amd64,arm64
target-registry: quay.io
target-registry-user: ${{ vars.QUAY_USER }}
target-registry-password: ${{ secrets.QUAY_PASSWORD }}

cerberus-notify:
name: Report failure to Slack (Cerberus)
needs:
Expand All @@ -286,6 +407,8 @@ jobs:
- build-deb
- test-deb-renaming
- build-agent-image
- publish-agent-image
- merge-agent-manifest
if: >-
always()
&& github.event_name == 'push'
Expand Down
5 changes: 3 additions & 2 deletions Dockerfiles/agent/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
ARG BASE_IMAGE_REGISTRY=registry.tooling.stackstate.io/docker/library
ARG BASE_IMAGE_UBUNTU_VERSION=24.04
ARG BCI_IMAGE_REGISTRY=registry.tooling.stackstate.io/suse/bci

##########################################
# Preparation stage: extract and cleanup #
Expand Down Expand Up @@ -27,15 +28,15 @@ RUN dpkg -x /stackstate-agent*_${ARCH}.deb . \
# - copy default config files
COPY stackstate*.yaml etc/stackstate-agent/

FROM registry.suse.com/bci/bci-base:latest AS software
FROM ${BCI_IMAGE_REGISTRY}/bci-base:latest AS software
# as per comment here: https://github.com/rancher/rancher/blob/main/package/Dockerfile#L12
# No-op command to create an explicit layer - this fixes a weird buildkit/buildx bug on macos arm
RUN :
####################################
# Actual docker image construction #
####################################

FROM registry.suse.com/bci/bci-micro:latest
FROM ${BCI_IMAGE_REGISTRY}/bci-micro:latest
LABEL maintainer="StackState <info@stackstate.com>"
# Install system packages using builder image that has zypper

Expand Down
9 changes: 6 additions & 3 deletions Dockerfiles/cluster-agent/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,11 @@
ARG BASE_IMAGE_REGISTRY=registry.tooling.stackstate.io/docker/library
ARG BCI_IMAGE_REGISTRY=registry.tooling.stackstate.io/suse/bci

########################################
# Preparation stage: layout and chmods #
########################################

FROM registry.tooling.stackstate.io/docker/library/ubuntu:20.04 as builder
FROM ${BASE_IMAGE_REGISTRY}/ubuntu:20.04 as builder

WORKDIR /output

Expand All @@ -22,12 +25,12 @@ RUN chmod 755 entrypoint.sh \
# Actual docker image construction #
####################################

FROM registry.suse.com/bci/bci-base:latest AS software
FROM ${BCI_IMAGE_REGISTRY}/bci-base:latest AS software
# as per comment here: https://github.com/rancher/rancher/blob/main/package/Dockerfile#L12
# No-op command to create an explicit layer - this fixes a weird buildkit/buildx bug on macos arm
RUN :

FROM registry.suse.com/bci/bci-micro:latest AS final
FROM ${BCI_IMAGE_REGISTRY}/bci-micro:latest AS final
LABEL maintainer="StackState <info@stackstate.com>"

ARG user="stackstate-agent"
Expand Down
26 changes: 26 additions & 0 deletions exceptions/stackstate-k8s-agent/CVE-2025-15366.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
schema_version: '1'
vulnerability:
id: CVE-2025-15366
severity: MEDIUM
product:
consumer: stackstate-k8s-agent
image: quay.io/stackstate/stackstate-k8s-agent
component:
purl: pkg:generic/python@3.13.13
paths:
- opt/stackstate-agent/embedded/bin/python3.13
status: accepted_pending_upstream_fix
reason: no_upstream_fix_for_pinned_runtime_line
expires: '2026-08-20'
owner: "@StackVista/observability-team"
upstream_owner: python
upstream_reference: https://www.cve.org/CVERecord?id=CVE-2025-15366
statement: |
No fix is published for the CPython 3.13 maintenance line. Upstream
currently carries the fix only in 3.15.0a6, a pre-release of a different
minor version, so there is no compatible patch we could apply without
moving the embedded interpreter to an unreleased runtime. Under the
remediation SLA this falls outside the patch commitment, so the date below
is a short review deadline rather than an acceptance: re-check upstream for
a 3.13 backport before renewing. Tracked with the rest of the
embedded-runtime work in STAC-25556.
Loading
Loading