From 26bf33d90d08d3dd29613ff43a3e5658329db1f4 Mon Sep 17 00:00:00 2001 From: Louis Parkin Date: Fri, 7 Aug 2026 10:50:22 +0200 Subject: [PATCH 1/2] STAC-25555: fail closed when Cerberus secrets are missing The unconfigured guard warned and exited 0. That reported success in exactly the case where no Slack message was sent, and the warning rendered only inside an unattended failed run, where nobody reads it. Missing or drifted secrets therefore disabled the safeguard silently -- the failure mode the safeguard exists to prevent. Exit non-zero instead, and name the individual missing secret: the two are provisioned together, so exactly one missing distinguishes a partial apply or drift from a repo that was never wired up. The `workflow_call` secrets stay `required: false`, which is not in tension with this. A caller passing `${{ secrets.X }}` for a secret the repo does not hold yields an empty string, and GitHub rejects that against a required secret as "not provided" -- failing the call before any step runs, with an opaque error and no annotation naming what to fix. Accepting the empty value and checking it here fails just as closed and says who provisions what. The job is `if: failure() && github.event_name == 'push'`, so the only run it can ever redden is one that is already red. There the red is the point: it distinguishes "the build broke" from "the alarm is broken too". Raised by LouisLotter in review of #366. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/cerberus-notify.yml | 45 +++++++++++++++++++++------ 1 file changed, 35 insertions(+), 10 deletions(-) diff --git a/.github/workflows/cerberus-notify.yml b/.github/workflows/cerberus-notify.yml index 2b08e387..3a3cb79a 100644 --- a/.github/workflows/cerberus-notify.yml +++ b/.github/workflows/cerberus-notify.yml @@ -22,9 +22,17 @@ name: Cerberus notify # Prerequisites: CERBERUS_LAMBDA_URL and CERBERUS_API_TOKEN must both reach this # repo as REPO-level secrets. The org-level copies are visibility=private, which # excludes this PUBLIC repo. pulumi-infra provisions the pair together -# (github/repoVariables/resources.yaml, StackVista/pulumi-infra#277). If either -# is missing, this workflow warns and exits 0 rather than adding a second red job -# to an already-failed run -- the annotation is the signal. +# (github/repoVariables/resources.yaml, StackVista/pulumi-infra#277). +# +# If either is missing this job FAILS (STAC-25555). An earlier revision warned +# and exited 0, reasoning that a broken notification should not add a second red +# job to an already-failed run. That was the wrong trade: it reported success in +# exactly the case where no Slack message was sent, and the warning was only ever +# rendered inside an unattended failed run -- which is to say, seen by nobody. +# Secrets drift, and a safeguard that disables itself silently is not a +# safeguard. This job only runs on push failures, so the only run it can redden +# is one that is already red, and there the red is the point: it says the alarm +# itself is broken, not just the build. # # The bearer token is not optional going forward. StackVista/cerberus#4 # (STAC-24889) adds `Authorization: Bearer ` verification to every @@ -47,10 +55,13 @@ on: required: true type: string secrets: - # `required: false`. A caller passing `${{ secrets.X }}` for a secret the - # repo does not hold yields an empty string, which GitHub rejects as "not - # provided" against a required secret and fails the call before the run - # step's guard can warn -- the failure mode this workflow exists to avoid. + # `required: false`, deliberately, and it is not in tension with failing + # closed below. A caller passing `${{ secrets.X }}` for a secret the repo + # does not hold yields an empty string, which GitHub rejects as "not + # provided" against a required secret -- failing the call before any step + # runs, with an opaque error and no annotation. Accepting the empty value + # and checking it in the step below fails just as closed, and says which + # secret is missing and who provisions it. CERBERUS_LAMBDA_URL: required: false CERBERUS_API_TOKEN: @@ -81,9 +92,23 @@ jobs: run: | set -euo pipefail - if [ -z "${CERBERUS_LAMBDA_URL}" ] || [ -z "${CERBERUS_API_TOKEN}" ]; then - echo "::warning title=Cerberus not configured::CERBERUS_LAMBDA_URL and/or CERBERUS_API_TOKEN is not visible to this repo, so the ${SUITE} failure was not reported to Slack. Both are provisioned as repo-level secrets by pulumi-infra (STAC-25533)." - exit 0 + # Fail closed (STAC-25555). Naming the individual secret matters: the + # two are provisioned together, so exactly one missing means drift or a + # partial apply rather than a repo that was never wired up. + missing="" + if [ -z "${CERBERUS_LAMBDA_URL}" ]; then + missing="CERBERUS_LAMBDA_URL" + fi + if [ -z "${CERBERUS_API_TOKEN}" ]; then + if [ -n "${missing}" ]; then + missing="${missing} and CERBERUS_API_TOKEN" + else + missing="CERBERUS_API_TOKEN" + fi + fi + if [ -n "${missing}" ]; then + echo "::error title=Cerberus not configured::Missing secret(s): ${missing}. Not visible to this repository, so the ${SUITE} failure was NOT reported to Slack. Both are provisioned as repo-level secrets by pulumi-infra (github/repoVariables/resources.yaml, STAC-25533). This job fails rather than passing silently, because a notification path that quietly stops working is worse than a visibly broken one." + exit 1 fi COMMIT_TITLE=$(printf '%s' "${COMMIT_MESSAGE}" | head -n1) From 76d434e27e7cc384c5577eeab4d665cc2d36294b Mon Sep 17 00:00:00 2001 From: Louis Parkin Date: Fri, 7 Aug 2026 11:22:05 +0200 Subject: [PATCH 2/2] STAC-25555: drop YAML-level comments from cerberus-notify House convention is uncommented YAML. Removes every YAML-level comment from this workflow, including the ones the previous commit added. Shell comments inside the `run:` block are kept: they are string content rather than YAML syntax, and they explain non-obvious runtime choices at the point of use -- why the guard names each secret separately, and why curl is not run with --verbose. Verified as a pure comment removal: the parsed document is identical before and after, and the `run:` block is byte-identical. actionlint and Zizmor clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/cerberus-notify.yml | 58 --------------------------- 1 file changed, 58 deletions(-) diff --git a/.github/workflows/cerberus-notify.yml b/.github/workflows/cerberus-notify.yml index 3a3cb79a..d049d157 100644 --- a/.github/workflows/cerberus-notify.yml +++ b/.github/workflows/cerberus-notify.yml @@ -1,52 +1,5 @@ name: Cerberus notify -# New capability, not a port (STAC-25142 / STAC-25533). Unlike stackstate-agent -# and stackstate-process-agent, this repo's retired .gitlab-ci.yml had no notify -# job and no .cerberus directory, so a failed release-branch pipeline has always -# been silent here. STAC-25510 is what that costs: process-agent's image -# publishing broke on 2026-07-23 and went unnoticed for 12 days. -# -# Structure and calling convention follow -# stackstate-process-agent/.github/workflows/cerberus-notify.yml, which in turn -# follows `cerberus-block-on-master-fail` in StackVista/stackstate. Cerberus is -# the internal notify/block Lambda (source: https://github.com/StackVista/cerberus). -# `platform: github` makes it build GitHub pipeline/commit URLs rather than -# GitLab ones. -# -# `action: notify`, never `action: block`. Policy for migrated repos is notify by -# default. Blocking locks the branch (`lock_branch`), additionally requires the -# Cerberus GitHub App to be installed here, and mutates branch protection that -# pulumi-infra owns (STAC-25522) out from under it -- a subsequent pulumi apply -# would silently unlock the branch again. -# -# Prerequisites: CERBERUS_LAMBDA_URL and CERBERUS_API_TOKEN must both reach this -# repo as REPO-level secrets. The org-level copies are visibility=private, which -# excludes this PUBLIC repo. pulumi-infra provisions the pair together -# (github/repoVariables/resources.yaml, StackVista/pulumi-infra#277). -# -# If either is missing this job FAILS (STAC-25555). An earlier revision warned -# and exited 0, reasoning that a broken notification should not add a second red -# job to an already-failed run. That was the wrong trade: it reported success in -# exactly the case where no Slack message was sent, and the warning was only ever -# rendered inside an unattended failed run -- which is to say, seen by nobody. -# Secrets drift, and a safeguard that disables itself silently is not a -# safeguard. This job only runs on push failures, so the only run it can redden -# is one that is already red, and there the red is the point: it says the alarm -# itself is broken, not just the build. -# -# The bearer token is not optional going forward. StackVista/cerberus#4 -# (STAC-24889) adds `Authorization: Bearer ` verification to every -# non-Slack request; before it, the endpoint was entirely unauthenticated. -# Sending the header is forward-compatible -- the currently deployed Lambda -# ignores unknown headers -- so this works either side of that deploy. Without -# it, the first failure after cerberus#4 ships would get a 401 and no Slack -# message. -# -# The Slack channel is deliberately not sent. Cerberus resolves it as -# `util.GetOrDefault(req.Context, "channel", s.Channel)`, and GetOrDefault treats -# an empty or whitespace value as absent, so omitting `channel` falls back to the -# Lambda's own SLACK_CHANNEL. - on: workflow_call: inputs: @@ -55,20 +8,11 @@ on: required: true type: string secrets: - # `required: false`, deliberately, and it is not in tension with failing - # closed below. A caller passing `${{ secrets.X }}` for a secret the repo - # does not hold yields an empty string, which GitHub rejects as "not - # provided" against a required secret -- failing the call before any step - # runs, with an opaque error and no annotation. Accepting the empty value - # and checking it in the step below fails just as closed, and says which - # secret is missing and who provisions it. CERBERUS_LAMBDA_URL: required: false CERBERUS_API_TOKEN: required: false -# Nothing here reads the repository; the payload is built entirely from the -# github context. permissions: {} jobs: @@ -85,8 +29,6 @@ jobs: BRANCH: ${{ github.ref_name }} PIPELINE: ${{ github.run_id }} COMMIT_SHA: ${{ github.sha }} - # Empty on tag pushes, which carry no head_commit. COMMIT_TITLE below - # falls back to the sha so the Slack message is never blank. COMMIT_MESSAGE: ${{ github.event.head_commit.message }} SUITE: ${{ inputs.suite }} run: |