STAC-25519 Report failed master builds to Slack via Cerberus - #250
Merged
Conversation
The GitLab pipeline had a `notify-on-master-fail` job that posted to the team's CI channel through Cerberus. The GitHub migration dropped it, so a red master became completely silent. That gap has already cost us: image publishing to Quay broke on 2026-07-23 and nobody noticed for 12 days (STAC-25510) because there was nothing to tell us. Adds a reusable cerberus-notify.yml and hangs it off ci-success as a terminal job, so every failure funnels through one notification rather than one per job. Sends `platform: github` so Cerberus builds GitHub pipeline and commit URLs, and omits `channel` so the Lambda's own default applies. `action: notify`, not `block`: policy for migrated repos is notify by default, and blocking would lock master on every failure, need the Cerberus GitHub App installed here, and fight Pulumi over branch protection. CERBERUS_LAMBDA_URL does not reach this repo yet. The org-level secret is visibility=private and this repo is public, and widening it is not an option because the endpoint is unauthenticated -- the URL is the entire capability. It needs to arrive as a repo-level secret from pulumi-infra. Until then the workflow emits a warning annotation and exits 0 rather than adding a second red job to an already failed run. Verified locally: actionlint type-checks the reusable call (confirmed with a negative control), zizmor reports no findings, and the payload was exercised against a stubbed endpoint with a multi-line commit message containing quotes and shell metacharacters -- it is passed via env, not interpolated, so it is escaped rather than executed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Andreagit97
approved these changes
Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Restores the
notify-on-master-failjob we lost in the GitLab → GitHub migration. Right now a red master on this repo is completely silent.That is not hypothetical. Image publishing to Quay broke on 23 July and went unnoticed for 12 days (STAC-25510) — master was failing the whole time and nothing told us.
Part of the GitLab CI parity work under STAC-25142 / STAC-25519.
What this does
Adds a reusable
cerberus-notify.ymland calls it fromci.ymlas a terminal job:It hangs off
ci-success, which already aggregates every other job, so all failures funnel through one notification instead of one per job. Master pushes only — PRs stay quiet.Follows the
cerberus-block-on-master-failconvention fromStackVista/stackstate, sendingplatform: githubso Cerberus builds GitHub pipeline/commit URLs rather than GitLab ones.channelis deliberately omitted: Cerberus resolves it asutil.GetOrDefault(req.Context, "channel", s.Channel), so leaving it out uses the Lambda's own default (per @fzhdanov —SLACK_CHANNEL_IDis no longer needed caller-side).Two decisions worth reviewing
action: notify, notblock. Policy for migrated repos is notify by default. Blocking would lockmastervialock_branchon every failure, additionally requires the Cerberus GitHub App to be installed on this repo, and mutates Pulumi-managed branch protection — apulumi-infraapply during a block window silently unlocks it.It ships before the secret exists, on purpose.
CERBERUS_LAMBDA_URLis an org secret withvisibility=private, and this repo is public, so it does not reach us. Widening org visibility is not an option — the Cerberus endpoint is unauthenticated, so the URL is the capability. It needs to arrive as a repo-level secret frompulumi-infra, which is tracked separately and blocked on the same missing stack config key asstackstate-agent.Until that lands the workflow emits a warning annotation and exits 0, rather than adding a second red job to a run that is already failing. The secret is declared
required: falsefor a specific reason: passing${{ secrets.X }}for a secret the repo lacks yields an empty string, and GitHub rejects that against a required secret, failing the call before the guard can run — exactly the failure mode this is meant to avoid. So merging this now is safe, and it starts working the moment the secret appears, with no further change here.Validation
actionlinttype-checks the reusable-workflow call. I confirmed the check is real with a negative control — renamingsuite→suittecorrectly producedinput "suitte" is not defined. Clean on the new files; the two remaining findings are pre-existing in other jobs.zizmor --collect=workflows,actions,dependabot .→ no findings."quotes",$(whoami)and backticks. The message goes throughenv:rather than being interpolated into the script, so it is JSON-escaped, not executed, and only the first line is used as the title. Also verified: missing secret → warn + exit 0 with no call made; empty commit message → noset -utrip.Not covered here
The other two parity gaps have their own tickets and both need a decision before any code: STAC-25520 (S3 binary publishing) and STAC-25521 (beest verification trigger).