STAC-25500 Port the DEB signing and pre-release publishing lane to GitHub Actions

[LouisParkin]

Closes the last two GitLab jobs on this repo with no GitHub counterpart: sign_deb and pre_release_deb (.gitlab-ci-agent.yml lines 604 and 635). Stacked on #454.

Security prerequisite, not cleanup

sign_debian_package.sh called printenv unconditionally. On GitLab that dumped the GPG private key and its passphrase into the job log. stackstate-agent is a public repo, so on GitHub that log is world-readable. It also left the exported private key in the checkout as gpg_private.key, where any later artifact upload would collect it.

Both are removed before any signing secret is wired in, as STAC-25500 requires. Key setup now happens in an ephemeral GNUPGHOME (mktemp, mode 700) removed by an EXIT trap that also kills the gpg-agent, so a preset passphrase cannot outlive the job on a reused runner. The passphrase reaches gpg through a mode-600 file, never argv.

Fixes a latent signing bug

The old preset step was:

echo $SIGNING_PRIVATE_PASSPHRASE | gpg-preset-passphrase -v -c $(gpg --list-secret-keys ... | awk ...)

A key exposes one keygrip per primary and subkey. Unquoted, that substitution expands to multiple words, so with a signing subkey the second keygrip became a stray argument and was never presetted — leaving signing able to block on a pinentry prompt no CI runner can answer. Each keygrip is now presetted individually.

Verified against a generated throwaway key: two keygrips, both presetted, passphrase delivered on stdin, GNUPGHOME removed by the trap, nothing left in the workspace.

Shared setup

The two scripts need the same key in the same state but are separate processes, and on GitHub may be separate steps, so neither can rely on a keyring the other left behind. The setup moves into gpg_signing_setup.sh, sourced by both. gpg-preset-passphrase is auto-detected across /usr/lib/gnupg2, /usr/lib/gnupg, /usr/libexec and PATH (the path differs by distro and release), and it fails loudly rather than silently skipping the preset when absent.

Both scripts fall back CI_PROJECT_DIR → GITHUB_WORKSPACE, and publish_package.sh falls back CI_COMMIT_REF_NAME → GITHUB_REF_NAME, so the apt codename stays the release branch exactly as on GitLab.

Workflow

Job	Notes
sign-and-publish-deb	Downloads both arch artifacts and runs a single deb-s3 upload. GitLab fanned this out per arch, so two jobs rewrote the same apt index concurrently — collecting both first removes that race.
generate-install-script	Runs inv release.generate-install -t inside the build container: the task collection imports python-gitlab plus the GitHub/Datadog API helpers and only loads in the conda env. Asserts the rendered script contains no None.s3.amazonaws.com from an unset bucket variable — generate_install reads three bucket vars and silently renders None for any that are missing.
publish-install-script	Uploads it, so the container job never holds a credential.

Both publishing jobs are gated on push and bound to the agent-pre-release environment, which carries the deployment branch rule, the signing secrets and the AWS role. Pull requests cannot reach them.

Blocked on

This lane cannot go green until these exist — the code is ready and reviewable now:

1. GitHub environment agent-pre-release with a deployment branch rule for the release branch.
2. Secrets in it: SIGNING_PUBLIC_KEY, SIGNING_PRIVATE_KEY, SIGNING_PRIVATE_PASSPHRASE, SIGNING_KEY_ID.
3. Variable AGENT_PRERELEASE_ROLE_ARN, from StackVista/terraform-infra#76 (STAC-25545).

Validation

• shellcheck -x — clean on all three scripts.
• gpg_signing_setup exercised against a real generated key: explicit override, PATH auto-detection, and the missing-binary failure path (exits 1 with a clear message).
• actionlint — clean apart from the pre-existing self-hosted runner-label notices.
• zizmor v1.28.0 — No findings to report.
• YAML parses; all 10 needs: references resolve.

https://stackstate.atlassian.net/browse/STAC-25500
Refs https://stackstate.atlassian.net/browse/STAC-25546 and https://stackstate.atlassian.net/browse/STAC-25545

[LouisLotter]

[P1] Do not install an unpinned publisher into the signing job. gem install deb-s3 resolves the latest code at run time, and that executable later runs with the GPG key/passphrase and AWS credentials. Use a pinned, reviewed shared CI image, or fetch an exact gem version and verify its published checksum before execution.

[LouisParkin]

Fixed in f31b3c3. You were pointing at a real regression, not just a hardening nit: GitLab never resolved this at run time either. sign_deb ran in stackstate-agent-runner-gitlab:debian-20220826-patched-20230511 with RVM and deb-s3 already baked in, so the runtime gem install was something the port introduced.

deb-s3 and its full runtime tree are now pinned in .github/deb-s3-gems.sha256 and installed by .github/scripts/install-deb-s3.sh, which fetches each gem at its exact version and verifies it against the SHA256 RubyGems publishes for that release before anything is installed or executed.

I went with the pinned-manifest option rather than a shared CI image because the only image that already carries deb-s3 is that 2022 Debian one, and adopting it would drag an unmaintained base back into a new pipeline. Adding deb-s3 to stackstate-ci-images is the better long-term answer and I am happy to raise it separately.

Two things worth flagging, both found by actually running this:

The checksum gate works. My first manifest recorded bigdecimal as ccc836ea..., which is the java platform build. The RubyGems versions API lists one entry per platform and I had taken the first. The verify step caught it and aborted before installing. Resolution now filters platform == ruby.

The manifest is nine gems, not twelve. base64, bigdecimal and logger are excluded deliberately: aws-sdk-core requires them at >= 0 and they are Ruby default gems shipped by the distribution ruby package, so pinning them forced a native bigdecimal build for no supply-chain gain. Verified that deb-s3 26.1.0 activates and runs against the nine pinned gems plus the defaults, and that upload, the subcommand publish_package.sh uses, is present.

I also hit a portability trap while testing: RubyGems installed the binstub as deb-s3.ruby4.0, and publish_package.sh calls deb-s3 by bare name. The script now links the canonical executable when the plain name is missing, and finishes with a deb-s3 help smoke check that activates the whole pinned set, so a missing or incompatible gem fails at install time instead of midway through publishing.

[LouisLotter]

[P2] Keep install-script publication behind DEB publication. This job depends only on generation, so it can overwrite the public install.sh even when the build, signing, or apt upload fails. The GitLab pre_release_deb job explicitly needed sign_deb. Require both generate-install-script and sign-and-publish-deb here to preserve that release boundary.

[LouisParkin]

Fixed in f31b3c3. Agreed, and the GitLab precedent is explicit: pre_release_deb listed sign_deb in its needs. Dropping that let a green generate-install-script publish a fresh install.sh pointing at a pre-release apt repository that the failed run never actually populated.

publish-install-script now requires both generate-install-script and sign-and-publish-deb.