Skip to content

[stacked on #803] Add clusterTrustBundle as a SystemInfo volume data source - #941

Draft
Max Thompson (thompsonmax) wants to merge 7 commits into
agent-substrate:mainfrom
thompsonmax:ctb-source
Draft

[stacked on #803] Add clusterTrustBundle as a SystemInfo volume data source#941
Max Thompson (thompsonmax) wants to merge 7 commits into
agent-substrate:mainfrom
thompsonmax:ctb-source

Conversation

@thompsonmax

@thompsonmax Max Thompson (thompsonmax) commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Part of #932 (PR 1 of 3; doesn't close it). The SystemInfo volume mechanism this builds on is #802.

Stacked on #803. This PR includes #803's commits and shows a combined diff until that merges. The new work is one commit: Add clusterTrustBundle as a SystemInfo volume data source. Review just that delta via the branch compare, or select the commit in the Commits tab so comments thread here. After #803 merges I'll rebase, the diff will collapse to the one commit, and I'll mark this ready for review.

New changes:

  • clusterTrustBundle data source for SystemInfo volumes. It projects a named ClusterTrustBundle (certificates.k8s.io/v1beta1) to a PEM file, the substrate analog of the Kubernetes clusterTrustBundle projected volume source. Selection is by name; signerName and label selectors can come later if needed.
  • ateapi resolves the bundle at actor start through an informer-backed lister and sanitizes it the way kubelet does for projections: only CERTIFICATE blocks, deduplicated, headers stripped (new internal/pemutil). The wire spec carries {path, pem_bundle}, so atelet writes bytes and never talks to the Kubernetes API. A bundle that is missing, empty, or free of certificates fails the actor start with an error that names it.
  • The v1beta1 API is feature-gated, so ateapi probes for it at startup rather than registering the informer blindly. A blind registration hangs WaitForCacheSync and ateapi never finishes booting; the functional suite's envtest demonstrated this the hard way. envtest now serves the API, same as hack/create-kind-cluster.sh already enables on kind.
  • Validation: paths must be clean relative paths and unique across every data source in the volume. dataSources maxItems drops from 32 to 8 because the cross-source duplicate rule exceeded the apiserver's CEL cost budget at 32.
  • RBAC: the ate-api-server ClusterRole can read clustertrustbundles.
  • e2e: the identity suite creates a fixture bundle (junk PEM plus a duplicated certificate around a real one) and asserts the probe reads the sanitized PEM byte for byte. It then rotates the bundle and cycles an actor through suspend/resume to prove contents refresh on every Run/Restore. The suite is parameterized by sandbox class now, and a new CI step runs it on micro-VM after the runtime assets are staged, so both runtimes cover SystemInfo delivery (this also closes the micro-VM coverage gap left by feat: add systemInfo volume source with actorIdentity data source #803's actorMetadata work).

Not included (tracked in #932):

Review notes: the failure paths (missing bundle, API unavailable) are unit-tested rather than e2e-tested, since a missing bundle wedges the template's golden boot and costs minutes of suite time for little signal.

  • Tests pass (unit and envtest locally; internal/atunnel's unix-socket failures on macOS are pre-existing on upstream main)
  • Appropriate changes to documentation are included in the PR (api-guide: clusterTrustBundle section under SystemInfo Volumes)

Taahir Ahmed (ahmedtd) and others added 7 commits August 13, 2026 09:38
@ahmedtd @thompsonmax
Remove volume type enum
b4b4b68
@ahmedtd @thompsonmax
System Information Volumes: Part 1 (Actor Identity)
fc43b9d 
This commit defines a new volume type, SystemInfoVolume, that will serve
a similar purpose as Projected volumes in Kubernetes.  It will support
writing information from multiple sources to automatically-updating
files in the Actor's filesystem.

For a first pass, I have converted the existing hardcoded Actor ID file
to be one of the available information sources in a SystemInfoVolume.

Further work will add Actor Identity JWTs and Actor Identity
certificates.
@thompsonmax
System Information Volumes: Part 1 finishing touches
3609e52 
Complete the initial actorIdentity data source support:

- e2e: declare a systemInfo volume in the identity probe's ActorTemplate,
  mounted at /run/ate, replacing the removed automatic identity mount so
  the restore-identity regression gate exercises the new API.
- Validate actorIdentity paths at admission: must be a clean relative
  Unix path (no absolute paths, '..', '.', '//', ':', or control
  characters), and paths must be unique within a volume. Previously bad
  paths were only rejected by the atomic writer at Run/Restore time.
- Unit tests for the ateapi systemInfo conversion and for atelet's
  system-info volume population (extracted into writeSystemInfoVolume).
- Update the stale micro-VM known-gap comment to reference systemInfo
  volumes instead of the removed /run/ate identity mount.
@thompsonmax
Restructure SystemInfo data sources around actorMetadata
d1c5052 
Per the API discussion on agent-substrate#802: substrate has no "actor ID" concept --
resource identity is (atespace, name) plus a server-generated UID. Replace
the actorIdentity data source with an actorMetadata source that projects
each identity field to its own file, downwardAPI-style:

  systemInfo:
    dataSources:
    - actorMetadata:
        items:
        - field: name       # enum: name | atespace | uid
          path: actor-name

- CRD: ActorMetadataDataSource with a field enum and per-item path;
  admission validation for unknown fields, duplicate fields, duplicate
  paths, and non-clean/absolute paths; at most one actorMetadata entry
  per volume keeps paths unique volume-wide.
- atelet proto: ActorMetadataDataSource/ActorMetadataItem with a field
  enum; ateapi converts CRD items to wire items.
- atelet: writeSystemInfoVolume projects name/atespace/uid from the
  Run/Restore request; unknown fields (newer ateapi) are skipped rather
  than written empty.
- e2e: the identity probe projects and serves all three fields; the
  suite now also asserts atespace matches and the projected UID equals
  the control plane's authoritative UID per actor, distinct across
  actors seeded from the same snapshot.
- docs: api-guide section rewritten for actorMetadata.

This also frees the "identity" naming for the planned credential data
sources (actorIdentityToken, actorIdentityCertificate), which relate to
the existing ateapi.ActorIdentity service.
@thompsonmax
microvm: mount system-info volumes over a read-only virtio-fs share
4c34cdc 
SystemInfo volumes were gVisor-only; per the agent-substrate#802 discussion, micro-VM
support lands with Part 1 rather than as a follow-up. The mechanism
mirrors the durable-dir share:

- ateom proto: containers carry system_info_volume_mounts (volume name +
  mount path), populated by atelet's buildAteomWorkloadSpec.
- ateom-microvm serves ateompath.SystemInfoVolumeRootsDir(actorUID) over
  a third virtiofsd (cache=auto: atelet rewrites the contents underneath
  the guest on every restore). The agent mounts the share at sandbox
  creation, and each declaring container gets a READ-ONLY bind from the
  share's per-volume subdirectory to its declared mount path.
- Restore restarts the share's virtiofsd and rewrites its vhost-user
  socket in the snapshot's VM config (matched by fs tag). Nothing is
  restored from the snapshot itself: atelet has already regenerated the
  files with the resumed actor's values, which is the point of
  system-info volumes.
- Checkpoint deliberately ignores the share: the volume roots live
  outside the durable-dir tree precisely so the durable tar can never
  capture generated identity data.
- Replace the stale "KNOWN GAP" comment in spec.go: dropping host-path
  binds in the kata spec shaper is fine because volumes reach micro-VM
  containers via the shares, not spec.Mounts.
@thompsonmax
third_party/atomicwriter: record provenance and delineate local changes
22da91b 
Add a README pinning the upstream source (k8s.io/kubernetes
pkg/volume/util, delta verified against kubernetes/kubernetes@52ba9013)
and enumerating every class of local modification, plus maintenance
rules (mechanical adaptations only in upstream-derived files; behavioral
changes go in substrate-owned files) and a re-sync procedure.

Mark each copied file with a greppable '// substrate:' header so the
patch surface is discoverable without diffing against upstream.
@thompsonmax
Add clusterTrustBundle as a SystemInfo volume data source
8afe3db 
A clusterTrustBundle data source projects the trust anchors of a named
ClusterTrustBundle (certificates.k8s.io/v1beta1) to a PEM file in a
SystemInfo volume -- the substrate analog of the Kubernetes
clusterTrustBundle projected volume source:

  systemInfo:
    dataSources:
    - clusterTrustBundle:
        name: my-trust-bundle
        path: ca.pem

ateapi resolves the bundle at actor start through a new informer-backed
lister and sanitizes the PEM the way kubelet does for projections (only
CERTIFICATE blocks, deduplicated, headers stripped -- internal/pemutil).
The wire spec carries only {path, pem_bundle}: atelet writes the
resolved bytes via the existing atomic writer and never talks to the
Kubernetes API. Starting the actor fails with an error naming the
bundle when it is missing, empty, or contains no certificates.

Because certificates.k8s.io/v1beta1 is feature-gated (enabled by
hack/create-kind-cluster.sh), ateapi probes for the API at startup and
degrades to a nil lister when absent -- registering the informer
blindly would hang WaitForCacheSync and with it ateapi boot. envtest
now serves the API so the functional suite matches real clusters.

Validation extends the SystemInfo CEL rules: clean relative paths,
unique paths across all data sources (dataSources MaxItems drops 32->8
to keep the cross-source rule inside the CEL cost budget), and the
ate-api-server ClusterRole gains read access to clustertrustbundles.

The identity e2e suite now creates a fixture bundle (junk PEM and a
duplicate certificate around a real one), projects it into the probe,
and asserts the delivered file is the sanitized PEM byte-for-byte.

Part of agent-substrate#932 (PR 1); the SystemInfo volume mechanism is agent-substrate#802.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@thompsonmax @ahmedtd