egress: MITM tunneled TLS with per-SNI certificate minting

[haiyanmeng]

New changes:

• atenet sdsmint — an SDS server that mints a short-lived TLS leaf per SNI on demand. Runs as a native sidecar beside Envoy, sharing the atenet image but staying a separate container so the MITM signing key never lands on the data plane.
• mitm_listener — the CONNECT route now hands the upgraded stream here instead of to the forward proxy. Two chains,
  picked by tls_inspector: tls terminates using Envoy 1.37's on_demand_secret selector + sni cert mapper (hence the
  1.34 → 1.37 bump), raw_buffer proxies plaintext http:// and enforces the allowlist as an :authority match, since
  there's no mint on that path to refuse.
• CA plumbing — internal/localca gains GenerateCA(GenerateOptions); kubectl-ate admin make-ca-pool gains --key-type,
  --common-name; install-ate.sh creates the egress-mitm-ca-pool secret before the Deployment.
• e2e (internal/e2e/suites/sdsmint) covers the tunnelled handshake, the leaf Envoy serves, and actor identity across the MITM, against the new egressprobe fixture.

Not included:

• how to make actors trust egress-mitm-ca-pool

This is to address #823

It's a good idea to open an issue first for discussion.

• Tests pass
• Appropriate changes to documentation are included in the PR

[Bowei Du (bowei)]

• Remove the intermediate CA code
• Some of the options seem like we don't need to expose. If so, then the logic inside should be simplified.
• Move the CA generating logic into its own package (suggest sdsmint/certauth) that exports a very narrow interface out to the server. Seem like the only things that need to be exported are a Get() and Forget()
• I don't know if we need an optional minter.Forgetter() -- this comment applies to other places in the code that seem to have more options than we need.
• Go through the comments and scrub for the agent-based self-conversation. The comments are pretty verbose in some places and look more like agent thinking tokens than code comments.

I mostly looked at the server code. Once you get the simplifications in, I will take a more detailed look at the rest of the change.

[haiyanmeng]

• Remove the intermediate CA code

Done

• Some of the options seem like we don't need to expose. If so, then the logic inside should be simplified.

Done

• Move the CA generating logic into its own package (suggest sdsmint/certauth) that exports a very narrow interface out to the server. Seem like the only things that need to be exported are a Get() and Forget()

Done

• I don't know if we need an optional minter.Forgetter() -- this comment applies to other places in the code that seem to have more options than we need.

minter.Forgetter() iis called through the optional interface in cmd/atenet/internal/sdsmint/deltastream.go:391:

  if f, ok := d.srv.minter.(Forgetter); ok {
      for _, name := range expired {
          f.Forget(name)
      }
  }

• Go through the comments and scrub for the agent-based self-conversation. The comments are pretty verbose in some places and look more like agent thinking tokens than code comments.

Done

[Taahir Ahmed (ahmedtd)]

Let's leave the behavior to just take the first --- localca.Pool should be tracking which localca.CA is active for signing. I will send a PR to do this (and add rotation commands for the CA secrets).

[haiyanmeng]

Taahir Ahmed (@ahmedtd) , are you suggesting removing this flag and take the first CA from the pool by default?

[haiyanmeng]

Does your PR need to block this PR?

[Bowei Du (bowei)]

Can we default to first for now or it doesn't work without this flag.

[haiyanmeng]

It works as is. Here is how the sdsmint container in the egress gateway Pod sets it

        args:
        - "sdsmint"
        - "--uds-path=/var/run/sdsmint/sdsmint.sock"
        - "--ca-pool-path=/run/ca-state/mitm-pool.json"
        - "--ca-id=mitm"

[Lior Lieberman (LiorLieberman)]

I think we need to start more gradual. PR is doing way too many things that we dont need right now;

• remove all the state of the world code (SotW) and all of its StreamSecrets/FetchSecrets/buildSotW. We dont need it since we are doing delta xds.

• remove all metrics - lets have a separate follow-up pr for metrics

• I think we should remove all caching for now, we should have a separate issue to add caching properly. For the first implementation we dont need to have cache in sds minter. Envoy is caching the certificate anyway. We can look into existing lru solutions later.

Also, mint is super cheap. The generation of the key pair is what takes time. so if we generate keypair at startup, every mint is super fast.

• we dont need all the proactive rotation in deltastream.go + leafkey.go (rotateStale, ticker, reuseFraction/rotateFraction) - this is just because the PR is opinionated. In Delta xds - can do something like per-resource ttl - which would require envoy to re-request the secret. - but we dont need it right now

• All the withdrawal login in deltastreams.go - is it just a memory optimization? why do we need it right now?

• I dont think we need the pproof endpoint for this PR

[haiyanmeng]

Also, mint is super cheap. The generation of the key pair is what takes time. so if we generate keypair at startup, every mint is super fast.

Agreed

• remove all the state of the world code (SotW) and all of its StreamSecrets/FetchSecrets/buildSotW. We dont need it since we are doing delta xds.
• remove all metrics - lets have a separate follow-up pr for metrics
• I dont think we need the pproof endpoint for this PR

These can be done.

[haiyanmeng]

• I think we should remove all caching for now, we should have a separate issue to add caching properly. For the first implementation we dont need to have cache in sds minter. Envoy is caching the certificate anyway. We can look into existing lru solutions later.

This can be done too.

• All the withdrawal login in deltastreams.go - is it just a memory optimization? why do we need it right now?

This is to prevent Envoy from being OOM.

• we dont need all the proactive rotation in deltastream.go + leafkey.go (rotateStale, ticker, reuseFraction/rotateFraction) - this is just because the PR is opinionated. In Delta xds - can do something like per-resource ttl - which would require envoy to re-request the secret. - but we dont need it right now

If we remove the rotation logic without adding an alternative like per-resource ttl, the UX may be bad. The secret Envoy holds expires, and Envoy keeps presenting it. Every client handshake to that host then fails on an expired cert. Let me test out this behavior.

[Lior Lieberman (LiorLieberman)]

If we remove the rotation logic without adding an alternative like per-resource ttl, the UX will be bad. The secret Envoy holds expires, and Envoy keeps presenting it. Every client handshake to that host then fails on an expired cert.

And would it retry and then fetch the new cert/secret again?

[Taahir Ahmed (ahmedtd)]

Also, mint is super cheap. The generation of the key pair is what takes time. so if we generate keypair at startup, every mint is super fast.

This isn't true for ECDSA as I understand it --- ECDSA keygen just reading random bytes from crypto/rand, whereas the signatures for creating the certificates require bigint math.

It's more performant than RSA, but I don't think we can get away without caching for a production system.

[Bowei Du (bowei)]

We should also be careful about exhausting randomness sources.

[haiyanmeng]

Bowei Du (@bowei) , Taahir Ahmed (@ahmedtd) , Lior Lieberman (@LiorLieberman) , I updated the PR:

1. Removed cache, metrics, pprof, SotW, cert rotation from sdsmint;
2. Added resource TTL. (I verified the resource TTL works as expected. See https://github.com/haiyanmeng/substrate/blob/sds-3/poc/sdsmint/expiry/README.md).

PTAL.

[Lior Lieberman (LiorLieberman)]

Thanks for all the hard work, sds server code mostly looks good to me.

Left some comments. Plus we need to think how to gurad this with a feature flag or something.

Also - any resolution on https://github.com/agent-substrate/substrate/pull/871/changes#r3771056979?

[Lior Lieberman (LiorLieberman)]

what is this for?

[Lior Lieberman (LiorLieberman)]

dont we already have some egress testing go code machinery (if not, how do we test egress now?) maybe this can be integrated there vs creating a new package?

[Lior Lieberman (LiorLieberman)]

are we going to have an internal listener now for all envoy deployments? we should likely guard it with a feature flag.

Also does this code/config right now only pass to the envoy internal listener when MITM is needed? (e.g actor dialed HTTPS) or every time? the impl should be the former.

[haiyanmeng]

are we going to have an internal listener now for all envoy deployments? we should likely guard it with a feature flag.

#926 adds --experimental-use-sdsmint to select the egress manifest. The plan is to submit #871 first, then submit #926 as a fast followup.

[haiyanmeng]

Also does this code/config right now only pass to the envoy internal listener when MITM is needed? (e.g actor dialed HTTPS) or every time? the impl should be the former.

mitm_listener has two filter chains. One for tls traffic (transport_protocol: tls), the other for plaintext traffic (transport_protocol: raw_buffer).

[haiyanmeng]

Thanks for all the hard work, sds server code mostly looks good to me.

Left some comments. Plus we need to think how to gurad this with a feature flag or something.

Also - any resolution on https://github.com/agent-substrate/substrate/pull/871/changes#r3771056979?

Not yet. Will follow with Bowei and Taahir on that.