From 07e1f3a8e12940d89ba19f72ef78f0bdc86465ec Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Mon, 27 Jul 2026 20:09:30 -0700 Subject: [PATCH 1/2] feat(kms): make RPC certificate attestation configurable --- docs/security/public-security-reports.md | 2 +- docs/security/security-best-practices.md | 2 +- docs/security/security-model.md | 2 +- dstack/kms/kms.toml | 7 ++ dstack/kms/src/config.rs | 41 ++++++++++ dstack/kms/src/main.rs | 8 ++ dstack/kms/src/onboard_service.rs | 76 ++++++++++++++----- .../full-stack-compose/scripts/runner.sh | 2 +- tools/dev-stack.sh | 6 +- 9 files changed, 122 insertions(+), 24 deletions(-) diff --git a/docs/security/public-security-reports.md b/docs/security/public-security-reports.md index 06a8ce690..2c516080d 100644 --- a/docs/security/public-security-reports.md +++ b/docs/security/public-security-reports.md @@ -49,7 +49,7 @@ These issues were filed as concrete vulnerability reports, security audit findin | [#606](https://github.com/Dstack-TEE/dstack/issues/606) App keys and decrypted env files world-readable | Open | Valid hardening, open | Tightening secret-bearing file writes to owner-only permissions (`0600`) is valid defense-in-depth work with no expected compatibility cost | | [#607](https://github.com/Dstack-TEE/dstack/issues/607) `gateway_app_id = "any"` disables gateway identity pinning | Closed | Not a production vulnerability | `gateway_app_id` is KMS contract configuration and is publicly auditable. Production deployments must not use `"any"`. No code fix was applied | | [#608](https://github.com/Dstack-TEE/dstack/issues/608) `auth_api.type = "dev"` allows all authorization | Closed | Not a production vulnerability | Dev auth is measured runtime configuration, not a production mode. Production must use webhook/on-chain authorization. No code fix was applied | -| [#609](https://github.com/Dstack-TEE/dstack/issues/609) `quote_enabled = false` bypasses attestation | Closed | Not a production vulnerability | The flag is measured in runtime configuration and should fail production attestation policy. No code fix was applied | +| [#609](https://github.com/Dstack-TEE/dstack/issues/609) `quote_enabled = false` bypasses attestation | Closed | Not a production vulnerability | The flag was measured in runtime configuration and would fail production attestation policy, so no code fix was applied at the time. The setting has since been retired: current KMS has no `quote_enabled` and always attests its RPC certificate | | [#610](https://github.com/Dstack-TEE/dstack/issues/610) Unauthenticated bootstrap endpoint can overwrite root keys | Closed | Not a production vulnerability | The bootstrap endpoint does not accept caller-supplied root key material. Root keys are generated server-side, and the operator chooses which result to publish. No code fix was applied | | [#611](https://github.com/Dstack-TEE/dstack/issues/611) Unauthenticated `/finish` endpoint can shut down KMS onboard service | Closed | Not a production vulnerability | The onboard service is a short-lived setup flow. Premature shutdown causes operator retry, not persistent compromise or data loss. No code fix was applied | | [#612](https://github.com/Dstack-TEE/dstack/issues/612) Gateway `register_cvm` prefers stale `app_info` over live attestation | Closed | Not a production vulnerability | Cert-embedded `app_info` is extracted from attestation and signed by KMS. Preferring it avoids redundant extraction and is not a trust bypass. No code fix was applied | diff --git a/docs/security/security-best-practices.md b/docs/security/security-best-practices.md index b6dff185e..87cefccc1 100644 --- a/docs/security/security-best-practices.md +++ b/docs/security/security-best-practices.md @@ -84,7 +84,7 @@ Example app-compose.json: Development settings are intentionally easy to audit, but they are not production-safe. A production deployment should satisfy all of the following: -- KMS quote verification remains enabled. Do not deploy production KMS with `quote_enabled = false`. +- The KMS attests its own RPC certificate. Do not deploy production KMS with `attest_rpc_cert = false`. - KMS authorization uses webhook/on-chain policy. Do not use `auth_api.type = "dev"` with real key material. - The KMS contract pins a concrete gateway app id. Do not use `gateway_app_id = "any"` for production traffic. - TEE quotes are evaluated by deployment policy, including TCB status and expected OS/application measurements. diff --git a/docs/security/security-model.md b/docs/security/security-model.md index baab5d161..a3614548e 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -307,7 +307,7 @@ The one case dstack does not leave to downstream is a genuinely invalid TCB: `dc ### Development modes are auditable, not production-safe -dstack keeps several development switches as runtime or on-chain configuration rather than Cargo feature flags. Examples include KMS `quote_enabled = false`, `auth_api.type = "dev"`, and KMS contract `gateway_app_id = "any"`. These settings exist for local development and integration tests, not for production deployments. +dstack keeps several development switches as runtime or on-chain configuration rather than Cargo feature flags. Examples include KMS `attest_rpc_cert = false`, gateway `core.debug.insecure_skip_attestation = true`, KMS `auth_api.type = "dev"`, and KMS contract `gateway_app_id = "any"`. These settings exist for local development and integration tests, not for production deployments. This is intentional. Runtime configuration that affects the trust boundary is visible in attestation measurements or public contract state. Cargo feature gates are not automatically more auditable because feature unification can enable a feature through a dependency graph, and the resulting runtime behavior is not represented as a measured deployment setting. diff --git a/dstack/kms/kms.toml b/dstack/kms/kms.toml index 1ba51390b..08e954662 100644 --- a/dstack/kms/kms.toml +++ b/dstack/kms/kms.toml @@ -31,6 +31,13 @@ site_name = "" # outside a TEE (e.g. local dev/testing) where the local guest agent socket # is unavailable. enforce_self_authorization = true +# Whether the KMS embeds an attestation in its own RPC certificate. Set false +# only for local dev/testing where the KMS runs outside a TEE. This narrows what +# the KMS asserts about itself and relaxes no verification: quotes presented to +# the KMS are still fully checked, a guest that accepts an unattested +# certificate simply does not extend mr-kms, and another KMS refuses to onboard +# from one. +attest_rpc_cert = true # AMD SEV-SNP key/cert release remains disabled unless this local KMS gate is # explicitly enabled. External auth policy must still allow the verified # BootInfo before any sensitive material is returned. diff --git a/dstack/kms/src/config.rs b/dstack/kms/src/config.rs index 614c0e125..4d95b0e17 100644 --- a/dstack/kms/src/config.rs +++ b/dstack/kms/src/config.rs @@ -55,6 +55,18 @@ pub(crate) struct KmsConfig { pub aws_nitro_tpm_key_release: bool, #[serde(default)] pub site_name: String, + /// Whether the KMS embeds an attestation in its own RPC certificate. + /// Defaults to `true`. Set `false` only for local dev/testing where the KMS + /// runs outside a TEE and cannot reach a guest agent socket. + /// + /// This narrows what the KMS asserts about itself; it relaxes no + /// verification anywhere. Quotes presented *to* the KMS are still fully + /// checked, so key release stays gated, and relying parties keep their own + /// policy: a guest accepts an unattested KMS certificate but then does not + /// extend `mr-kms`, so a remote verifier can still tell, and another KMS + /// refuses to onboard from one. + #[serde(default = "default_true")] + pub attest_rpc_cert: bool, /// Whether trusted RPCs require the KMS to first attest itself to its /// own auth API. Defaults to `true` (strict). Set `false` only for local /// dev/testing where the KMS runs outside a TEE and cannot reach a guest @@ -183,6 +195,7 @@ pub(crate) struct OnboardConfig { #[cfg(test)] mod tests { use super::*; + use rocket::figment::providers::{Format, Toml}; #[test] fn default_config_parses_with_admin_disabled_and_no_hash() { @@ -198,4 +211,32 @@ mod tests { ); assert!(!config.admin.insecure_no_auth); } + + #[test] + fn rpc_cert_is_attested_by_default() { + let figment = load_config_figment(None); + let config: KmsConfig = figment + .focus("core") + .extract() + .expect("kms.toml must parse into KmsConfig"); + assert!( + config.attest_rpc_cert, + "the KMS must attest its own RPC certificate unless explicitly told not to" + ); + } + + #[test] + fn omitting_the_key_keeps_the_attested_default() { + // Configs written before this key existed must keep working, and must + // land on the attested side. + #[derive(Deserialize)] + struct Probe { + #[serde(default = "default_true")] + attest_rpc_cert: bool, + } + let probe: Probe = Figment::from(Toml::string("")) + .extract() + .expect("an absent attest_rpc_cert must parse"); + assert!(probe.attest_rpc_cert); + } } diff --git a/dstack/kms/src/main.rs b/dstack/kms/src/main.rs index ba51ebb20..9b94b7e30 100644 --- a/dstack/kms/src/main.rs +++ b/dstack/kms/src/main.rs @@ -116,6 +116,14 @@ async fn main() -> Result<()> { let figment = config::load_config_figment(args.config.as_deref()); let config: KmsConfig = figment.focus("core").extract()?; + if !config.attest_rpc_cert { + warn!( + "attest_rpc_cert = false; the KMS RPC certificate carries no attestation, so \ + guests cannot verify which KMS they are talking to and will not extend \ + mr-kms. Intended for local development only" + ); + } + if config.onboard.enabled && !config.keys_exists() { info!("Onboarding"); run_onboard_service(config.clone(), figment.clone()).await?; diff --git a/dstack/kms/src/onboard_service.rs b/dstack/kms/src/onboard_service.rs index 8734d151d..d3db6485b 100644 --- a/dstack/kms/src/onboard_service.rs +++ b/dstack/kms/src/onboard_service.rs @@ -27,6 +27,7 @@ use ra_tls::{ }; use safe_write::safe_write; use sha2::Digest; +use tracing::info; use crate::{ config::KmsConfig, @@ -76,7 +77,7 @@ impl OnboardRpc for OnboardHandler { ensure_self_kms_allowed(&self.state.config, &self.state.attestation_verifier) .await .context("KMS is not allowed to bootstrap")?; - let keys = Keys::generate(&request.domain) + let keys = Keys::generate(&request.domain, self.state.config.attest_rpc_cert) .await .context("Failed to generate keys")?; @@ -355,12 +356,20 @@ struct Keys { } impl Keys { - async fn generate(domain: &str) -> Result { + async fn generate(domain: &str, attest_rpc_cert: bool) -> Result { let tmp_ca_key = KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256)?; let ca_key = KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256)?; let rpc_key = KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256)?; let k256_key = SigningKey::random(&mut rand::rngs::OsRng); - Self::from_keys(tmp_ca_key, ca_key, rpc_key, k256_key, domain).await + Self::from_keys( + tmp_ca_key, + ca_key, + rpc_key, + k256_key, + domain, + attest_rpc_cert, + ) + .await } async fn from_keys( @@ -369,6 +378,7 @@ impl Keys { rpc_key: KeyPair, k256_key: SigningKey, domain: &str, + attest_rpc_cert: bool, ) -> Result { let tmp_ca_cert = CertRequest::builder() .org_name("Dstack") @@ -386,20 +396,32 @@ impl Keys { .key(&ca_key) .build() .self_signed()?; - let pubkey = rpc_key.public_key_der(); - let report_data = QuoteContentType::RaTlsCert.to_report_data(&pubkey); - let response = app_attest(report_data.to_vec()) - .await - .context("Failed to get quote")?; - let attestation = VersionedAttestation::from_bytes(&response.attestation) - .context("Invalid attestation")?; + // The only place the KMS embeds its own attestation. Skipping it lets + // the KMS run outside a TEE for development; it does not affect the + // verification of quotes presented *to* the KMS, which is a separate + // path (main_service::ensure_app_attestation_allowed) and stays on. + let attestation = if attest_rpc_cert { + let pubkey = rpc_key.public_key_der(); + let report_data = QuoteContentType::RaTlsCert.to_report_data(&pubkey); + let response = app_attest(report_data.to_vec()).await.context( + "failed to get a quote for the KMS RPC certificate. The KMS attests \ + itself through the dstack guest agent, so it must run inside a dstack \ + CVM. For local development set attest_rpc_cert = false", + )?; + Some( + VersionedAttestation::from_bytes(&response.attestation) + .context("Invalid attestation")?, + ) + } else { + None + }; // Sign WWW server cert with KMS cert let rpc_cert = CertRequest::builder() .subject(domain) .alt_names(&[domain.to_string()]) .special_usage("kms:rpc") - .maybe_attestation(Some(&attestation)) + .maybe_attestation(attestation.as_ref()) .key(&rpc_key) .build() .signed_by(&ca_cert, &ca_key)?; @@ -483,7 +505,15 @@ impl Keys { KeyPair::from_pem(&tmp_ca_key_pem).context("Failed to parse tmp CA key")?; let ecdsa_key = SigningKey::from_slice(&root_k256_key).context("Failed to parse ECDSA key")?; - Self::from_keys(tmp_ca_key, ca_key, rpc_key, ecdsa_key, domain).await + Self::from_keys( + tmp_ca_key, + ca_key, + rpc_key, + ecdsa_key, + domain, + cfg.attest_rpc_cert, + ) + .await } fn store(&self, cfg: &KmsConfig) -> Result<()> { @@ -527,12 +557,22 @@ pub(crate) async fn update_certs(cfg: &KmsConfig) -> Result<()> { let domain = domain.trim(); // Regenerate certificates using existing keys - let keys = Keys::from_keys(tmp_ca_key, ca_key, rpc_key, k256_key, domain) - .await - .context("Failed to regenerate certificates")?; - - // Write the new certificates to files + let keys = Keys::from_keys( + tmp_ca_key, + ca_key, + rpc_key, + k256_key, + domain, + cfg.attest_rpc_cert, + ) + .await + .context("Failed to regenerate certificates")?; + + // Write the new certificates to files. This runs on every start, so a + // hand-placed certificate is replaced -- say so, because the old silence + // made that look like the file had survived. keys.store_certs(cfg)?; + info!("Reissued the KMS RPC certificate for {domain}"); Ok(()) } @@ -541,7 +581,7 @@ pub(crate) async fn bootstrap_keys(cfg: &KmsConfig, verifier: &AttestationVerifi ensure_self_kms_allowed(cfg, verifier) .await .context("KMS is not allowed to auto-bootstrap")?; - let keys = Keys::generate(&cfg.onboard.auto_bootstrap_domain) + let keys = Keys::generate(&cfg.onboard.auto_bootstrap_domain, cfg.attest_rpc_cert) .await .context("Failed to generate keys")?; keys.store(cfg)?; diff --git a/test-suites/full-stack-compose/scripts/runner.sh b/test-suites/full-stack-compose/scripts/runner.sh index 285d6a0c6..0143ff6fb 100755 --- a/test-suites/full-stack-compose/scripts/runner.sh +++ b/test-suites/full-stack-compose/scripts/runner.sh @@ -697,7 +697,7 @@ stop_network_probe() { assert_no_insecure_shortcuts() { log "auditing rendered manifests for production trust settings" local manifest - local forbidden='quote_enabled[[:space:]]*=[[:space:]]*false|enforce_self_authorization[[:space:]]*=[[:space:]]*false|verify[[:space:]]*=[[:space:]]*false' + local forbidden='quote_enabled[[:space:]]*=[[:space:]]*false|attest_rpc_cert[[:space:]]*=[[:space:]]*false|insecure_skip_attestation[[:space:]]*=[[:space:]]*true|enforce_self_authorization[[:space:]]*=[[:space:]]*false|verify[[:space:]]*=[[:space:]]*false' if grep -R -E "$forbidden" "$WORK_DIR"/*.app-compose.json; then die "rendered app manifest contains a forbidden development trust setting" fi diff --git a/tools/dev-stack.sh b/tools/dev-stack.sh index af43e8351..8c900bc51 100755 --- a/tools/dev-stack.sh +++ b/tools/dev-stack.sh @@ -181,6 +181,7 @@ mandatory = false [core] cert_dir = "$CERTS_DIR" +attest_rpc_cert = false [core.gpu] enabled = $VMM_ENABLE_GPU @@ -189,7 +190,6 @@ enabled = $VMM_ENABLE_GPU type = "dev" [core.onboard] -quote_enabled = false address = "127.0.0.1" port = $KMS_RPC_LISTEN_PORT auto_bootstrap_domain = "$KMS_DOMAIN" @@ -215,7 +215,9 @@ mandatory = false [core] kms_url = "https://localhost:$KMS_RPC_LISTEN_PORT" rpc_domain = "$GATEWAY_DOMAIN" -run_in_dstack = false + +[core.debug] +insecure_skip_attestation = true [core.sync] enabled = false From 2d43948fa6d360d33671d11955cb927ce616e998 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Mon, 27 Jul 2026 20:30:13 -0700 Subject: [PATCH 2/2] fix(kms): enforce RPC certificate attestation mode --- docs/security/public-security-reports.md | 2 +- dstack/kms/src/main.rs | 6 ++++++ tools/dev-stack.sh | 1 + 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/security/public-security-reports.md b/docs/security/public-security-reports.md index 2c516080d..bf935c4de 100644 --- a/docs/security/public-security-reports.md +++ b/docs/security/public-security-reports.md @@ -49,7 +49,7 @@ These issues were filed as concrete vulnerability reports, security audit findin | [#606](https://github.com/Dstack-TEE/dstack/issues/606) App keys and decrypted env files world-readable | Open | Valid hardening, open | Tightening secret-bearing file writes to owner-only permissions (`0600`) is valid defense-in-depth work with no expected compatibility cost | | [#607](https://github.com/Dstack-TEE/dstack/issues/607) `gateway_app_id = "any"` disables gateway identity pinning | Closed | Not a production vulnerability | `gateway_app_id` is KMS contract configuration and is publicly auditable. Production deployments must not use `"any"`. No code fix was applied | | [#608](https://github.com/Dstack-TEE/dstack/issues/608) `auth_api.type = "dev"` allows all authorization | Closed | Not a production vulnerability | Dev auth is measured runtime configuration, not a production mode. Production must use webhook/on-chain authorization. No code fix was applied | -| [#609](https://github.com/Dstack-TEE/dstack/issues/609) `quote_enabled = false` bypasses attestation | Closed | Not a production vulnerability | The flag was measured in runtime configuration and would fail production attestation policy, so no code fix was applied at the time. The setting has since been retired: current KMS has no `quote_enabled` and always attests its RPC certificate | +| [#609](https://github.com/Dstack-TEE/dstack/issues/609) `quote_enabled = false` bypasses attestation | Closed | Not a production vulnerability | The flag was measured in runtime configuration and would fail production attestation policy, so no code fix was applied at the time. The setting has since been retired; current KMS uses `attest_rpc_cert`, which defaults to `true`, to control RPC certificate attestation | | [#610](https://github.com/Dstack-TEE/dstack/issues/610) Unauthenticated bootstrap endpoint can overwrite root keys | Closed | Not a production vulnerability | The bootstrap endpoint does not accept caller-supplied root key material. Root keys are generated server-side, and the operator chooses which result to publish. No code fix was applied | | [#611](https://github.com/Dstack-TEE/dstack/issues/611) Unauthenticated `/finish` endpoint can shut down KMS onboard service | Closed | Not a production vulnerability | The onboard service is a short-lived setup flow. Premature shutdown causes operator retry, not persistent compromise or data loss. No code fix was applied | | [#612](https://github.com/Dstack-TEE/dstack/issues/612) Gateway `register_cvm` prefers stale `app_info` over live attestation | Closed | Not a production vulnerability | Cert-embedded `app_info` is extracted from attestation and signed by KMS. Preferring it avoids redundant extraction and is not a trust bypass. No code fix was applied | diff --git a/dstack/kms/src/main.rs b/dstack/kms/src/main.rs index 9b94b7e30..478c5c52a 100644 --- a/dstack/kms/src/main.rs +++ b/dstack/kms/src/main.rs @@ -134,6 +134,12 @@ async fn main() -> Result<()> { info!("Updating certs"); if let Err(err) = onboard_service::update_certs(&config).await { + if config.attest_rpc_cert { + return Err(err).context( + "Failed to reissue the attested KMS RPC certificate; refusing to start with a \ + potentially unattested certificate", + ); + } warn!("Failed to update certs: {err}"); }; diff --git a/tools/dev-stack.sh b/tools/dev-stack.sh index 8c900bc51..51bce204f 100755 --- a/tools/dev-stack.sh +++ b/tools/dev-stack.sh @@ -182,6 +182,7 @@ mandatory = false [core] cert_dir = "$CERTS_DIR" attest_rpc_cert = false +enforce_self_authorization = false [core.gpu] enabled = $VMM_ENABLE_GPU