feat(inference): multi-provider LLM upstreams (Anthropic, Ollama) + pluggable guardrail pipeline (OpenAI Moderation)

[John Seong (sandole)]

Summary

This is a first slice of the multi-cloud provider + guardrails roadmap theme. It lets an InferencePolicy pick the inference provider and declare a guardrail pipeline, and teaches the router to actually honor both:

spec:
  provider: anthropic          # or: ollama, azure-openai (default), bedrock (schema only, router says 501)
  guardrails:
    - provider: openai-moderation
      applyTo: both            # input | output | both

I started with Anthropic and Ollama because they're the two cheapest to do properly: Anthropic gets native Messages pass-through on /anthropic/v1/messages (streaming, tool use and multi-modal all survive, which the existing translation route couldn't offer), and Ollama is OpenAI-compatible so chat completions just work, buffered and SSE, with token metering intact.

The part I'd most like eyes on is the streaming guardrail. Scanning a stream after the fact is theater, so the SSE guard holds chunks back and only releases a window once a scan has covered it (default 1000 chars per window, GUARDRAIL_STREAM_SCAN_CHARS to tune). Flagged streams get cut with a structured error frame. The trade-off is chunkier delivery when a pipeline is active; policies without guardrails don't pay anything.

Everything is fail-closed on purpose. A policy that names a provider the router can't serve gets a 501 (bedrock) or 503 (missing key/endpoint) instead of quietly falling back to Azure, and a declared guardrail stage that can't run blocks the request rather than becoming an open gate. Provider keys (ANTHROPIC_API_KEY, OPENAI_MODERATION_API_KEY) only ever exist on the router sidecar - the reconciler forwards them from controller env the same way the dev creds already travel, and the router strips/replaces any x-api-key an agent tries to smuggle through (there's an integration test for exactly that).

Helm CRD template regenerated via the drift-test dumper. Endpoints (not secrets) joined CONFIG_HASH_INPUTS. Both Copilot review comments are addressed in 4b48b87c (outcome-accurate SSE error frames; bounded stream scan context).

Related Issues

No existing issue tracks this - the provider-expansion note in docs/architecture.md asks for a feature request per provider, so I'm happy to file one (auth model, Foundry-feature preservation, the works) and link it here if that's the flow you want. Follow-up slices (Bedrock, Vertex, vLLM; Bedrock Guardrails, Model Armor) would extend the same two seams: provider::ProviderKind and the guardrails::Guardrail trait.

Type of Change

• New feature

Checklist

• Code compiles/builds without errors (cargo check --workspace --tests)
• Tests pass (857 controller + 1,094 router, including the new integration suite)
• Linting passes (clippy adds zero warnings from this change; the one collapsible_if in routes/mod.rs also fires on main with this toolchain)
• Documentation updated (docs/api/crd-reference.md, docs/architecture.md, CHANGELOG.md)
• No secrets committed
• Helm chart updated (crd-inferencepolicy.yaml regenerated)

Testing

Automated (all offline):

• Controller: compile round-trips for the new fields, enum wire-tag pins, version-hash change detection, CEL coverage, helm drift.
• Router units: provider resolution truth table, loader back-compat with pre-slice profiles, moderation response parsing (fail-closed on malformed), and the SSE hold-and-release guard against a fake backend - clean stream intact, flagged text withheld and cut, events split across chunk boundaries, scan errors, keepalives, threshold windowing, bounded scan context.
• Wiremock end-to-end (inference-router/tests/multi_provider_guardrails.rs): fake Ollama (URL shape, no credentials sent), fake Anthropic (router key replaces agent-supplied key, anthropic-version injected), fake moderation endpoint (flag/pass/500→fail-closed), and missing-key pipeline construction failure.

Live, running the router binary locally:

• Anthropic Messages against the real api.anthropic.com, non-streaming and SSE, plus the 501 on chat/completions under an Anthropic policy.
• Chat completions (buffered + SSE) against a local Ollama, and a provider hot-swap through the policy reload watcher without a restart.
• Guardrails: fail-closed with no key configured (503 guardrail_misconfigured), input blocked pre-flight (403), and a mid-stream output cut - clean prompt, model emits the flagged word, zero bytes of model text reach the client, just the structured error frame + [DONE]. kars_guardrail_scans_total counted every scan by direction/outcome. The moderation verdict itself came from a local stub (no OpenAI key on this machine); everything around it was real.

Enforcement edge cases the tests + live run cover explicitly:

• The streaming guard holds bytes until a scan covers them even when a chunk boundary falls inside a data: line, and scans non-JSON data: payloads as raw text rather than fast-releasing an unrecognised frame. A 40-char-threshold live run delivered only the error frame on a cut, zero content deltas.
• All three Responses-API recovery paths (cached responses-only, stream-400 fallback, buffered-400 fallback) run the output scan.
• spec.provider is the sole routing selector; the pre-existing modelPreference.primary.provider tag stays informational, so an unchanged CR that only set a model preference keeps its Azure upstream (verified live).
• The sibling inference routes (/v1/completions, /v1/responses, /v1/embeddings, image generation) don't implement provider routing or guardrails, so they fail closed (501 on a non-Azure provider, 403 when guardrails are declared) rather than silently bypass the policy and live-verified 501 on /v1/embeddings and /v1/responses. Text over 16k chars is scanned in successive windows, not truncated.

[Copilot]

Pull request overview

This PR extends Kars’ inference plane to support policy-driven multi-provider routing (Anthropic, Ollama; Bedrock explicitly unimplemented) and introduces a pluggable guardrail pipeline (first backend: OpenAI Moderation) enforced for buffered and SSE streaming responses.

Changes:

• Adds InferencePolicy.spec.provider (typed enum with kebab-case wire tags) and spec.guardrails[] (ordered stages) with CRD/CEL validation, compilation into the router-consumed profile JSON, and updated docs/changelog.
• Implements router-side provider resolution + per-provider proxy behavior (URL shapes + auth schemes), keeping credentials router-side only.
• Implements guardrail pipeline enforcement for input/output, including hold-and-release scanning for SSE streams, plus metrics for scan outcomes.

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
inference-router/tests/proxy_fake_upstream.rs	Updates tests to populate new UpstreamConfig fields (provider, api_key).
inference-router/tests/policy_status_endpoint.rs	Extends test router Config literals with new provider/guardrail fields.
inference-router/tests/multi_provider_guardrails.rs	Adds end-to-end wiremock tests for Anthropic/Ollama forwarding and moderation guardrail behavior.
inference-router/tests/failover_walk.rs	Updates failover tests for expanded UpstreamConfig.
inference-router/tests/egress_blocked_endpoint.rs	Extends test Config literals with new provider/guardrail fields.
inference-router/tests/agt_governance_integration.rs	Extends test Config literals with new provider/guardrail fields.
inference-router/src/routes/mod.rs	Adds apply_provider_resolution to retarget upstream per policy/provider precedence.
inference-router/src/routes/chat_completions.rs	Enforces provider selection + guardrails for chat completions, including streaming and recovery paths.
inference-router/src/routes/anthropic_messages.rs	Applies provider resolution + guardrails to the Anthropic Messages route (pass-through + translated paths).
inference-router/src/proxy.rs	Makes proxy forwarding provider-aware (auth header scheme + URL shaping) and adds UpstreamConfig::azure constructor.
inference-router/src/provider.rs	Introduces provider tag parsing/resolution with fail-closed semantics and config-backed targets.
inference-router/src/metrics.rs	Adds kars_guardrail_scans_total{provider,direction,outcome} metric.
inference-router/src/lib.rs	Exports new guardrails and provider modules.
inference-router/src/inference_policy_loader.rs	Loads compiled policy provider + guardrails fields into snapshots for per-request enforcement.
inference-router/src/guardrails.rs	Implements guardrail pipeline (OpenAI Moderation) and SSE hold-and-release stream guarding.
inference-router/src/failover.rs	Updates test helpers for expanded UpstreamConfig.
inference-router/src/config.rs	Adds router config surface for Anthropic/Ollama endpoints and Moderation backend config + secret mount loading.
docs/architecture.md	Updates architecture notes to include Anthropic/Ollama as wired providers.
docs/api/crd-reference.md	Documents spec.provider and spec.guardrails[] and clarifies precedence/semantics.
deploy/helm/kars/templates/crd-inferencepolicy.yaml	Regenerates CRD schema to include provider/guardrails fields + validations.
controller/src/reconciler/mod.rs	Forwards router-only env vars for provider endpoints/keys and moderation backend to sidecars (skipping empty).
controller/src/inference_policy.rs	Adds typed InferenceProvider + guardrail stage types to the CRD model.
controller/src/inference_policy_reconciler.rs	Enforces bundleRef mutual exclusion with provider/guardrails and keeps bundle canonical format unchanged.
controller/src/inference_policy_compile.rs	Emits compiled profile JSON keys provider and guardrails (null when absent) with wire-tag pin tests.
controller/src/crd_validations.rs	Adds CEL validations for bundleRef exclusivity and guardrail stage count bounds (1–8).
controller/src/config_hash.rs	Adds provider/guardrail endpoint env vars (non-secret) to config hash inputs.
CHANGELOG.md	Documents the multi-provider + guardrails slice and associated behavior/constraints.

[John Seong (sandole)]

John Seong (@sandole) please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

Contribution License Agreement

This Contribution License Agreement (“Agreement”) is agreed to by the party signing below (“You”), and conveys certain license rights to Microsoft Corporation and its affiliates (“Microsoft”) for Your contributions to Microsoft open source projects. This Agreement is effective as of the latest signature date below.

1. Definitions.
   “Code” means the computer software code, whether in human-readable or machine-executable form,
   that is delivered by You to Microsoft under this Agreement.
   “Project” means any of the projects owned or managed by Microsoft and offered under a license
   approved by the Open Source Initiative (www.opensource.org).
   “Submit” is the act of uploading, submitting, transmitting, or distributing code or other content to any
   Project, including but not limited to communication on electronic mailing lists, source code control
   systems, and issue tracking systems that are managed by, or on behalf of, the Project for the purpose of
   discussing and improving that Project, but excluding communication that is conspicuously marked or
   otherwise designated in writing by You as “Not a Submission.”
   “Submission” means the Code and any other copyrightable material Submitted by You, including any
   associated comments and documentation.
2. Your Submission. You must agree to the terms of this Agreement before making a Submission to any
   Project. This Agreement covers any and all Submissions that You, now or in the future (except as
   described in Section 4 below), Submit to any Project.
3. Originality of Work. You represent that each of Your Submissions is entirely Your original work.
   Should You wish to Submit materials that are not Your original work, You may Submit them separately
   to the Project if You (a) retain all copyright and license information that was in the materials as You
   received them, (b) in the description accompanying Your Submission, include the phrase “Submission
   containing materials of a third party:” followed by the names of the third party and any licenses or other
   restrictions of which You are aware, and (c) follow any other instructions in the Project’s written
   guidelines concerning Submissions.
4. Your Employer. References to “employer” in this Agreement include Your employer or anyone else
   for whom You are acting in making Your Submission, e.g. as a contractor, vendor, or agent. If Your
   Submission is made in the course of Your work for an employer or Your employer has intellectual
   property rights in Your Submission by contract or applicable law, You must secure permission from Your
   employer to make the Submission before signing this Agreement. In that case, the term “You” in this
   Agreement will refer to You and the employer collectively. If You change employers in the future and
   desire to Submit additional Submissions for the new employer, then You agree to sign a new Agreement
   and secure permission from the new employer before Submitting those Submissions.
5. Licenses.

• Copyright License. You grant Microsoft, and those who receive the Submission directly or
  indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license in the
  Submission to reproduce, prepare derivative works of, publicly display, publicly perform, and distribute
  the Submission and such derivative works, and to sublicense any or all of the foregoing rights to third
  parties.
• Patent License. You grant Microsoft, and those who receive the Submission directly or
  indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license under
  Your patent claims that are necessarily infringed by the Submission or the combination of the
  Submission with the Project to which it was Submitted to make, have made, use, offer to sell, sell and
  import or otherwise dispose of the Submission alone or with the Project.
• Other Rights Reserved. Each party reserves all rights not expressly granted in this Agreement.
  No additional licenses or rights whatsoever (including, without limitation, any implied licenses) are
  granted by implication, exhaustion, estoppel or otherwise.

6. Representations and Warranties. You represent that You are legally entitled to grant the above
   licenses. You represent that each of Your Submissions is entirely Your original work (except as You may
   have disclosed under Section 3). You represent that You have secured permission from Your employer to
   make the Submission in cases where Your Submission is made in the course of Your work for Your
   employer or Your employer has intellectual property rights in Your Submission by contract or applicable
   law. If You are signing this Agreement on behalf of Your employer, You represent and warrant that You
   have the necessary authority to bind the listed employer to the obligations contained in this Agreement.
   You are not expected to provide support for Your Submission, unless You choose to do so. UNLESS
   REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING, AND EXCEPT FOR THE WARRANTIES
   EXPRESSLY STATED IN SECTIONS 3, 4, AND 6, THE SUBMISSION PROVIDED UNDER THIS AGREEMENT IS
   PROVIDED WITHOUT WARRANTY OF ANY KIND, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTY OF
   NONINFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
7. Notice to Microsoft. You agree to notify Microsoft in writing of any facts or circumstances of which
   You later become aware that would make Your representations in this Agreement inaccurate in any
   respect.
8. Information about Submissions. You agree that contributions to Projects and information about
   contributions may be maintained indefinitely and disclosed publicly, including Your name and other
   information that You submit with Your Submission.
9. Governing Law/Jurisdiction. This Agreement is governed by the laws of the State of Washington, and
   the parties consent to exclusive jurisdiction and venue in the federal courts sitting in King County,
   Washington, unless no federal subject matter jurisdiction exists, in which case the parties consent to
   exclusive jurisdiction and venue in the Superior Court of King County, Washington. The parties waive all
   defenses of lack of personal jurisdiction and forum non-conveniens.
10. Entire Agreement/Assignment. This Agreement is the entire agreement between the parties, and
    supersedes any and all prior agreements, understandings or communications, written or oral, between
    the parties relating to the subject matter hereof. This Agreement may be assigned by Microsoft.

@microsoft-github-policy-service agree

[Copilot]

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated 4 comments.

Suppressed comments (5)

inference-router/src/guardrails.rs:731

• If an SSE data: payload is not valid JSON (or doesn't match the expected dialect), unscanned stays at 0 and the code may treat the chunk as safe to release. That breaks the “no model text reaches the client before a scan has covered it” contract and makes the guard bypassable by non-JSON/variant frames. A safe fallback is to treat unparseable data: payloads as text-bearing and include them in the scan context (or fail closed).

            if let Ok(event) = serde_json::from_str::<serde_json::Value>(payload)
                && let Some(text) = delta_text_from_event(self.dialect, &event)
            {
                self.unscanned += text.chars().count();
                self.accumulated.push_str(&text);
            }

inference-router/src/guardrails.rs:745

• In on_chunk, when unscanned == 0 the code immediately releases all held bytes. This is unsafe when line_carry contains a partial data: line split across chunks: the first chunk can contain unscanned model output bytes but no newline yet, so it gets released before the JSON payload is parsed and scanned. Only release immediately when there is no partial line carry buffered.

            // Fast path: window not full. Chunks carrying no delta
            // text at all (keepalives, role/annotation frames) are
            // safe to release immediately when nothing text-bearing
            // is being held alongside them.
            if self.unscanned == 0 {
                return SseGuardStep::Release(std::mem::take(&mut self.held));
            }

inference-router/src/routes/anthropic_messages.rs:287

• PR description claims x-kars-decision* headers are attached on every block. The Anthropic route returns deny_response(...) for provider-resolution and guardrail failures, but this helper currently doesn’t inject the decision headers (unlike the /v1/chat/completions path). That’s observable API drift for clients relying on those headers for auditing/telemetry.

    // Multi-provider slice: retarget at the policy-selected provider.
    // Fails closed — see `routes::apply_provider_resolution`.
    if let Err(e) = crate::routes::apply_provider_resolution(&state, &mut upstream, &policy) {
        tracing::warn!(
            target: "inference.audit",
            sandbox = %sandbox_name,
            inference_policy_digest = %policy.digest,
            decision = "deny",
            gate = "provider_resolution",
            error = %e,
            "InferencePolicy provider could not be resolved (anthropic route)"
        );
        let status = match e {
            ProviderError::Unimplemented { .. } => StatusCode::NOT_IMPLEMENTED,
            _ => StatusCode::SERVICE_UNAVAILABLE,
        };
        return deny_response(status, &e.to_string(), "api_error");
    }

docs/api/crd-reference.md:495

• Docs currently state spec.bundleRef is an alternative to inline provider / guardrails, but the controller-side reconciler explicitly notes the signed-bundle canonical format does not carry provider/guardrails yet (they’re forced to None when a bundle is used). As written, this suggests operators can move these fields into bundles when they currently cannot.

| `spec.bundleRef` | Signed OCI artifact alternative to inline `tokenBudget` / `contentSafety` / `modelPreference` / `provider` / `guardrails` / `displayName`. `appliesTo` always comes from the CR. |

deploy/helm/kars/templates/crd-inferencepolicy.yaml:198

• The CRD schema text says guardrail stages are materialized “at policy load time”, but the router builds the GuardrailPipeline per request from the policy snapshot (build_guardrail_pipeline in routes/chat_completions.rs). This description looks stale/misleading for operators troubleshooting runtime failures (which happen at request time).

                    A single stage of the router-side guardrail pipeline. The router
                    materialises each stage into a scanner (network client + policy)
                    at policy load time; a stage whose backend is not configured on
                    the router (e.g. missing moderation API key) fails the *request*

[Copilot]

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated no new comments.

Suppressed comments (6)

inference-router/src/proxy.rs:527

• Same as the buffered forward path: this wraps provider-specific credential resolution, not just token acquisition. The current message is misleading for Anthropic/Ollama.

    let credential = credential_for_upstream(&auth, copilot.as_deref(), &upstream)
        .await
        .context("Failed to acquire auth token")?;
    let headers = build_upstream_headers(&request_headers, &auth, &credential, &upstream.endpoint)?;

inference-router/src/routes/chat_completions.rs:387

• The explicit 501 returned when a policy selects the Anthropic provider doesn’t attach the canonical x-kars-decision* headers, unlike other provider/guardrail blocks in this handler. This makes denial telemetry inconsistent for downstream tooling that relies on these headers.

    if upstream.provider == ProviderKind::Anthropic {
        return errors::openai(
            StatusCode::NOT_IMPLEMENTED,
            "InferencePolicy selects provider 'anthropic', which serves the Anthropic \
             Messages API — send Anthropic-shaped requests to /anthropic/v1/messages \
             (chat-completions translation for Anthropic is not implemented)",
            "provider_unimplemented",
        )
        .into_response();

inference-router/src/proxy.rs:300

• This error context is now used for all provider types (including Anthropic API keys and unauthenticated Ollama), so the message “auth token” is misleading. Renaming it to “upstream credential” makes logs/errors accurate across providers.

This issue also appears on line 524 of the same file.

    let credential = credential_for_upstream(auth, copilot, upstream)
        .await
        .context("Failed to acquire auth token")?;

controller/src/crd_validations.rs:283

• The validation message claims “the bundle carries those content fields”, but this repo currently explicitly drops provider/guardrails when bundleRef is used (see merge_bundle_with_selector in this same PR). The message should not imply the bundle contains provider/guardrails yet.

        ValidationRule {
            rule: "!has(self.bundleRef) || (!has(self.tokenBudget) && !has(self.contentSafety) && !has(self.modelPreference) && !has(self.provider) && !has(self.guardrails) && !has(self.displayName))".into(),
            message: Some("spec.bundleRef is mutually exclusive with spec.tokenBudget, spec.contentSafety, spec.modelPreference, spec.provider, spec.guardrails, and spec.displayName; the bundle carries those content fields".into()),
            reason: Some("FieldValueInvalid".into()),
            ..ValidationRule::default()

deploy/helm/kars/templates/crd-inferencepolicy.yaml:331

• This CRD CEL validation message also claims “the bundle carries those content fields”, but bundle-sourced policies currently do not carry provider/guardrails yet (they are forced to defaults). The message should avoid implying otherwise.

            - message: spec.bundleRef is mutually exclusive with spec.tokenBudget, spec.contentSafety, spec.modelPreference, spec.provider, spec.guardrails, and spec.displayName; the bundle carries those content fields
              reason: FieldValueInvalid
              rule: '!has(self.bundleRef) || (!has(self.tokenBudget) && !has(self.contentSafety) && !has(self.modelPreference) && !has(self.provider) && !has(self.guardrails) && !has(self.displayName))'

docs/api/crd-reference.md:495

• The docs currently describe bundleRef as an alternative to inline provider/guardrails, which implies those fields can come from the signed bundle. In this PR they’re explicitly inline-only and mutually exclusive with bundleRef, and bundle-sourced policies drop them to defaults. Updating this line would prevent operator confusion.

| `spec.bundleRef` | Signed OCI artifact alternative to inline `tokenBudget` / `contentSafety` / `modelPreference` / `provider` / `guardrails` / `displayName`. `appliesTo` always comes from the CR. |

[Copilot]

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated no new comments.

Suppressed comments (3)

inference-router/src/proxy.rs:526

• Same as above: this path resolves provider-specific credentials (including unauthenticated Ollama), so “auth token” is no longer an accurate description of what failed.

    let credential = credential_for_upstream(&auth, copilot.as_deref(), &upstream)
        .await
        .context("Failed to acquire auth token")?;

inference-router/src/guardrails.rs:725

• SseGuardState::ingest_text decodes arbitrary upstream bytes via String::from_utf8_lossy(chunk) and appends into line_carry. If a multi-byte UTF-8 sequence is split across chunk boundaries, from_utf8_lossy can replace it with U+FFFD, so the guardrail scans a different text than what the client ultimately receives. This can weaken the “no unscanned text is released” guarantee for non-ASCII output.

Consider buffering as raw bytes and only decoding complete lines (split on \n) with std::str::from_utf8, failing closed (cut the stream with a guardrail error frame) on invalid UTF-8 / unparseable data: payloads.

    fn ingest_text(&mut self, chunk: &[u8]) {
        self.line_carry.push_str(&String::from_utf8_lossy(chunk));
        // Keep the trailing partial line (no '\n' yet) in the carry.

inference-router/src/proxy.rs:299

• The error context string is now misleading: credential_for_upstream can return an Anthropic API key or None for Ollama, not just an auth token. Using “auth token” in the context makes failures harder to interpret when debugging non-Azure providers.

This issue also appears on line 524 of the same file.

    let credential = credential_for_upstream(auth, copilot, upstream)
        .await
        .context("Failed to acquire auth token")?;

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

controller/src/inference_policy.rs:166

• The doc comment claims modelPreference.primary.provider “takes precedence” over spec.provider, but the router implementation and docs state spec.provider is the only routing selector and modelPreference.*.provider stays informational. This comment is misleading and suggests behavior that does not exist.

    /// process. `modelPreference.primary.provider`, when it names a
    /// recognised tag, takes precedence over this field so a fallback
    /// chain can pin its own route. Mutually exclusive with

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (3)

controller/src/inference_policy.rs:167

• The doc comment says modelPreference.primary.provider takes precedence over spec.provider, but the router code explicitly ignores modelPreference.*.provider for routing (see inference-router/src/routes/mod.rs where apply_model_preference_override ignores primary.provider and provider routing is driven by policy.provider). This comment is misleading for CRD/API consumers and contradicts the fail-closed routing semantics described elsewhere in the PR.

    /// process. `modelPreference.primary.provider`, when it names a
    /// recognised tag, takes precedence over this field so a fallback
    /// chain can pin its own route. Mutually exclusive with

inference-router/src/proxy.rs:118

• The comment implies an inbound SDK-provided auth value is copied through, but build_upstream_headers explicitly strips inbound x-api-key (and other creds) before injecting provider auth. Clarifying that only anthropic-version is preserved (while x-api-key is always replaced) makes the security behavior unambiguous.

        // Anthropic's Messages API authenticates with `x-api-key` and
        // requires an `anthropic-version` header. The inbound SDK
        // value (when present) was already copied through above —
        // only the default is filled in here.

controller/src/reconciler/mod.rs:1964

• The router supports OPENAI_MODERATION_MODEL and GUARDRAIL_STREAM_SCAN_CHARS env overrides (see inference-router/src/config.rs and inference-router/src/guardrails.rs), but the controller only forwards moderation key/endpoint into the router sidecar env. In controller-managed sandboxes this makes those overrides effectively unusable, despite being documented/tunable in the router layer.

        // Multi-provider inference + guardrail backends. Router-only —
        // never on the agent container. Empty values are skipped so
        // Azure-only clusters keep an identical env surface (and
        // config-hash) to previous releases.
        for (name, value) in [
            ("ANTHROPIC_API_KEY", &ctx.anthropic_api_key),
            ("ANTHROPIC_ENDPOINT", &ctx.anthropic_endpoint),
            ("OLLAMA_ENDPOINT", &ctx.ollama_endpoint),
            ("OPENAI_MODERATION_API_KEY", &ctx.openai_moderation_api_key),
            (
                "OPENAI_MODERATION_ENDPOINT",
                &ctx.openai_moderation_endpoint,
            ),
        ] {

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (3)

inference-router/src/guardrails.rs:414

• scan_windows is documented as splitting by characters, but the early-return check uses text.len() (bytes). For non-ASCII text this can enter the slow path unnecessarily and makes the implementation diverge from the stated contract. Use a char-count check for consistency with the rest of the function.

fn scan_windows(text: &str) -> Vec<&str> {
    if text.len() <= MAX_SCAN_CHARS {
        return vec![text];
    }

inference-router/src/routes/chat_completions.rs:249

• The error code used for RouteGap::NonAzureProvider is provider_unimplemented, but the message indicates the provider is implemented and only this route lacks enforcement. Using a distinct code (e.g. provider_route_unsupported) avoids conflating “provider not implemented by this router build” (bedrock) with “implemented elsewhere but unsupported on this endpoint”.

        RouteGap::NonAzureProvider => (
            StatusCode::NOT_IMPLEMENTED,
            "provider_unimplemented",
            format!(

inference-router/src/config.rs:108

• The doc comment for openai_moderation_api_key says it “falls back to OPENAI_API_KEY, then the openai-moderation-api-key secret mount”, but the implementation checks the openai-moderation-api-key mount (via secret_from_env_or_mount) before falling back to OPENAI_API_KEY. Updating the comment avoids confusing operators about precedence.

    /// OpenAI Moderation key — `OPENAI_MODERATION_API_KEY` env (falls
    /// back to `OPENAI_API_KEY`, then the `openai-moderation-api-key`
    /// secret mount). `None` ⇒ `openai-moderation` stages fail closed.

[John Seong (sandole)]

thanks for the thorough security pass Pal Lakatos-Toth (@pallakatos). confirmed that this was a real bypass.

foundry_proxy now runs the same fail-closed guard_unenforced_route used by the sibling /v1/* routes as its first check for the inference-bearing path families (/agents*, /openai/responses*, /openai/conversations*, whole-segment match).

the taxonomy is identical to the existing guarded routes: 403 guardrail_route_unsupported when the policy declares guardrails, 501 provider_unimplemented when it selects a non-Azure provider, with the x-kars-decision* headers and route_enforcement_gap audit line attached.

ready for the functional/compatibility review whenever you are

[Pal Lakatos-Toth (pallakatos)]

Thanks for the fast follow-up. I re-reviewed feca96a7 against the original finding. The guard is now placed correctly and blocks the canonical /openai/responses*, /openai/conversations*, and /agents* paths, but the bypass remains exploitable because classification and forwarding use different path representations.

HIGH — raw-path guard bypass via dot-segment normalization

inference_bearing_foundry_route() classifies the raw uri.path(). foundry_proxy later concatenates that path into the upstream URL, and reqwest/Url::parse normalizes ./.. segments. An unguarded Foundry family can therefore pivot into a guarded inference path after the guard has already passed.

Verified examples:

/openai/files/../responses
/openai/files/%2e%2e/responses
/openai/vector_stores/../responses
/openai/evals/../responses
/memory_stores/../openai/responses
/evaluations/../openai/responses
/connections/../openai/responses

These route through an unguarded wildcard, then normalize upstream to /openai/responses; the same pattern reaches /openai/conversations. That preserves the original moderation bypass.

Please canonicalize once before classification and use that same canonical value for forwarding, or reject dot-segment / percent-encoded-dot paths at the start of foundry_proxy. A safer posture is to guard every Foundry proxy request whenever a guardrail/non-Azure policy is active and explicitly exempt a small canonical management set.

Please add real-router tests for literal .., %2e%2e, double-slash/trailing variants, and pivots from at least /openai/files, /memory_stores, and /evaluations. The current tests exercise canonical paths only and therefore miss this class.

Also, no build/test/Clippy/CodeQL workflow has run on the new head yet—only CLA is present. The PR remains blocked pending this fix and full CI.

[Pal Lakatos-Toth (pallakatos)]

Additional functional/CI review findings from independent execution and the now-approved GitHub workflows:

Independent local validation

• cargo fmt --all -- --check: pass
• local Clippy: pass
• cargo test --all: 2,186 passed, 0 failed, 3 ignored
• multi_provider_guardrails: 5 passed
• foundry_route_guard: 4 passed
• controller Helm drift tests: 18 passed
• Helm packaging/lint: pass

GitHub CI blockers

1. Rust CI fails on the current stable runner — inference-router/src/routes/chat_completions.rs:192 returns Result<(), axum::response::Response>, triggering Rust 1.98 clippy::result_large_err under -D warnings. Please use a boxed error response and update every call site, following the pattern now on main.

2. LOC gate fails:

   • controller/src/reconciler/mod.rs: 3,741 LOC, above the 3,700 cap
   • new inference-router/src/guardrails.rs: 1,487 LOC, above the 800-line Rust cap

   Please decompose these modules rather than adding a blanket LOC exemption. Guardrail parsing, provider adapters, streaming state, and tests are natural split points.

3. Required security audit is missing — capability-bearing router/controller paths require a tracked docs/security-audits/YYYY-MM-DD-<slug>.md with threat triage and two real sign-offs.

4. The branch is stale — current base is still cb2f687b; main has since moved through fix(ci): restore dependency security baseline #499 and chore(deps): refresh verified build dependencies #500. Please rebase before the next review so the current security/dependency baseline and Clippy fixes are included.

These are in addition to the unresolved raw-path/dot-segment guardrail bypass in the previous comment. The PR remains unapproved pending all four items and the continuing full functional logic review.

[Pal Lakatos-Toth (pallakatos)]

Full functional/compatibility review is now complete. The architecture and default compatibility path are generally sound: when spec.provider and spec.guardrails are absent, existing Azure/Foundry/Copilot/GitHub-Models behavior remains unchanged; provider credentials are stripped/replaced correctly; failover preserves provider context; CRD/CEL/Helm parity and protocol translations are mostly clean.

In addition to the previously posted traversal and CI blockers, these functional defects need resolution:

HIGH — buffered chat output skips guardrails when upstream body is not JSON

chat_completions.rs nests enforce_openai_output_guardrails(...) inside if let Ok(body_json) = serde_json::from_slice(...). An unparseable/truncated/NDJSON upstream body exits that block and is returned verbatim, despite scan_text_or_raw explicitly providing a raw-text fallback. This contradicts the changelog claim that unparseable bodies are scanned. Hoist output enforcement outside the JSON parse block and add a non-JSON upstream regression test.

MEDIUM — Anthropic streaming never records token usage

The buffered Anthropic passthrough records usage, but the stream: true branch never parses message_start.usage.input_tokens / message_delta.usage.output_tokens and never calls budget.record_usage. Streaming therefore bypasses daily/monthly token accumulation for the new first-class Anthropic provider. Wrap the stream and record both counters at completion.

MEDIUM — SSE moderation text is corrupted when UTF-8 code points cross chunk boundaries

SseGuardState::ingest_text applies String::from_utf8_lossy independently to each raw HTTP chunk before carrying incomplete lines. A CJK/emoji/accented character split mid-codepoint becomes replacement characters in the text sent to moderation, while the client receives the original bytes. Buffer incomplete UTF-8 bytes across chunks and add non-ASCII split tests.

MEDIUM — malformed compiled guardrail config silently disables enforcement

GuardrailStageCfg::from_compiled_json returns an empty vector when guardrails is not an array and silently drops entries missing a string provider. Downstream treats empty as “no guardrails”, which both skips scanning and disables the route-gap guard. Return a configuration error when the key is present but malformed; declared-but-unbuildable controls must fail closed.

LOW — streaming scans only choices[0]

Buffered extraction scans every choice, but streaming delta_text_from_event reads only the first. OpenAI-compatible upstreams can emit multiple choices in one frame, allowing later choices through unscanned. Iterate all choices.

LOW — policy error response contracts diverge by route

The same provider/guardrail condition produces different error.type/missing error.code shapes across chat, sibling OpenAI routes, and Anthropic Messages. Status codes are correct, but clients cannot reliably switch on the error code introduced by this PR. Please use a shared code-carrying error helper.

The independent execution evidence remains strong (2,186 tests plus targeted provider/guardrail/Helm suites passed locally), but remote CI is red and these uncovered paths are not represented in the current tests. The PR remains unapproved pending the security bypass, CI/LOC/audit fixes, and the HIGH/MEDIUM items above.

[John Seong (sandole)]

thanks for the second pass Pal Lakatos-Toth (@pallakatos). All items addressed

Traversal bypass (HIGH): foundry_proxy now percent-decodes each path segment and rejects (400 invalid_path) any dot/empty/encoded-slash segment before classification or forwarding, so a path can't normalize into a different route than the one classified. Guard is now default-deny (explicit exempt set for management/storage families). Added your full matrix plus a reqwest::Url normalization premise test.

CI blockers: result_large_err boxed (clippy -D warnings clean); guardrails.rs split into guardrails/{mod,backend,stream,tests}.rs and reconciler/pod_spec.rs extracted, all under the caps; security-audit doc added. Rebase not done yet: the result_large_err fix landed on main, so I'll rebase carefully as a separate step to handle the chat_completions.rs conflict.

Functional defects: non-JSON buffered output now scanned (enforcement hoisted out of the JSON-parse block); Anthropic streaming records usage; SSE UTF-8 reconstructed across chunk boundaries; malformed guardrail config fails closed (poisons the policy instead of degrading to "no guardrails"); streaming scans all choices[]; denials carry a stable error.code.

Local: cargo test --workspace 2201 passed, clippy/fmt clean, all ci/*.sh gates pass. Also smoke-tested against real Ollama and real Anthropic: buffered + streaming both return real completions,1and an agent-supplied x-api-key is confirmed stripped and replaced (real 200, not 401).

[Pal Lakatos-Toth (pallakatos)]

Security re-review of 708b2401 confirms the technical security blockers are fixed: Foundry path traversal/dot-segment pivots are rejected before classification/forwarding, the Foundry proxy guard is default-deny, malformed guardrail configuration fails closed, prior route/provider/credential findings remain clean, LOC and Clippy blockers are resolved, and the remediation tests are genuinely adversarial.

One process/security-control blocker remains in docs/security-audits/2026-08-25-multi-provider-guardrails.md: commit 708b2401 was authored by the contributor but includes Signed-off-by: Pal Lakatos-Toth <pallakatos@microsoft.com>. I had not approved or authored that sign-off. The CI gate currently validates only distinct email strings, so this self-attests the required independent review.

Please remove the maintainer sign-off you added and leave only your own author sign-off. After the independent functional re-review and final CI are complete, I will add/authorize the independent reviewer sign-off through a maintainer-owned commit or explicit approval. Please also correct the audit statement claiming the existing blocklist/egress guard applies to Anthropic/Ollama provider calls; those routes do not currently invoke is_blocked, although their destinations remain operator-controlled.

[Pal Lakatos-Toth (pallakatos)]

Functional re-review of 708b2401 confirms the prior major fixes are real: non-JSON buffered output is scanned, split UTF-8 is preserved, malformed guardrails fail closed, every streaming choice is scanned, Rust/LOC decomposition is clean, and the default Azure path remains unchanged.

Three remaining functional/compatibility items need resolution before approval:

MEDIUM — Anthropic streaming usage is lost when a guardrail cuts mid-stream

tap_anthropic_stream_usage records only when its inner stream returns Err or None. It is wrapped inside guard_sse_stream; on SseGuardStep::Cut, the guard sets finished and drops the inner tap without polling it to completion. The claimed “tap before guarding” does not record input or partial output tokens on a mid-stream violation/backend outage, so budget counters remain zero despite consumed upstream tokens.

Please record from the guard terminal path or share usage state with an outer wrapper, and add a test that drains the guard through a mid-stream cut and asserts exactly-once input/output accounting.

MEDIUM — Anthropic buffered error codes are still inconsistent

The same guardrail outcome differs by route/transport:

• chat: guardrail_blocked / guardrail_misconfigured / guardrail_unavailable
• Anthropic buffered: content_policy_violation or generic api_error
• Anthropic streaming: the specific guardrail codes

This defeats the stated stable error.code contract and collapses misconfiguration vs backend outage. Preserve the Anthropic-native error.type, but put v.code() / e.code() / provider-specific code in error.code, with tests across buffered and streaming paths.

COMPATIBILITY — trailing/double-slash paths now fail even with no policy

decoded_path_segments unconditionally rejects empty segments, so historically proxied management paths such as POST /openai/files/ or /memory_stores/ now return 400 even when no InferencePolicy/guardrail is configured. We require non-breaking behavior for live customers. Normalize benign trailing/double slashes to their canonical form while still rejecting ., .., encoded slash/backslash, malformed escapes, and traversal pivots; add no-policy compatibility tests.

Lower-severity cleanup: one chat denial (provider: anthropic on the OpenAI-shaped route) still bypasses the coded-error, decision-header, and audit-line helpers; the Anthropic deny helper also lacks the decision headers claimed by the changelog. Please align these while fixing the error contract.

The contributor-authored maintainer sign-off remains a separate blocker from the previous comment. All current code/tests/gates are otherwise clean in independent execution.

[Pal Lakatos-Toth (pallakatos)]

Final functional re-review found one remaining blocker in the actual native Anthropic buffered path, plus documentation/message corrections. The second audit sign-off cannot be added yet.

HIGH — buffered native Anthropic passthrough still uses legacy error codes and no decision headers

inference-router/src/routes/anthropic_messages.rs still calls deny_response in two output-guardrail branches inside forward_anthropic_passthrough:

• around line 905: violation returns content_policy_violation, with no x-kars-decision* headers
• around line 921: pipeline error returns generic api_error with a hard-coded 502, no decision headers, and collapses guardrail_misconfigured vs guardrail_unavailable

The translated path and streaming native path now emit the stable specific codes, so the identical request differs solely by stream: false vs true: buffered returns legacy/generic codes while streaming returns guardrail_blocked / guardrail_misconfigured / guardrail_unavailable. This directly violates the new stable error.code contract and loses policy attribution.

Please route both branches through deny_policy: use v.code() for violations and guardrail_error_status(&e) + e.code() for errors, matching the translated path. Add route-level tests that exercise the real buffered native Anthropic passthrough, not deny_policy directly.

Documentation and runtime message mismatch

decoded_path_segments now collapses benign empty segments, but the security audit, CRD reference, audit log, and client-facing 400 text still claim empty segments are rejected. Please update all four to state that empty segments are canonicalized, while dot segments, encoded dot/slash/backslash forms, and malformed escapes are rejected.

Everything else in this delta is verified clean: mid-stream usage records once through the composed observer→guard→finalizer path; traversal remains closed; chat Anthropic denial aligns; supply-chain review is clean; 2,206 tests and technical CI pass. The audit sign-off remains pending these final corrections.

[Pal Lakatos-Toth (pallakatos)]

Approved after final independent review of f3200672.

Supply chain and provenance: reviewed all 17 contributor commits as hostile input. The contribution adds no dependency/lockfile, workflow, build/release, Dockerfile, vendored, binary, generated-artifact, custom-crypto, or unsafe-code surface. Network destinations and provider credentials remain router/operator controlled; no request-controlled SSRF or credential-forwarding path was found. Contributor commits and authorship remain intact; current main was merged without rewriting history.

Security and compatibility: all previously reported route-coverage, path-normalization, malformed-policy, non-JSON output, streaming UTF-8/multi-choice, Anthropic accounting, stable-error, decision-header, and audit-control issues are fixed and exercised through real route-level regressions. Default Azure behavior with no provider/guardrail policy remains unchanged, while vulnerable bypass paths fail closed.

Independent execution: formatting and Clippy pass; 18 focused route tests pass; the full Rust suite passes with 2,208 tests and zero failures; all seven technical gates pass. The full remote matrix is green, including Rust/CLI/runtime/mesh tests, E2E, chaos, benchmarks, CodeQL analyses, dependency review, RustSec, cargo-deny, Trivy, secret scanning, image scanning, and the independent security-audit gate.

No high-confidence security, supply-chain, functional, or live-customer compatibility blockers remain.