From a2c2c590a2de441a5f79bd2e2174db1ca799aee8 Mon Sep 17 00:00:00 2001 From: rkoster Date: Mon, 24 Aug 2026 15:04:03 +0200 Subject: [PATCH 1/3] Add research note: Heroku AI platform --- research/heroku-ai-platform.md | 91 ++++++++++++++++++++++++++++++++++ 1 file changed, 91 insertions(+) create mode 100644 research/heroku-ai-platform.md diff --git a/research/heroku-ai-platform.md b/research/heroku-ai-platform.md new file mode 100644 index 0000000..d92abce --- /dev/null +++ b/research/heroku-ai-platform.md @@ -0,0 +1,91 @@ +--- +title: "Heroku AI — Managed Inference, Agents, and MCP on a Buildpack-Era PaaS" +author: Ruben Koster (@rkoster) +date: 2026-08-24 +tags: [runtime-lifecycle, inter-agent-comms, ecosystem-survey, observability-governance, identity] +cf_areas: [buildpacks, capi] +status: draft +sources: + - https://www.heroku.com/ai/ + - https://www.heroku.com/blog/heroku-ai-studio-workspace-for-smarter-faster-ai-apps/ + - https://devcenter.heroku.com/categories/ai-integrations + - https://www.heroku.com/blog/accelerating-ai-dev-new-models-performance-improvements-messages-api/ +--- + +## Summary + +Heroku — the platform whose buildpack concept Cloud Foundry adopted and co-evolved with — +is repositioning itself as an "AI PaaS." It layers a managed inference/agents add-on, an +MCP tool-gateway, pgvector-backed vector search, and an interactive prompt-tuning workspace +(AI Studio) on top of its existing dyno/add-on model, rather than introducing a new agent +runtime primitive. Apps keep their normal dyno lifecycle; AI capability is delivered as +attached add-ons and API endpoints (OpenAI- and Anthropic-compatible) that any language +buildpack can consume via environment-variable config. + +## Key findings + +- **AI-as-add-on, not new compute primitive**: "Heroku Managed Inference and Agents" is + delivered as a standard add-on (`heroku-inference`), attached like Postgres or Redis. It + injects `INFERENCE_URL` / `INFERENCE_KEY` / `INFERENCE_MODEL` config vars into the dyno's + environment — the same attach-and-inject pattern used for every other Heroku add-on. No + new dyno type or process model was introduced for agent workloads. +- **API compatibility over new protocols**: Inference is exposed via an OpenAI-compatible + Chat Completions surface plus a newer Anthropic-compatible `v1/messages` endpoint + (currently in preview, Anthropic models only). This lets existing SDKs (OpenAI, Anthropic, + LangChain, LlamaIndex, Vercel AI SDK, Pydantic AI, LiteLLM) "just work" against + Heroku-hosted models with a base-URL swap — echoed directly by a customer quote in the + marketing page ("it uses the OpenAI APIs, so all the existing SDKs just work"). +- **MCP as the tool/agent-interop layer**: "MCP Toolkits" provide a managed gateway to run + and expose multiple MCP servers, giving both Heroku-hosted agents and external clients + (Claude Desktop, Cursor) one consistent interface to call internal or external tools. + This is Heroku's answer to inter-agent/tool-comms standardization — comparable in spirit + to A2A adoption elsewhere (see `a2a-protocol.md`). +- **Vector search reuses the existing data service**: RAG/embeddings use `pgvector` on + Heroku Postgres rather than a new managed vector DB product — extending an existing + data-service instead of adding infrastructure surface area. +- **AI Studio = interactive workspace bolted onto the add-on**: A hosted UI + (`aistudio.heroku.com`) for live prompt/tool iteration against provisioned models, + reachable directly from the add-on. It's dev-tooling, not a runtime change — no isolated + execution sandbox described, unlike Azure Foundry's per-session VM model + (`azure-hosted-agents.md`). +- **Fast-moving model catalog with lifecycle policy**: Regular catalog churn (Claude Opus + 4.5 / Sonnet 4.5 / Haiku 4.5, Amazon Nova 2, Kimi K2 Thinking, MiniMax M2, Qwen3) with + published deprecation dates (e.g., Claude 3 family retiring Jan 30 2026) and a + stated goal of automatic fallback to a successor model on deprecation — an operational + concern platforms will need to own as "model EOL" becomes analogous to stack/runtime EOL. +- **Data governance framed explicitly**: Open-weight models are stated to run on + Heroku/AWS-controlled compute with neither Heroku nor the model provider retaining access + to customer data or using it for training — a claim made prominently in the same breath as + performance features, suggesting this is a purchase-decision driver for enterprise AI + workloads. +- **Prompt caching as a default optimization**: System prompts and tool/function + definitions are cached automatically to cut repeated-request latency (opt-out via an HTTP + header); user messages/history are explicitly excluded from caching for privacy. This is a + concrete example of a platform absorbing an agent-specific performance concern (repeated + large tool-schema payloads) rather than leaving it to app code. + +## CF relevance + +Heroku's approach validates a "no new primitive" path: treat AI inference/agents/tools as +attachable services behind standard interfaces, and let the existing buildpack/app/dyno +model carry the workload unchanged. For Cloud Foundry — which inherited the buildpack +concept from this same lineage — this suggests an achievable near-term model: an +AI-service-broker (OSBAPI) pattern exposing OpenAI/Anthropic-compatible endpoints and an +MCP gateway, with credentials injected via `VCAP_SERVICES`, rather than inventing a new +agent-runtime abstraction. It also surfaces adjacent platform concerns CF doesn't yet +address: model lifecycle/deprecation policy as a first-class platform signal, and +prompt/tool-schema caching as a routing-layer optimization for agentic traffic patterns. + +## Open questions + +- Would an OSBAPI service broker exposing an OpenAI/Anthropic-compatible endpoint plus + MCP-gateway wiring be sufficient for CF to reach functional parity with Heroku's AI + add-on, or does agent orchestration need a runtime-level primitive after all? +- How should a platform surface upstream model deprecation/EOL to app owners — is this + analogous to stack deprecation notices, and could `cf` tooling reuse that mechanism? +- Is prompt/tool-definition caching something the platform's routing tier (Gorouter) could + offer generically, or does it inherently belong to the inference provider? +- Heroku frames "no training on customer data" as a headline trust claim — should CF define + a similar baseline expectation for any AI-service broker admitted into the marketplace? +- Does an MCP gateway belong at the platform layer (like Heroku's MCP Toolkits) or is it + better left as an ordinary buildpack-deployed app, given CF's app-centric model? From 319886704c96ed6e25c52c54904fe2e7dd45ea0a Mon Sep 17 00:00:00 2001 From: rkoster Date: Mon, 24 Aug 2026 15:12:57 +0200 Subject: [PATCH 2/3] research/heroku-ai-platform: add sandboxing and durable-execution findings --- research/heroku-ai-platform.md | 48 ++++++++++++++++++++++++++++++---- 1 file changed, 43 insertions(+), 5 deletions(-) diff --git a/research/heroku-ai-platform.md b/research/heroku-ai-platform.md index d92abce..8a2d372 100644 --- a/research/heroku-ai-platform.md +++ b/research/heroku-ai-platform.md @@ -2,14 +2,16 @@ title: "Heroku AI — Managed Inference, Agents, and MCP on a Buildpack-Era PaaS" author: Ruben Koster (@rkoster) date: 2026-08-24 -tags: [runtime-lifecycle, inter-agent-comms, ecosystem-survey, observability-governance, identity] -cf_areas: [buildpacks, capi] +tags: [runtime-lifecycle, sandboxing-isolation, inter-agent-comms, ecosystem-survey, observability-governance, identity] +cf_areas: [buildpacks, capi, diego] status: draft sources: - https://www.heroku.com/ai/ - https://www.heroku.com/blog/heroku-ai-studio-workspace-for-smarter-faster-ai-apps/ - https://devcenter.heroku.com/categories/ai-integrations - https://www.heroku.com/blog/accelerating-ai-dev-new-models-performance-improvements-messages-api/ + - https://www.heroku.com/blog/code-execution-sandbox-for-agents-on-heroku/ + - https://www.heroku.com/ai/mcp-on-heroku/ --- ## Summary @@ -45,9 +47,28 @@ buildpack can consume via environment-variable config. data-service instead of adding infrastructure surface area. - **AI Studio = interactive workspace bolted onto the add-on**: A hosted UI (`aistudio.heroku.com`) for live prompt/tool iteration against provisioned models, - reachable directly from the add-on. It's dev-tooling, not a runtime change — no isolated - execution sandbox described, unlike Azure Foundry's per-session VM model - (`azure-hosted-agents.md`). + reachable directly from the add-on. It's dev-tooling, not a runtime change. +- **Sandboxing reuses an existing primitive — one-off dynos**: Heroku ships a "Code + Execution Sandbox for Agents" (launched alongside the inference add-on, May 2025) that + runs untrusted LLM-generated code inside **one-off dynos** — the decade-old mechanism + behind `heroku run` — spun up on demand and terminated after use. No new sandbox runtime + was built; isolation is achieved by reusing the ephemeral-container lifecycle Heroku + already had, so blast radius is bounded to one throwaway container per invocation. + Exposed two ways: as built-in tools (`code_exec_python/ruby/node/go`) in the Managed + Inference and Agents API (`v1/agents/heroku`), and as open-source MCP servers + (e.g. `heroku/mcp-code-exec-python`) any MCP client can attach to. This implements + Anthropic's "programmatic tool calling" pattern — the model writes code that loops/filters + server-side and only a summary re-enters the model's context, which Heroku cites as + cutting token consumption ~37% on average (up to 98% in some cases). A `max_calls` + runtime param bounds sandbox invocations per agent loop. +- **No durable-execution primitive found**: nothing in the surveyed material describes + workflow-style checkpointing, replay, or resumable agent state across steps/restarts (the + kind of thing Temporal/Restate provide). Sandboxes are stateless and ephemeral — state + dies with the one-off dyno — and inference calls are plain stateless request/response. + Any multi-step agent loop that must survive beyond a single request/dyno is left to the + app developer to persist itself (e.g., in Postgres). This is a notable contrast with + Azure Foundry's per-session VM model, which offers a persistent filesystem and 30-day + resumable sessions (`azure-hosted-agents.md`). - **Fast-moving model catalog with lifecycle policy**: Regular catalog churn (Claude Opus 4.5 / Sonnet 4.5 / Haiku 4.5, Amazon Nova 2, Kimi K2 Thinking, MiniMax M2, Qwen3) with published deprecation dates (e.g., Claude 3 family retiring Jan 30 2026) and a @@ -76,6 +97,15 @@ agent-runtime abstraction. It also surfaces adjacent platform concerns CF doesn' address: model lifecycle/deprecation policy as a first-class platform signal, and prompt/tool-schema caching as a routing-layer optimization for agentic traffic patterns. +The sandbox design is directly portable: Heroku's "reuse an existing ephemeral-compute +primitive for untrusted code execution" maps almost one-to-one onto CF's task model — +Diego one-off tasks are CF's equivalent of one-off dynos, and a "code execution sandbox" +add-on could plausibly be built as a broker that runs agent-submitted code as a short-lived +Diego task rather than requiring a new isolation layer. Conversely, the absence of any +durable-execution story on Heroku means CF gets no free lunch there either — a resumable, +checkpointed agent-loop primitive (if the working group decides it's needed) would be new +ground for both platforms, not something to crib from Heroku's design. + ## Open questions - Would an OSBAPI service broker exposing an OpenAI/Anthropic-compatible endpoint plus @@ -89,3 +119,11 @@ prompt/tool-schema caching as a routing-layer optimization for agentic traffic p a similar baseline expectation for any AI-service broker admitted into the marketplace? - Does an MCP gateway belong at the platform layer (like Heroku's MCP Toolkits) or is it better left as an ordinary buildpack-deployed app, given CF's app-centric model? +- Could a "code execution sandbox for agents" be implemented on CF today as a broker that + dispatches agent-submitted code to a short-lived Diego task, mirroring Heroku's + one-off-dyno approach — or does untrusted-code execution need stronger isolation + (gVisor/Kata, a dedicated runc profile) than a standard Diego cell provides? +- Neither Heroku nor (per the earlier Azure/other notes) most surveyed platforms offer true + durable execution for agent loops. Is this a genuine gap the working group should design + for, or is "persist state yourself in a bound service" (Heroku's implicit answer) good + enough for CF's Phase 1 scope? From b959091566418fa068a99e003cc3908ec2d23cbf Mon Sep 17 00:00:00 2001 From: rkoster Date: Mon, 24 Aug 2026 16:19:21 +0200 Subject: [PATCH 3/3] research/heroku-ai-platform: add sandbox dependency-resolution finding --- research/heroku-ai-platform.md | 35 ++++++++++++++++++++++++++++++---- 1 file changed, 31 insertions(+), 4 deletions(-) diff --git a/research/heroku-ai-platform.md b/research/heroku-ai-platform.md index 8a2d372..268a0e1 100644 --- a/research/heroku-ai-platform.md +++ b/research/heroku-ai-platform.md @@ -12,6 +12,7 @@ sources: - https://www.heroku.com/blog/accelerating-ai-dev-new-models-performance-improvements-messages-api/ - https://www.heroku.com/blog/code-execution-sandbox-for-agents-on-heroku/ - https://www.heroku.com/ai/mcp-on-heroku/ + - https://github.com/heroku/mcp-code-exec-python --- ## Summary @@ -61,6 +62,19 @@ buildpack can consume via environment-variable config. server-side and only a summary re-enters the model's context, which Heroku cites as cutting token consumption ~37% on average (up to 98% in some cases). A `max_calls` runtime param bounds sandbox invocations per agent loop. +- **Sandbox dependency resolution happens live, per call, with no offline path**: reading + the open-source `heroku/mcp-code-exec-python` implementation shows two separate layers. + The Python *interpreter* version is fixed at deploy time by the buildpack (a + `.python-version` file pins it into the slug every one-off dyno boots from). Requested + *libraries*, however, are resolved at request time: the `code_exec_python` tool accepts + an optional `packages` list, and on each call the server creates a throwaway venv inside + the dyno, runs `pip install ` against the public PyPI index, executes the code, + then deletes the venv. The implementation's own docstring notes this "does NOT mean the + code is fully isolated or secure — it just means the package installations are isolated"; + the dyno container boundary is what actually isolates execution. This design requires + live egress to a public package index at execution time, with no documented offline + buildpack/mirror path and no caching or version pinning across calls — every invocation + re-resolves and re-downloads from scratch. - **No durable-execution primitive found**: nothing in the surveyed material describes workflow-style checkpointing, replay, or resumable agent state across steps/restarts (the kind of thing Temporal/Restate provide). Sandboxes are stateless and ephemeral — state @@ -101,10 +115,17 @@ The sandbox design is directly portable: Heroku's "reuse an existing ephemeral-c primitive for untrusted code execution" maps almost one-to-one onto CF's task model — Diego one-off tasks are CF's equivalent of one-off dynos, and a "code execution sandbox" add-on could plausibly be built as a broker that runs agent-submitted code as a short-lived -Diego task rather than requiring a new isolation layer. Conversely, the absence of any -durable-execution story on Heroku means CF gets no free lunch there either — a resumable, -checkpointed agent-loop primitive (if the working group decides it's needed) would be new -ground for both platforms, not something to crib from Heroku's design. +Diego task rather than requiring a new isolation layer. However, Heroku's dependency +resolution (live `pip install` per call against a public package index) assumes +always-on internet egress from the sandbox — a pattern that doesn't fit enterprise CF +deployments that typically run offline/mirrored buildpacks with no direct app egress. CF's +existing staging/droplet split (resolve dependencies once at `cf push` time, run app +instances from the resulting immutable droplet) is a different model that might be a +better fit for offline environments, but would need its own design (see +`ideas/staged-sandbox-environments.md`). Separately, the absence of any durable-execution +story on Heroku means CF gets no free lunch there either — a resumable, checkpointed +agent-loop primitive (if the working group decides it's needed) would be new ground for +both platforms, not something to crib from Heroku's design. ## Open questions @@ -123,6 +144,12 @@ ground for both platforms, not something to crib from Heroku's design. dispatches agent-submitted code to a short-lived Diego task, mirroring Heroku's one-off-dyno approach — or does untrusted-code execution need stronger isolation (gVisor/Kata, a dedicated runc profile) than a standard Diego cell provides? +- Heroku resolves sandbox dependencies live per call against public PyPI, with no offline + or caching path. Could CF instead reuse its existing staging/droplet mechanism — resolve + a dependency manifest once into a cacheable droplet, then execute against an + already-staged environment — to support offline/mirrored-buildpack deployments and avoid + re-resolving dependencies on every sandbox invocation? (see + `ideas/staged-sandbox-environments.md`) - Neither Heroku nor (per the earlier Azure/other notes) most surveyed platforms offer true durable execution for agent loops. Is this a genuine gap the working group should design for, or is "persist state yourself in a bound service" (Heroku's implicit answer) good