Add draft endpoint prototyping loop

[peterschmidt85]

What This PR Does

Adds a draft endpoint configuration type for model inference endpoints and an agent-driven prototyping loop.

This is a draft PR for reviewing endpoint UX and the efficiency of the prototyping loop. The code is intentionally still subject to review and rewrite; the goal now is to validate whether the workflow, CLI surface, agent behavior, logs, model variant selection, and preset reuse model feel right before polishing internals.

Main pieces:

• Adds type: endpoint with model, preset_policy, and normal profile constraints such as fleets, backends, spot_policy, and max_price.
• Supports exact model repo requests and base-model requests where the agent may choose a compatible model variant.
• Stores endpoint model identity as model_base and model_repo so clients can call the base/API model while the service may load a selected repo/path.
• Adds endpoint CLI/API/server lifecycle: create, list, get, stop, logs, and endpoint presets.
• Adds project-local endpoint presets: base-level recipes that can be reused without invoking the agent.
• Adds a Claude Code based endpoint agent for preset_policy: create / reuse-or-create when no usable preset is available.
• Adds endpoint progress logs so users can follow the agent’s decisions while it prototypes and deploys.
• Adds tests for endpoint model parsing, planning, lifecycle, CLI output, presets, agent report handling, and restart/session behavior.

Preset mechanics and model variant selection are implemented and validated on initial e2e scenarios, but recipe quality, placement semantics, and optimization behavior still need broader testing.

How To Use

Start the server with the endpoint agent enabled:

export DSTACK_AGENT_ANTHROPIC_API_KEY=...
uv run dstack server --port 3000

For local development only, the server can use the existing Claude CLI login instead of DSTACK_AGENT_ANTHROPIC_API_KEY:

export DSTACK_AGENT_CLAUDE_USE_EXISTING_AUTH=1
uv run dstack server --port 3000

Production servers must use DSTACK_AGENT_ANTHROPIC_API_KEY. Do not set DSTACK_AGENT_CLAUDE_USE_EXISTING_AUTH together with DSTACK_AGENT_ANTHROPIC_API_KEY; the endpoint agent treats that as a configuration error.

Optionally pass Claude CLI effort for endpoint agent sessions:

export DSTACK_AGENT_CLAUDE_EFFORT=high

Supported values are low, medium, high, xhigh, and max. If unset, dstack does not pass --effort and Claude CLI uses its default.

Create or select a project with at least one active fleet. Endpoint creation does not create fleets.

Exact model repo:

type: endpoint
name: qwen25

model: Qwen/Qwen2.5-0.5B-Instruct
preset_policy: create

fleets:
  - endpoint-gpu

Exact model repo with a client-facing model name:

type: endpoint
name: qwen27-gptq

model:
  repo: Qwen/Qwen3.5-27B-GPTQ-Int4
  name: Qwen/Qwen3.5-27B
preset_policy: create

fleets:
  - endpoint-gpu

Base model request, where the agent may choose a compatible repo/path variant:

type: endpoint
name: qwen27-cheap-48gb

model:
  base: Qwen/Qwen3.5-27B
preset_policy: create

fleets:
  - qwen27-cheap-48gb

spot_policy: auto
max_price: 0.65

Apply it:

uv run dstack apply --project <project> -f endpoint.dstack.yml

Watch endpoint progress:

uv run dstack endpoint logs --project <project> qwen27-cheap-48gb -w

Inspect the endpoint:

uv run dstack endpoint get --project <project> qwen27-cheap-48gb --json

List learned presets:

uv run dstack endpoint preset --project <project>
uv run dstack endpoint preset get --project <project> Qwen/Qwen3.5-27B --json

Reuse a learned preset without running the agent:

type: endpoint
name: qwen27-reuse

model:
  base: Qwen/Qwen3.5-27B
preset_policy: reuse

fleets:
  - qwen27-cheap-48gb

Stop the endpoint:

uv run dstack endpoint stop --project <project> qwen27-cheap-48gb -y

Example

Representative e2e: deploy Qwen/Qwen3.5-27B on cheaper 48GB hardware instead of using A100-class capacity.

Fleet used for the test:

type: fleet
name: qwen27-cheap-48gb
nodes: 0..2
backends: [runpod, verda]
spot_policy: auto
max_price: 0.65
idle_duration: 600
resources:
  gpu: A40,A6000:48GB
  disk: 200GB

Endpoint config:

type: endpoint
name: qwen27-cheap-48gb-e2e

model:
  base: Qwen/Qwen3.5-27B
preset_policy: create

fleets:
  - qwen27-cheap-48gb

env:
  - HF_TOKEN

spot_policy: auto
max_price: 0.65

Observed endpoint progress:

Backend/model choice: RunPod A40 48GB was available under max_price. The base bf16 model was too large for 48GB, so the agent selected Qwen/Qwen3.5-27B-GPTQ-Int4 as the deployed repo/path while keeping Qwen/Qwen3.5-27B as the service model name.
Task verification: initial 32768 context attempt OOMed during vLLM profiling. The agent retried with max_model_len=8192, max_num_seqs=8, enforce_eager, and gpu_memory_utilization=0.95.
Service verification: /v1/chat/completions returned HTTP 200 through the dstack service URL with model Qwen/Qwen3.5-27B.
Preset saved: base=Qwen/Qwen3.5-27B, recipe model=Qwen/Qwen3.5-27B-GPTQ-Int4.

The same saved preset was then reused with preset_policy: reuse and deployed directly without running the agent/prototyping task again.

Backlog for v0 (WIP)

• Broader e2e coverage across model sizes, backends, and fleet shapes. Qwen 27B base-to-GPTQ variant selection on 48GB hardware passed, but it is still deployment validation, not benchmarked optimization.
• Fix the agent prompt split between # Final Service and # Final Report. # Final Service must describe only the actual service YAML that the agent submits and verifies. It must not tell the agent to loosen resources before verification. Any broader resource envelope inferred from the tested run belongs to final_report.json / preset-building logic, not to the service YAML that is being tested.
• Define preset reuse placement semantics. Current reuse can deploy the saved service recipe on a different backend/GPU if the recipe resource envelope allows it. We need to decide how strict reuse should be relative to validation evidence.
• Make endpoint JSON/config output secret-safe. Endpoint configs can include secret env references such as HF_TOKEN; API/CLI JSON must not expose resolved secret values.
• Add benchmark and trial artifacts for real optimization work: throughput, TTFT, ITL, request latency, error rate, context length, concurrency/request rate, backend/GPU, and price.
• Review reasoning-model response UX. The Qwen3 reasoning setup can return useful text in message.reasoning with message.content: null; this may be surprising for a generic model inference endpoint.
• More testing for preset quality, recipe ordering, and reuse under capacity churn.
• Stronger crash/restart reconciliation tests around final report handoff, submitted runs, and multi-server locking.
• More validation of task-first prototyping across VM, SSH, Kubernetes, and container backends.
• Raw Claude traces in debug mode, without mixing them into normal endpoint progress logs.
• Docker/server packaging validation for the bundled agent runtime.
• Public docs and UX polish after the draft behavior stabilizes.

Out Of Scope For V1

• Endpoint-level resources overrides.
• Agent budget/cost governance.
• Recipe ranking or explicit recipe selection.
• Frontend UI.
• Autoscaling optimization, benchmarking, and metrics collection.
• Fixing drivers, generating custom kernels, or doing deep metrics-driven optimization.
• P/D disaggregation and distributed serving.
• Using host CUDA/driver versions for offer matching, because offers/fleets do not expose that today.
• Letting the agent target specific fleet instances for follow-up runs when it intentionally wants to reuse a warmed instance or instance volumes.

Notes On Implementation

• The endpoint agent currently invokes the Claude CLI instead of the Claude Python SDK because the SDK depends on Pydantic v2 while dstack still uses Pydantic v1.
• The production path uses DSTACK_AGENT_ANTHROPIC_API_KEY and keeps Claude in --bare mode.
• DSTACK_AGENT_CLAUDE_USE_EXISTING_AUTH=1 is a local-development-only fallback. It is mutually exclusive with DSTACK_AGENT_ANTHROPIC_API_KEY; setting both is an endpoint agent configuration error.
• DSTACK_AGENT_CLAUDE_EFFORT passes Claude CLI --effort for endpoint agent sessions. If unset, Claude CLI uses its default.
• When no API key is set and existing-auth mode is enabled, Claude runs without --bare so it can read the server user's existing Claude CLI auth and settings; the server logs a warning when this mode is used.
• In existing-auth mode, Claude runs with the server user's real HOME, and the agent installs workspace-local dstack and ssh wrappers so dstack CLI config and SSH aliases still use the endpoint workspace home.
• Agent process abort handling uses POSIX process groups when available and falls back to signaling the stored process id on platforms without os.killpg; this keeps the Windows pyright matrix clean without changing POSIX behavior.
• model: <string> is parsed as an exact repo/path request. model: {repo, name} deploys repo but exposes name to clients. model: {base} lets the agent choose a compatible deployed repo/path and stores that selected repo/path in the saved preset recipe.
• E2E-tested paths include exact/base Qwen2.5 0.5B deployment, Qwen3.5 27B base-model variant selection on 48GB hardware, preset reuse for both small and 27B presets, existing Claude CLI auth, and DSTACK_AGENT_ANTHROPIC_API_KEY.

AI Assistance

This PR was developed with AI assistance from Codex.