llama.cpp Studio

llama.cpp Studio is a local control plane for downloading, configuring, and serving LLMs from a single machine.

The project combines:

• a Vue 3 frontend
• a FastAPI backend
• YAML-backed state under data/
• a unified llama-swap OpenAI-compatible endpoint on port 2000

Today, the app manages five runtime families:

• llama.cpp for GGUF models
• ik_llama.cpp for GGUF models
• LMDeploy for safetensors models
• 1Cat-vLLM for vLLM-backed models
• audio.cpp for prepared audio model bundles (TTS, ASR, VAD, and related tasks)

This README has been rebuilt to match the current repository layout and runtime behavior.

What the app does

• Search Hugging Face and the audio.cpp package catalog for compatible models
• Download and refresh GGUF quantizations, optional mmproj / MTP / DFlash companion files, safetensors bundles, and prepared audio.cpp packages
• Store model and engine state in YAML instead of SQLite
• Build llama.cpp, ik_llama.cpp, and audio.cpp from source and manage multiple installed versions
• Install LMDeploy and 1Cat-vLLM from releases or source into dedicated virtual environments
• Install CUDA Toolkit versions into the persistent app data directory
• Configure models per engine using a parameter catalog parsed from the active runtime binary
• Serve models through one OpenAI-compatible endpoint exposed by llama-swap
• Stream progress and notifications over Server-Sent Events

Ports and endpoints

Purpose	Docker / container	Local dev
Web UI + FastAPI API	http://localhost:8080	frontend: http://localhost:5173, backend API: http://localhost:8081
OpenAI-compatible model endpoint	http://localhost:2000	http://localhost:2000
OpenAPI docs	http://localhost:8080/docs	http://localhost:8081/docs
Raw schema	http://localhost:8080/openapi.json	http://localhost:8081/openapi.json

In local dev, Vite proxies /api from 5173 to 127.0.0.1:8081.

How the system is wired

Browser UI (Vue 3)
  -> FastAPI backend
    -> YAML config in data/config/
    -> Hugging Face downloads in data/models/ and data/hf-cache/
    -> engine installs in data/llama-cpp/, data/lmdeploy/, data/1cat-vllm/, and data/audio-cpp/
    -> CUDA installs in data/cuda/
    -> llama-swap config in data/llama-swap-config.yaml
  -> llama-swap on :2000
    -> llama.cpp / ik_llama.cpp / LMDeploy / 1Cat-vLLM / audio.cpp runtimes

The backend starts llama-swap automatically when there is at least one active runtime binary available from any registered engine (including audio-only installs).

First-run workflow

1. Start the app.
2. Open Engines.
3. Build and activate a llama.cpp or ik_llama.cpp version for GGUF models.
4. If you want safetensors support, install and activate LMDeploy or 1Cat-vLLM.
5. If you want audio tasks (TTS, ASR, VAD, and related), build and activate audio.cpp from source.
6. If you need gated Hugging Face access, set HUGGINGFACE_API_KEY or enter a token in the UI.
7. Open Search, find a model, and download or install it.
8. Open Models, configure the model, and choose its engine.
9. If the UI says the llama-swap config is stale, apply the pending config.
10. Start the model from the library.
11. Call it through http://localhost:2000/v1/....

Important:

• Saving model config updates the YAML store immediately.
• Applying pending llama-swap config rewrites data/llama-swap-config.yaml and unloads models before regenerating proxy state.
• GGUF models require an active llama.cpp or ik_llama.cpp build.
• safetensors models require an active LMDeploy or 1Cat-vLLM install.
• audio.cpp models require a prepared bundle installed or imported locally, plus an active audio.cpp build.

Docker quick start

Prerequisites

• Docker
• Docker Compose
• For NVIDIA GPU use: NVIDIA drivers on the host plus the NVIDIA Container Toolkit

CPU mode

git clone <repo-url>
cd llama-cpp-studio
docker compose -f docker-compose.cpu.yml up --build

This mode:

• exposes 8080 for the UI/API
• exposes 2000 for llama-swap
• mounts ./data to /app/data
• mounts ./backend to /app/backend
• enables backend reload with RELOAD=true

This is the best Docker option for backend-focused development. The frontend is still the built bundle from the image, not a live Vite dev server.

NVIDIA / CUDA mode

docker compose -f docker-compose.cuda.yml up --build -d

This mode:

• exposes the same ports: 8080 and 2000
• mounts ./data to /app/data
• reserves NVIDIA GPUs for the container
• disables backend reload

Manual image build

docker build -t llama-cpp-studio .

docker run -d \
  --name llama-cpp-studio \
  -p 8080:8080 \
  -p 2000:2000 \
  -v "$(pwd)/data:/app/data" \
  llama-cpp-studio

For NVIDIA GPUs, add --gpus all.

After startup

Open:

• UI: http://localhost:8080
• OpenAPI docs: http://localhost:8080/docs
• model endpoint: http://localhost:2000/v1/models

Local development

Prerequisites

• Node.js 20+
• Python 3
• a virtual environment tool such as venv
• a python executable on PATH if you want to use the provided npm scripts as-is

If you want to build runtimes on the host instead of inside Docker, you will also need native build tooling such as:

• cmake
• build-essential
• git
• pkg-config
• libopenblas-dev for OpenBLAS-backed CPU builds

Install dependencies

The repository scripts use python, so if your system only provides python3 you should either add a python alias or translate the commands below accordingly.

npm install
python -m venv .venv
source .venv/bin/activate
pip install -r requirements-dev.txt

Run frontend and backend together

npm run dev:all

That starts:

• frontend Vite dev server on 5173
• backend on 8081

You can also run them separately:

npm run dev:frontend
npm run dev:backend

Production-style local run

Build the frontend, then let FastAPI serve the built assets:

npm run build
python backend/main.py

When running outside Docker, the app stores persistent state under ./data.

Testing

npm run test

Or run suites individually:

npm run test:frontend
python -m pytest backend/tests -q

What lives in data/

The app is built around a persistent writable data directory. In Docker that is /app/data. Outside Docker it is ./data.

Typical layout:

data/
  config/
    models.yaml
    engines.yaml
    settings.yaml
    engine_params_catalog.yaml
    audio-cpp/
      servers/
  models/
    gguf/
    safetensors/
    audio-cpp/
  hf-cache/
  llama-cpp/
  lmdeploy/
  1cat-vllm/
  audio-cpp/
    builds/
    tools/
  cuda/
  logs/
  temp/
  llama-swap-config.yaml

What these are used for:

• config/models.yaml: downloaded models, their file ledgers, and per-model configuration
• config/engines.yaml: installed engine versions, active versions, and build settings
• config/settings.yaml: app settings such as Hugging Face token and proxy port
• config/llama_swap_routing.yaml: Studio-managed llama-swap profiles and selectors
• config/engine_params_catalog.yaml: parsed CLI parameter catalog used by the model config UI
• config/audio-cpp/servers/: generated one-model audiocpp_server JSON sidecars for llama-swap
• models/: downloaded GGUF, safetensors, and prepared audio.cpp bundles
• models/audio-cpp/: versioned prepared audio model packages
• hf-cache/: Hugging Face cache
• llama-cpp/: source checkouts and build artifacts for llama.cpp and ik_llama.cpp
• lmdeploy/: LMDeploy virtual environments and source installs
• 1cat-vllm/: 1Cat-vLLM virtual environments and source installs
• audio-cpp/builds/: source checkouts and build artifacts for audio.cpp
• audio-cpp/tools/: isolated Python virtual environment for the upstream model manager
• cuda/: CUDA Toolkit installs managed by the app
• logs/: installer and background task logs
• llama-swap-config.yaml: generated proxy configuration

Runtime and engine behavior

GGUF

GGUF models are managed as quantized entries grouped by Hugging Face repo. They run through either:

• llama.cpp
• ik_llama.cpp

Current engine management behavior:

• multiple versions can be installed and retained
• versions can be activated or deleted
• updates build the latest source ref with the saved build settings
• custom source repositories are labeled as forks while retaining source-sync behavior
• parameter support is discovered by scanning the active binary's --help output

For Hugging Face-backed GGUF models, the model config page can attach or replace mmproj, MTP, and DFlash companions without repeating the weight download. MTP and DFlash drafts are mutually exclusive. The Search and Model Config pages can also check the recorded weights and selected companions against Hugging Face and redownload only files whose remote metadata changed.

safetensors

safetensors repos are managed as logical model bundles and run through LMDeploy or 1Cat-vLLM.

Current Python-engine flows:

• install the latest or a specific LMDeploy version from PyPI
• install the latest or a specific 1Cat-vLLM release
• install either engine from a source repository and branch
• save default release/PyPI versions and source repo/branch settings without installing
• keep multiple installs in the engine registry
• label custom source repositories as forks while keeping them syncable
• activate or remove installs from the UI

CUDA

CUDA is managed as a persistent install under data/cuda.

The Docker image is prepared to use:

• CUDA_HOME
• CUDA_PATH
• LD_LIBRARY_PATH
• NCCL-related include and library paths

The app can install multiple CUDA versions and keeps a current symlink for the active one.

audio.cpp

audio.cpp is a native engine for prepared audio model bundles. It is gated by AUDIO_CPP_ENABLED (enabled by default). Maturity is per surface, not a single blanket label:

Surface	Maturity
Speech (/v1/audio/speech) and ASR (/v1/audio/transcriptions) via llama-swap	Primary Studio path; treat as stable once smoke-tested against your pin
Generic tasks via /upstream/{model}/v1/tasks/run	Supported but second-class until llama-swap routes /v1/tasks/run
Catalog discovery from upstream JSON (--list-loaders --json, package family / standalone, --inspect --json)	Stable on modern audio.cpp tips that advertise those contracts
Heuristic discovery fallback (fuzzy package→family matching)	Experimental; logged via discovery_source; controlled by AUDIO_CPP_HEURISTIC_DISCOVERY (default on)

Supported tasks include TTS, ASR, VAD, diarization, separation, generation, voice conversion, speech-to-speech, and alignment. Build backends include CPU, CUDA, and Vulkan on Linux; Metal is exposed only when the host supports it.

Current audio.cpp flows:

• build/update from a user-chosen tracking ref (branch/tag) persisted in engine settings; Update syncs in place when possible
• on activate/sync, rescan --help / --list-loaders (JSON preferred) and rediscover packages from the active checkout (no Studio package YAML mirror); persist a capability_delta for operator review
• after contract drift, review affected model configs
• discover verified-compatible packages through the normalized model catalog (/api/model-catalog/search), preferring authoritative package JSON fields over fuzzy matching
• install packages via tools/model_manager_v2.py when the active audio.cpp checkout has it (spec-backed GGUF downloads from model_specs/*.json); fall back to Studio HF snapshot download or the legacy model_manager.py / model_manager_deprecated.py for composite/converter packages
• install direct HF snapshots, composite/converter packages via the upstream model manager, or import a local prepared directory
• configure family, task, mode, backend, device, load options, and session options from scanned model profiles (generic request UI for unknown families); instructions policy prefers upstream inspect JSON
• run one audiocpp_server process per Studio model behind llama-swap using a generated JSON sidecar

Model manager behavior:

• Prefer tools/model_manager_v2.py for catalog listing and ready snapshot installs (no Torch required)
• Keep tools/model_manager_deprecated.py (or older tools/model_manager.py) for assemble/convert packages not supported by specs
• Python/Torch are required only by the legacy model manager helper path
• Studio creates an isolated helper venv under data/audio-cpp/tools/ on first use of legacy assemble/convert installs
• installs are validated with audiocpp_cli --inspect before promotion into data/models/audio-cpp/

Pinned upstream versions:

Component	Pin
audio.cpp repository	https://github.com/0xShug0/audio.cpp.git
Tracking ref	User-configurable (bootstraps from GitHub latest release / default branch)
llama-swap	v249

Studio proxies OpenAI audio under /v1/audio on :8080 (with WAV conversion for ASR). llama-swap v249 still does not route /v1/tasks/run; generic non-OpenAI audio tasks use the direct upstream fallback at /upstream/{model}/v1/tasks/run until that lands. When it does, Studio will bump the LLAMA_SWAP_VERSION pin and point examples at /v1/tasks/run through the proxy.

Virtual models & profiles (llama-swap selectors / profiles)

Studio generates llama-swap selectors (virtual model IDs) and profiles (runtime pin maps) into llama-swap-config.yaml. Edit them under Engines → Virtual models & profiles (deep link: /engines#ev-section-routing). Per-model API aliases remain on the model config page and are valid selector/profile targets.

Concept	Purpose
Selector (warm)	Prefer a target that is already loaded; otherwise start the first available
Selector (pin)	Always route to the first target
Selector (spillover)	Fill each target up to the spillover limit, then overflow to the next
Profile	Named map of client model ids → model / alias / selector (or empty to disable)

Workflow:

1. Edit selectors/profiles in the Engines panel and Save (writes config/llama_swap_routing.yaml, marks config stale).
2. Apply llama-swap config (panel button or header notice). This regenerates llama-swap-config.yaml and reloads the proxy; all loaded models stop.
3. Optionally switch the active profile live (panel dropdown or API) — only profiles already present in the applied YAML can be activated.

Example: coding selector with warm routing + a day/night profile:

curl -s -X PUT http://localhost:8080/api/llama-swap/routing \
  -H 'Content-Type: application/json' \
  -d '{
    "selectors": {
      "coding-model": {
        "strategy": "warm",
        "targets": ["my-fast-alias", "my-heavy-alias"],
        "name": "Coding"
      }
    },
    "profiles": {
      "day": { "description": "Daytime pins", "pins": { "llm": "coding-model" } },
      "night": { "description": "Night pins", "pins": { "llm": "my-fast-alias" } }
    }
  }'

curl -s -X POST http://localhost:8080/api/llama-swap/apply-config

curl -s -X PUT http://localhost:8080/api/llama-swap/profiles/active \
  -H 'Content-Type: application/json' \
  -d '{"name":"day"}'

Model configuration behavior

Each model stores configuration per engine. In practice that means:

• switching a model from llama.cpp to ik_llama.cpp or LMDeploy does not destroy the other engine sections
• the UI shows parameters based on the scanned catalog for the active engine
• unsupported flags can be hidden in the config view
• raw custom CLI args can be appended
• the saved llama-swap command can be previewed from the UI and API
• Hugging Face-backed GGUF and safetensors models keep a file ledger containing each recorded weight, shard, or selected companion plus available size and remote identity metadata
• checking for updates compares that ledger with Hugging Face and starts a background download only for changed files; files removed upstream are reported but not silently detached from the local model

Useful model-related routes:

• GET /api/models
• POST /api/models/search
• POST /api/models/download
• GET /api/models/{id}/companions
• POST /api/models/{id}/refresh
• POST /api/models/{id}/projector
• POST /api/models/{id}/mtp
• POST /api/models/{id}/dflash
• GET /api/model-catalog/search
• POST /api/model-catalog/install
• POST /api/model-catalog/import
• GET /api/models/{id}/config
• PUT /api/models/{id}/config
• GET /api/models/{id}/saved-llama-swap-cmd
• POST /api/models/{id}/preview-llama-swap-cmd
• POST /api/models/{id}/start
• POST /api/models/{id}/stop

Serving models

The user-facing inference endpoint is the llama-swap proxy on port 2000.

Useful requests:

curl http://localhost:2000/v1/models

curl http://localhost:2000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "replace-with-a-model-id-from-v1-models",
    "messages": [
      {"role": "user", "content": "Say hello in one sentence."}
    ]
  }'

The model value should come from GET /v1/models. It may be a sanitized Hugging Face repo plus quantization or a custom alias if you set one in model config.

Audio workspace

Studio’s Audio page (/audio) is a first-class workspace for audio.cpp models—same navigation weight as Models / Search / Engines. It reuses the model library, saved Model Config (family/task/defaults/voice presets), and reference-audio Assets. Deep links:

• Models row → Audio (audio.cpp packages)
• Model Config → Open Audio
• Query params: /audio?model=<id>&tab=speech|transcribe|…

Speech and Transcribe call Studio’s OpenAI-compatible proxy (below). Other task panels use llama-swap’s upstream /v1/tasks/run fallback when needed.

Audio models (TTS / ASR / voices)

llama-swap still serves OpenAI audio routes on port 2000. Studio also mounts the same paths under http://localhost:8080/v1/audio so clients can stay on the Studio origin. On POST /v1/audio/transcriptions, Studio converts non-WAV uploads (OGG/Opus, MP3, WebM, …) to WAV via ffmpeg, then forwards to llama-swap. Speech and other /v1/audio/* routes are passed through unchanged. Generic task calls use POST /v1/audio/tasks/run (Studio → llama-swap /upstream/{model}/v1/tasks/run) so the browser never needs cross-origin access to :2000. Dev (vite on :5173) proxies /api and /v1 to the backend.

Prefer Studio for ASR clients (including Hermes voice messaging):

# Hermes / OpenAI-compatible STT base URL
STT_OPENAI_BASE_URL=http://localhost:8080/v1

curl http://localhost:8080/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "replace-with-an-audio-model-id",
    "input": "Hello from audio.cpp",
    "voice": "default"
  }' \
  --output speech.wav

# Non-WAV (e.g. OGG) is converted before audio.cpp sees it
curl http://localhost:8080/v1/audio/transcriptions \
  -H "Authorization: Bearer local" \
  -F model="replace-with-an-asr-model-id" \
  -F file="@memo.ogg"

Direct llama-swap (WAV or already-compatible files):

curl http://localhost:2000/v1/audio/voices \
  -H "Authorization: Bearer local"

curl http://localhost:2000/v1/audio/transcriptions \
  -H "Authorization: Bearer local" \
  -F model="replace-with-an-asr-model-id" \
  -F file="@sample.wav"

For generic audio tasks exposed by /v1/tasks/run, use the direct upstream path until llama-swap adds unified routing:

curl http://localhost:2000/upstream/replace-with-model-id/v1/tasks/run \
  -H "Content-Type: application/json" \
  -d '{"task":"vad","input":{"audio_path":"/path/in/container.wav"}}'

App API surface

The FastAPI app exposes a small number of main route groups:

• /api/models: model library, downloads, config, start/stop, metadata
• /api/model-catalog: normalized search, install, import, and task status for verified packages
• /api/engines: engine capability descriptors used by the UI
• /api/llama-versions: engine versions, build settings, source builds, CUDA actions
• /api/audio-cpp: audio.cpp build, activation, status, and update checks
• /api/lmdeploy: LMDeploy install/remove/status/update checks and saved install defaults
• /api/1cat-vllm: 1Cat-vLLM install/remove/status/update checks and saved install defaults
• /api/status: system status and proxy health
• /api/gpu-info: GPU and CPU capability information
• /api/events: Server-Sent Events for progress and notifications
• /api/llama-swap: stale/apply/pending proxy configuration endpoints
• /api/llama-swap/routing: GET/PUT Studio-managed profiles and selectors
• /api/llama-swap/profiles: live profile listing; PUT .../active switches the active profile on the running proxy
• /v1/audio: OpenAI-compatible speech / transcriptions / voices proxy to llama-swap (ASR multipart may be ffmpeg-converted to WAV)

OpenAPI docs are available at /docs.

Environment variables

Most users only need a few environment variables:

Variable	Purpose
HUGGINGFACE_API_KEY	Access gated Hugging Face models and authenticated downloads
HF_HUB_ENABLE_HF_TRANSFER=1	Enable faster Hugging Face transfer support when available
HF_HOME	Base Hugging Face cache directory
HUGGINGFACE_HUB_CACHE	Hugging Face hub cache directory
CUDA_VISIBLE_DEVICES	Limit visible GPUs to comma-separated indices/UUIDs (e.g. 0,1). Do not set all — leave unset for all devices
RELOAD	Enable or disable backend auto-reload
BACKEND_CORS_ORIGINS	Comma-separated allowed origins
BACKEND_CORS_ALLOW_CREDENTIALS	Toggle credentialed CORS requests
CPU_ONLY_MODE	Force GPU detection into CPU-only mode
AUDIO_CPP_ENABLED	Enable or disable the audio.cpp integration (1 default, set 0/false to disable)
AUDIO_CPP_HEURISTIC_DISCOVERY	Allow fuzzy package→family discovery when upstream JSON omits fields (1 default; set 0 to require package JSON contracts)

Advanced / less common:

Variable	Purpose
LMDEPLOY_BIN	Override the LMDeploy executable path used by the backend
CMAKE or CMAKE_EXECUTABLE	Override the CMake executable used for source builds

Troubleshooting

data/ is not writable

The app needs write access to the mounted data directory. If the container logs complain about /app/data permissions, fix ownership on the host volume before continuing.

No models can start

Check these in order:

1. an engine version is installed and active
2. the model was downloaded successfully
3. pending llama-swap changes were applied
4. http://localhost:2000/health is reachable

Config changes are not reflected at inference time

Saving model config updates the database state, but the generated proxy config may still be stale. Use the UI's apply flow or call:

curl -X POST http://localhost:8080/api/llama-swap/apply-config

Parameter list is empty or outdated

Use the Engines page action to rescan CLI parameters for the active engine. The backend builds the parameter registry from the runtime binary's --help output.