llama-dash

What llama-dash needs from the inference backend, and how much of it you have to set up yourself.

llama-dash does not run models itself. It sits in front of two upstream projects:

llama.cpp provides llama-server, the process that loads a GGUF model and serves inference.
llama-swap starts and stops llama-server processes on demand, based on the models declared in its config.yaml.

llama-dash proxies client requests to llama-swap and reads and edits llama-swap's config.yaml. How much of the upstream setup you have to do depends on how you install llama-dash.

	Docker Compose	Manual setup
Install llama.cpp	No, bundled in the llama-swap image	Yes
Install llama-swap	No, the Compose file runs it	Yes
Write a llama-swap `config.yaml`	Yes	Yes
Download GGUF model files	Yes	Yes

Supported llama-swap versions

The current llama-dash release is tested against llama-swap v266. Use v266 or newer. Earlier llama-dash releases were also run against v202 and v229.

Some llama-dash features rely on newer llama-swap APIs. On an older llama-swap, the rest of the dashboard and the proxy keep working, but these features are missing:

Feature	Minimum llama-swap
Logs page, which reads llama-swap's split proxy and upstream log streams	v259
Inference-host hardware on the System page	v247
Model capability badges on the Models page	v229

Run llama-swap -version to check your version. The upstream panel on the Dashboard also shows the version of the connected llama-swap.

The Docker Compose files use llama-swap's floating unified-* image tags, which always pull the newest build. To stay on a tested version, pin a release tag such as unified-cuda13-266. See Updating.

Docker Compose

The Docker Compose setup runs llama-swap's recommended unified image. It contains llama-swap and llama-server, along with whisper.cpp, stable-diffusion.cpp, and audio.cpp, so you don't need to install or build llama.cpp or llama-swap.

Compose file	Image	GPUs
`docker-compose.nvidia.yaml`	`unified-cuda13`	NVIDIA Turing (RTX 20xx, GTX 16xx) and newer
`docker-compose.amd.yaml`	`unified-vulkan`	AMD and other Vulkan GPUs

For older NVIDIA cards (Pascal, such as the GTX 10xx or P40), change the image to unified-cuda.

You still provide two things:

One or more GGUF model files in ./models/. Hugging Face hosts most models in GGUF format.
A llama-swap config.yaml in ./config/ that declares each model and the llama-server command that runs it. The repository's config/config.example.yaml is a working starting point.

From llama-swap's documentation, you only need the configuration overview. Skip its installation and Docker sections; the Compose file handles them.

Manual setup

For a manual setup, llama-dash expects a llama-swap instance that is already running and reachable over HTTP. Set it up before you install llama-dash.

Install llama.cpp

Install a llama-server build that supports your GPU. llama.cpp's install guide covers package managers and the prebuilt release binaries. If no prebuilt binary matches your hardware, follow the build guide for your backend, such as CUDA, ROCm/HIP, Vulkan, or Metal.

Confirm the binary runs:

llama-server --version

You only need llama-server. llama.cpp's other tools are optional.

Install llama-swap

Follow llama-swap's installation instructions. Homebrew or a release binary is the simplest option on a bare-metal host. You can also run llama-swap with Docker while llama-dash runs directly on the host.

Write the llama-swap config

Create a config.yaml that declares your models. Model paths are host paths in this setup:

models:
  my-model:
    cmd: llama-server --port ${PORT} --model /srv/models/my-model.gguf

my-model is the model ID clients send. llama-swap replaces ${PORT} with a free port when it starts the model. See llama-swap's configuration overview for TTLs, aliases, groups, and other options.

Start llama-swap

llama-swap -config /etc/llama-swap/config.yaml -listen 127.0.0.1:8080 -watch-config

-watch-config makes llama-swap reload the file when it changes, which is how edits from the dashboard's config editor take effect. Run llama-swap under systemd or another supervisor so it restarts with the host.

Confirm llama-swap responds and lists your models:

curl http://127.0.0.1:8080/v1/models

Then continue with Manual setup and set INFERENCE_BASE_URL to this address and INFERENCE_CONFIG_FILE to the absolute path of this config.yaml.

Where the config fits

The Config page in the dashboard edits this llama-swap config.yaml, not a separate llama-dash config. INFERENCE_CONFIG_FILE tells llama-dash where that file is. llama-dash's own settings live in .env and in the dashboard's database. See Configuration.

llama.cpp & llama-swap