# llama.cpp & llama-swap (/docs/getting-started/llama-swap)



llama-dash does not run models itself. It sits in front of two upstream projects:

* **[llama.cpp](https://github.com/ggml-org/llama.cpp)** provides `llama-server`, the process that loads a GGUF model and serves inference.
* **[llama-swap](https://github.com/mostlygeek/llama-swap)** starts and stops `llama-server` processes on demand, based on the models declared in its `config.yaml`.

llama-dash proxies client requests to llama-swap and reads and edits llama-swap's `config.yaml`. How much of the upstream setup you have to do depends on how you install llama-dash.

|                                  | Docker Compose                      | Manual setup |
| -------------------------------- | ----------------------------------- | ------------ |
| Install llama.cpp                | No, bundled in the llama-swap image | Yes          |
| Install llama-swap               | No, the Compose file runs it        | Yes          |
| Write a llama-swap `config.yaml` | Yes                                 | Yes          |
| Download GGUF model files        | Yes                                 | Yes          |

## Supported llama-swap versions [#supported-llama-swap-versions]

The current llama-dash release is tested against **llama-swap v266**. Use v266 or newer. Earlier llama-dash releases were also run against v202 and v229.

Some llama-dash features rely on newer llama-swap APIs. On an older llama-swap, the rest of the dashboard and the proxy keep working, but these features are missing:

| Feature                                                                      | Minimum llama-swap |
| ---------------------------------------------------------------------------- | ------------------ |
| **Logs** page, which reads llama-swap's split proxy and upstream log streams | v259               |
| Inference-host hardware on the **System** page                               | v247               |
| Model capability badges on the **Models** page                               | v229               |

Run `llama-swap -version` to check your version. The upstream panel on the **Dashboard** also shows the version of the connected llama-swap.

The Docker Compose files use llama-swap's floating `unified-*` image tags, which always pull the newest build. To stay on a tested version, pin a release tag such as `unified-cuda13-266`. See [Updating](/docs/getting-started/updating#pinning-versions).

## Docker Compose [#docker-compose]

The [Docker Compose setup](/docs/getting-started/quick-start) runs llama-swap's recommended [unified image](https://github.com/mostlygeek/llama-swap#unified-container-recommended). It contains llama-swap and `llama-server`, along with whisper.cpp, stable-diffusion.cpp, and audio.cpp, so you don't need to install or build llama.cpp or llama-swap.

| Compose file                 | Image            | GPUs                                         |
| ---------------------------- | ---------------- | -------------------------------------------- |
| `docker-compose.nvidia.yaml` | `unified-cuda13` | NVIDIA Turing (RTX 20xx, GTX 16xx) and newer |
| `docker-compose.amd.yaml`    | `unified-vulkan` | AMD and other Vulkan GPUs                    |

For older NVIDIA cards (Pascal, such as the GTX 10xx or P40), change the image to `unified-cuda`.

You still provide two things:

1. One or more GGUF model files in `./models/`. [Hugging Face](https://huggingface.co/models?library=gguf) hosts most models in GGUF format.
2. A llama-swap `config.yaml` in `./config/` that declares each model and the `llama-server` command that runs it. The repository's `config/config.example.yaml` is a working starting point.

From llama-swap's documentation, you only need the [configuration overview](https://github.com/mostlygeek/llama-swap/blob/main/docs/kb/guides/configuration/configuration-overview.md). Skip its installation and Docker sections; the Compose file handles them.

## Manual setup [#manual-setup]

For a [manual setup](/docs/getting-started/manual-setup), llama-dash expects a llama-swap instance that is already running and reachable over HTTP. Set it up before you install llama-dash.

<div className="fd-steps">
  <div className="fd-step">
    ### Install llama.cpp [#install-llamacpp-step]

    Install a `llama-server` build that supports your GPU. llama.cpp's [install guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md) covers package managers and the [prebuilt release binaries](https://github.com/ggml-org/llama.cpp/releases). If no prebuilt binary matches your hardware, follow the [build guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md) for your backend, such as CUDA, ROCm/HIP, Vulkan, or Metal.

    Confirm the binary runs:

    ```bash
    llama-server --version
    ```

    You only need `llama-server`. llama.cpp's other tools are optional.
  </div>

  <div className="fd-step">
    ### Install llama-swap [#install-llama-swap-step]

    Follow llama-swap's [installation instructions](https://github.com/mostlygeek/llama-swap#installation). Homebrew or a [release binary](https://github.com/mostlygeek/llama-swap/releases) is the simplest option on a bare-metal host. You can also run llama-swap with Docker while llama-dash runs directly on the host.
  </div>

  <div className="fd-step">
    ### Write the llama-swap config [#write-the-llama-swap-config-step]

    Create a `config.yaml` that declares your models. Model paths are host paths in this setup:

    ```yaml
    models:
      my-model:
        cmd: llama-server --port ${PORT} --model /srv/models/my-model.gguf
    ```

    `my-model` is the model ID clients send. llama-swap replaces `${PORT}` with a free port when it starts the model. See llama-swap's [configuration overview](https://github.com/mostlygeek/llama-swap/blob/main/docs/kb/guides/configuration/configuration-overview.md) for TTLs, aliases, groups, and other options.
  </div>

  <div className="fd-step">
    ### Start llama-swap [#start-llama-swap-step]

    ```bash
    llama-swap -config /etc/llama-swap/config.yaml -listen 127.0.0.1:8080 -watch-config
    ```

    `-watch-config` makes llama-swap reload the file when it changes, which is how edits from the dashboard's [config editor](/docs/dashboard/config-editor) take effect. Run llama-swap under systemd or another supervisor so it restarts with the host.

    Confirm llama-swap responds and lists your models:

    ```bash
    curl http://127.0.0.1:8080/v1/models
    ```

    Then continue with [Manual setup](/docs/getting-started/manual-setup) and set `INFERENCE_BASE_URL` to this address and `INFERENCE_CONFIG_FILE` to the absolute path of this `config.yaml`.
  </div>
</div>

## Where the config fits [#where-the-config-fits]

The **Config** page in the dashboard edits this llama-swap `config.yaml`, not a separate llama-dash config. `INFERENCE_CONFIG_FILE` tells llama-dash where that file is. llama-dash's own settings live in `.env` and in the dashboard's database. See [Configuration](/docs/getting-started/configuration).