Introduction

llama-dash turns a self-hosted local inference box into an observable, policy-controlled AI gateway: one UI for model state, request history, API keys, routing rules, proxy metrics, and client setup. The implemented inference backend is currently llama-swap over llama.cpp.

It is the single public entrypoint for OpenAI-compatible and Anthropic-compatible clients. llama-dash owns proxy policy, logging, auth, routing, and backend normalization; your inference backend owns local model processes and inference when traffic is routed to local models.

What it does

• Watch the box — live request, token, model, upstream, GPU, and update status in one dashboard.
• Manage models — load/unload models, inspect per-model stats, view residency history, and edit llama-swap config with validation.
• Proxy clients — expose one OpenAI/Anthropic-compatible /v1/* endpoint for local models, peers, and direct upstreams.
• Track requests — searchable request history with filters, histograms, detail views, attribution, token counts, and cost estimates.
• Control access — dashboard login, hashed API keys, per-key RPM/TPM limits, and model allow-lists.
• Enforce policy — routing rules for model rewrites, rejects, passthrough auth, direct upstreams, and encrypted credentials.
• Test models — playgrounds for chat, image, speech, and transcription.
• Export ops data — raw log streams, retention controls, and low-cardinality Prometheus metrics at /metrics.

Where to next