The proxy

The proxy is the client-facing /v1/* endpoint. It sends OpenAI and Anthropic requests to the selected backend or a configured direct upstream, and logs the completed exchange.

Streaming and token counting

• Streaming stays streaming. SSE chunks reach the client as they arrive. llama-dash does not gather the response and send it at the end.
• Usage is read in flight. A separate scan reads the final usage chunk without consuming the response body before it is forwarded.
• One row represents one finished exchange. If the client disconnects mid-stream, llama-dash still records the request with any usage captured so far.

OpenAI and Anthropic compatibility

The proxy supports these request families:

• OpenAI: /v1/chat/completions and other /v1/* endpoints with flat usage fields.
• Anthropic: /v1/messages and /v1/messages/count_tokens, with nested message.usage and a message_stop stream terminator.
• llama.cpp and llama-swap extensions: native /infill and /audiocpp/* pass-throughs.

Cost estimates

llama-dash estimates cost from upstream usage counters and a models.dev pricing catalog loaded at startup. If it does not know the price, the request shows no estimate rather than an invented number.

Transform pipeline

POST /v1/* requests pass through an ordered transform pipeline between auth and forwarding. The body is parsed once when needed and re-serialized only if mutated:

1. API key allow-list check
2. Routing rule evaluation
3. Model alias resolution
4. System-prompt injection
5. Request size limits

Routing rules are ordered and first-match-wins. See Request logging for what is captured afterwards.