Skip to main content
The homelab runs its own ChatGPT — and every other model size below it — behind one endpoint: https://llm.<domain>/v1. No tokens metered by a vendor, no prompt ever leaving the network, and no consumer ever needing to know which machine actually served the request. The name resolves to a load-balanced pool of three stateless LiteLLM routers. The routers hold the model table and fan requests out by tier:
  • Large tier — the biggest models (gpt-oss-120b, qwen3-coder) run on an Apple-Silicon machine behind an authenticated TLS gate; the router holds the bearer token, consumers never see it.
  • Fast tier — small, quick models (qwen3-4b and the embeddings model) on a Radeon RX 6800 in a privileged LXC, served by llama-swap fronting llama.cpp’s llama-server (ROCm build, models co-resident).
  • CPU standby — the same llama.cpp lineup, CPU-only, on a node that sleeps most of the day; cold capacity if the GPU host is out.
Current state: the fast tier’s GPU serving is disabled in the configuration and the routers carry only the CPU standby for the small-model aliases — the degradation path above is what serves today. The endpoint, model list, and client usage below are unaffected; the GPU deployments can be restored by flipping the same configuration gates.

​
How you reach it

Teal is a client, ink is the DNS + reverse-proxy edge, coral is the serving fabric. Traefik health-checks the router pool (/health/liveliness) and drops a dead router without the endpoint changing — stopping one router live is a non-event. Every name resolves through Technitium and carries the wildcard certificate, so it is HTTPS end to end.

​
What’s in the stack

All guests are DHCP-first LXCs on the ai VLAN with deterministic MAC reservations, so every address is a DNS name. llm-fast is a privileged LXC with the GPU passed through (/dev/kfd + /dev/dri); models live on a 120 GB fast-pool volume. The LXCs, firewall groups, and the Traefik pool entries are provisioned by tofu-proxmox; llama.cpp, llama-swap, the routers, and Open WebUI are configured by ansible-proxmox-apps.
This is not the same “local AI” as the Apple Silicon stack. That one is the MLX server on this MacBook, tuned to hold one resident model for local-first work. This page is the shared fabric — always on, LAN-wide, and the MacBook falls back to it through the same router endpoint.

​
Use it from your Mac

Everything below is reachable by DNS name over HTTPS. Replace example.net with your homelab’s internal domain. The router requires a bearer key (issued per consumer), unlike the old unauthenticated single-backend endpoint.

​
1 · Browser

Open https://chat.example.net, sign in, pick a model, and chat. This is the full Open WebUI — conversation history, system prompts, file uploads — talking to the router like every other consumer.

​
2 · OpenAI-compatible API

Any OpenAI client works — change the base URL and send the key:
GET /v1/models lists every alias the fabric serves; picking a model picks a tier, and the router handles placement, fallbacks, and the heavy backend’s authentication for you.

tofu-proxmox

Provisions the LXCs, GPU passthrough, firewall groups, and the Traefik router pool.

ansible-proxmox-apps

Installs llama.cpp + llama-swap, the LiteLLM routers, and Open WebUI.

LXC vs Docker

Why the inference stack runs as native LXC, not Docker.

AI development pipeline

The other “local AI” — MLX on the workstation, local-first with fabric fallback.