Self-hosted ChatGPT

The homelab runs its own ChatGPT, and every other model size below it, behind one endpoint: https://llm.<domain>/v1. Cloud providers are reachable only through keys that explicitly grant access. No consumer needs to know which machine actually served the request. The name resolves to a load-balanced pool of stateless LiteLLM routers. The routers hold the model table and fan requests out by tier:

Large tier: the biggest models run on an Apple-Silicon machine behind an authenticated TLS gate; the router holds the bearer token, consumers never see it. Which models are actually resident changes as the roster is re-evaluated. See What is actually loaded for the dated roster; this docs pass did not verify live weights. Its role is memory capacity for serial service of models that only fit in 128 GB, not a claim that it is the fastest backend.
Router capabilities: small, medium, large, embed, and ocr. Within each capability, the router selects eligible deployments in registry order.
CPU candidate: qwen3.5-9b belongs to small at order 3. The router omits it while its CPU backend group is absent.
A free hosted tier, reachable through a stable handle only for keys with an explicit cloud grant. The underlying model lives in provider configuration rather than in anything committed here. See the stable-handle pattern. It is not an automatic fallback.
Paid providers, reached through the configured fallback policy or by an explicit model identifier, only for keys with an explicit cloud grant. The wildcard passthrough still requires an explicit id. See delegation across capable tiers.

The router follows configured route order. A per-token rate change does not reorder eligible tiers. See Model routing for the policy and LiteLLM price map for rate resolution. In one-user testing, the discrete GPU ran the same 27 B parameter model about 2.3 times faster than the Mac Studio. The tests used different backends and quantizations. The Mac Studio remains useful for serial service of large models that only fit in its 128 GB memory. This comparison describes measured configurations and does not establish a router priority or current live route.

How you reach it

Teal is a client, ink is the DNS and reverse-proxy edge, and coral is the local serving fabric. Dashed teal marks cloud routes, which require an explicit grant on the caller’s key. Traefik health-checks the router pool (/health/liveliness) and drops an unhealthy member without changing the endpoint. This describes the intended redundancy; check current monitoring before relying on pool health. Every name resolves through Technitium and carries the wildcard certificate, so it is HTTPS end to end.

What’s in the stack

Piece	Does	Reached at
LiteLLM router pool	Stateless proxies hold the model table, tier fallbacks, and the heavy-backend bearer	`https://llm.<domain>/v1` (pooled)
GPU deployments	Eligible members of the `small` and `medium` capabilities	via the router
CPU candidate (9-billion-parameter model)	`qwen3.5-9b` in `small`, order 3; omitted without a backend	not routed
`llm-light`	CPU deployment for the `embed` capability	via the router
free hosted tier	Reached through a stable handle only for keys with an explicit cloud grant; not an automatic fallback	via the router
paid providers	Reached through configured fallback or an explicit model identifier, only for keys with an explicit cloud grant	via the router
`open-webui`	Chat front-end, default model = the `medium` capability	`https://chat.<domain>`
`qdrant`	Vector store for retrieval	`https://qdrant.<domain>`
`llamaindex`	RAG engine; uses the router’s `embed` capability, stores in Qdrant	internal (RAG)

This is not the same “local AI” as the Apple Silicon stack. That one is the MLX server on this MacBook, tuned to hold one resident model for local-first work. This page is the shared fabric: always on, LAN-wide, and the MacBook falls back to it through the same router endpoint.

Use it from your Mac

Everything below is reachable by DNS name over HTTPS. Replace example.net with your homelab’s internal domain. The router requires a bearer key (issued per consumer), unlike the old unauthenticated single-backend endpoint.

1 · Browser

Open https://chat.example.net, sign in, pick a model, and chat. This is the full Open WebUI: conversation history, system prompts, and file uploads, all talking to the router like every other consumer.

2 · OpenAI-compatible API

Any OpenAI client works. Change the base URL and send the key:

curl https://llm.example.net/v1/chat/completions \
  -H "Authorization: Bearer $LLM_ROUTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"small","messages":[{"role":"user","content":"hello"}]}'

from openai import OpenAI

client = OpenAI(base_url="https://llm.example.net/v1", api_key=llm_router_key)
resp = client.chat.completions.create(
    model="small",  # use embed for vectorization; medium is the default
    messages=[{"role": "user", "content": "hello"}],
)
print(resp.choices[0].message.content)

GET /v1/models lists the names the key can use. A capability name lets the router pick an eligible deployment in registry order; the router handles fallbacks and the heavy backend’s authentication for you.

tofu-proxmox

Provisions the LXCs, GPU passthrough, firewall groups, and the Traefik router pool.

ansible-proxmox-ai

Installs llama.cpp + llama-swap, the LiteLLM routers, and Open WebUI.

LXC vs Docker

Why the inference stack runs as native LXC, not Docker.

AI development pipeline

The other “local AI”: MLX on the workstation, local-first with fabric fallback.

AI development

Autonomous agents

Agent ops

Local LLM

How you reach it

What’s in the stack

Use it from your Mac

1 · Browser

2 · OpenAI-compatible API

tofu-proxmox

ansible-proxmox-ai

LXC vs Docker

AI development pipeline

​How you reach it

​What’s in the stack

​Use it from your Mac

​1 · Browser

​2 · OpenAI-compatible API

​Related

tofu-proxmox

ansible-proxmox-ai

LXC vs Docker

AI development pipeline

How you reach it

What’s in the stack

Use it from your Mac

1 · Browser

2 · OpenAI-compatible API

Related