One resident model, behind a swapping proxy, tuned so the GPU serves tokens without starving the machine you’re actually using.The workstation serves its local model through two processes:
llama-swap, a
thin model-multiplexing proxy on localhost:11434, and one vllm-mlx worker
that does the MLX inference. Every caller speaks the OpenAI API; the proxy
routes to the resident worker and manages its lifecycle.
The hard part on a laptop isn’t raw speed — it’s coexistence. A large model
wires gigabytes of GPU memory that the kernel can’t cheaply reclaim, and a
desktop full of real work is competing for the same RAM. Most of the tuning
below exists to keep inference from quietly pushing the whole machine into swap.
The stack
llama-swap is launched by a macOS LaunchAgent and watches its own config, so a
nix-darwin rebuild can add or repoint models without a restart. The worker
spawns on an ephemeral port; the proxy is the only stable surface. Weights are
memory-mapped from an external volume so the multi-GB files don’t live on the
boot disk. The whole thing is packaged by nix-ai — nothing here
is configured by hand.
The tuning playbook
These are the knobs that matter and the reasoning behind each. The values are the module defaults, calibrated for a single 128 GB M4 Max; they are not portable to a smaller machine without re-measuring.The resident model id is not a tuning knob — it lives in the AI-stack
registry, not in these flags. Picking and quantizing that model is its own
topic: Models & quantization.
Concurrency is measured, not assumed
Raising concurrency on this stack ranges from useless to actively harmful. Measured on a large MoE model served with a single decode slot, weights proven by process inspection (verified 2026-07-26/27):
With one decode slot the server serializes rather than batches, so extra load
buys queueing, not throughput. For large mixture-of-experts models, concurrency
above 1 is a regression, not a missed opportunity. Whatever the right number is
for a small model, measure it per model — it does not scale from one model
to the next.
Single-slot is the current setting, not a permanent verdict. It is what
holds while single-stream stability is being settled. The intended direction is
to raise concurrency once concurrency-1 is solid — per model, on measured
evidence, never as a global flip.
The proxy rejects; the server serializes
Two layers behave differently under load, and a client has to know which one it is talking to.- The swapping proxy rejects. A second simultaneous request gets an immediate HTTP 429 with a “too many requests” body. No queue, no partial service. Every client must treat 429 as a normal condition and back off, not raise an alert.
- The model server serializes. Behind the proxy, with a single decode slot, requests wait their turn instead of being refused. That is where the table above comes from.
One definition, and it is derived
Express the concurrency limit as one definition consumed everywhere, never as a literal repeated in two places. A derived1 and a hardcoded 1 are
textually identical — only the source line tells them apart, which is exactly
why two copies drift without anyone noticing. That happened here before the
limit became a single derived value with a build-time assert that the emitted
serve flags equal the declared one.
Speculative decoding is off — on purpose
Speculative decoding and multi-token prediction (MTP) are marketed as free speedups. On Apple Metal, for the sparse-MoE models this stack runs, they are a net loss: the draft-evaluation overhead doesn’t amortize, and for a small-active-parameter MoE the regression is severe (a documented case collapses from tens of tok/s to under 2). Upstreamllama.cpp reports the same on Metal
(#23752,
#23011). Draft flags are
banned from this stack until a measured win on this hardware says otherwise.
Apple’s own recurrent-drafter research shows speculative methods can help in
other settings — so this is “off because measured slower here,” not dogma.
Why the laptop slows down under sustained use
Even when no single layer is broken, a laptop running inference for days gets sluggish — because the kernel never gets an idle window to drain its memory compressor and swap. The ingredients are generic: a wired model weight pool, other heavy apps holding most of RAM, and background processes preventing idle sleep. The highest-leverage fixes are structural:- Hold one resident model and evict it aggressively (short idle TTL) so the wired pool doesn’t sit for days.
- Right-size the cache reservation so the worker’s footprint matches its real batch, not a worst case.
- Give the machine idle-sleep windows so the compressor can drain.
Operate it
1
Check health
mlx-status shows the resident model, memory, and uptime; mlx-models
lists what’s downloaded and whether it fits in memory.2
Send a request
Any OpenAI client against
http://localhost:11434/v1 works. mlx is a
one-shot prompt; mlx-chat is interactive.3
Reset the proxy
mlx-default restarts llama-swap and preloads the resident model — the
fix when a worker has been evicted under pressure.Related
Models & quantization
What model is resident, at what precision, and why.
Backends & tool calling
Why
vllm-mlx, and the tool-calling reliability problem.nix-ai
The module that generates the LaunchAgent and the
llama-swap config.Operational reference (private)
Real values, failure fingerprints, and the
/perf-snapshot playbook.