How the Nix stack serves a model

One host file names a couple of numbers. Nix turns them into a sysctl, a set of serve flags, a swap policy, and a build-time safety check — then launches the worker. This page follows that path from top to bottom.

Two Apple Silicon Macs run the local models. Neither is an agent. Each is just a model plus a serving stack — vllm-mlx workers behind a llama-swap proxy, answering an OpenAI-shaped API on loopback. The agent layer calls in over HTTP. See Local LLM for the strategy; this page is the mechanism.

​

Two hosts, two roles

The split is by job, not by hardware. One host is headless; the other runs an interactive desktop.

• The headless host is the always-on serving host for the whole LAN. It has no desktop working set to protect, so it reclaims that memory and holds two models warm at once, with more available on demand.
• The interactive laptop self-serves — localhost clients only, so serving never competes with the network host. It keeps one model resident, sized to coexist with an interactive desktop and the user’s own work.

The rule: heavy serving belongs on the headless host. Anything large or structured is delegated to it over the LAN, off the laptop’s battery and memory. The full model roster, verdicts, and performance numbers live in Mac Studio serving; the per-host memory numbers live in the gated operational reference.

​

Ports

Every serving host exposes the same shape on loopback; only the headless host adds a LAN-facing gate.

​

How the repos compose

Two repos meet to launch one worker. nix-darwin owns the host — the OS memory ceiling and power knobs. nix-ai owns the serving stack — which models exist, how each is served, and the command that starts it. A host file picks values; the modules turn them into a running process.

​

The nix-darwin side: the host ceiling

modules/darwin/apple-silicon-tunables.nix owns the machine’s memory ceiling and inference power knobs. Its main option is wiredLimitMb — the iogpu.wired_limit_mb sysctl, the OS ceiling on wired GPU memory. The sysctl is volatile (it resets on reboot), so the module re-applies it from a launchd daemon at boot and again at every rebuild. The ceiling is not static. A link watcher flips the sysctl to the current mode’s budget when the cluster cable is plugged or unplugged — the clustered budget on plug, the standalone budget on unplug. That is a live sysctl write, not a rebuild, so the mode’s memory envelope moves without one. Each host declares two budgets against its job — the headless host carries higher ones because it reclaims the desktop working set the laptop must leave free. The values are not typed by hand; each is derived from a memory budget, wiredLimitMb = maxLocalLlmGb × 1024. See the runtime-dynamic derivation for the whole scheme; the per-host budgets live in the gated operational reference. The module also handles power posture (Low Power Mode off, Power Nap off, App Nap disabled for the inference daemon) and a Metal debug-env guard. None of that picks a model; it shapes the box the model runs in.

​

The nix-ai side: the serving stack

nix-ai holds four kinds of file under modules/mlx/:

​

The launch path, end to end

1. A host file picks model entries, assigns each a class, and sets host-scoped posture — the preload list, the swap policy, the concurrency limit.
2. catalog-data.nix supplies each chosen model’s family args and its class profile; the host’s knob values merge on top.
3. vllm-cmd.nix renders one vllm-mlx serve command per resident model, folding in the merged flags.
4. llama-swap sits on the API port and manages the workers as children. The headless host keeps several resident at once (groupSwap = false); the laptop keeps exactly one, evicting the previous model on any switch.
5. A warmup agent faults the preload list into memory at boot, so the first request never pays a cold start.
6. nix-darwin has already set the wired ceiling the worker allocates under.

The registry that records which physical model is the current default is written at activation and read as ~/.config/ai-stack/registry.json. Docs describe the strategy; the live id is always a registry value, never hard-coded. For the packaging view of these repos, see AI tooling and macOS host.

​

The memory-safety control plane

Two memory behaviours ride on one flag, over a wired ceiling that must move with the serving mode. Setting either by hand is the usual reason a well-sized host still pages. But the safety is not baked in at build time — models and modes change while the host runs, and rebuilding to swap a model is friction the stack avoids. Nix ships the control plane; the plane enforces at runtime. Nix’s build-time job is narrow:

• the mode-aware budgets — one maxLocalLlmGb for standalone serving and a larger one for clustered mode, per host.
• the sysctl mechanism and the mode-switch watcher — the launchd daemon that applies iogpu.wired_limit_mb, and the link watcher that flips it to the current mode’s budget on plug or unplug.
• the admission-control wrapper and the model arch data — the wrapper that computes the per-model caps, and the catalog fields it reads.

Runtime does the rest, with no rebuild:

• the link watcher raises or lowers the wired ceiling to the mode’s budget when the cluster cable changes.
• the admission wrapper computes --max-cache-blocks and derived --gpu-memory-utilization at each model load from the live sysctl, the model’s architecture, and the requested concurrency and context — and rejects or clamps a load that will not fit. Swapping a model needs no nix change.

A build-time assert remains, demoted to a policy check: it verifies the declared budgets are internally consistent and each leaves an OS floor, not that any one model fits. The per-model guarantee is the runtime admission. The full derivation, the two-behaviour mechanism behind it, why --max-cache-blocks (not --cache-memory-mb) is the real footprint cap, and the seven invariants that make overload virtually impossible live in Memory ceilings on Apple Silicon. This page links to that mechanism rather than restating it.

​

Related