Local LLM

Run the model where it makes sense. Fast and resident on the laptop, big and always-on for the LAN, served from a desktop-class Apple Silicon host the laptops call as clients. “Local LLM” here means two distinct stacks that answer to the same OpenAI-shaped API, plus the strategy that decides which one runs what. Neither is an agent. Both are just the model plus a serving stack. The agent layer (Claude Code, Antigravity, the routines) runs outside them and calls in over HTTP.

• The client laptops: mlx-lm workers behind llama-swap on a laptop-class M4 Max, one resident model, tuned to coexist with a working desktop. This is a laptop’s own private model for delegated edits, drafts, and “don’t burn cloud tokens on this” tasks.
• The large-memory tier: a desktop-class Apple Silicon host provides serial capacity for models that only fit in its 128 GB of unified memory. A separate GPU tier provided higher throughput in the tested one-user comparison. Each tier serves a different memory and concurrency envelope; neither replaces the other.

The cloud frontier models still win on the hardest reasoning. The point of local isn’t to beat them. It’s to own the routine, private, and high-volume work without metering, and to keep a credible offline fallback.

​

One gateway, one endpoint

The design uses one self-hosted, OpenAI-compatible LLM gateway for the local models. Clients use one API, URL, and key. The gateway routes requests to registered deployments. The Mac Studio provides serial capacity for large models that only fit in its 128 GB memory. The separate GPU tier has higher throughput in the measured one-user comparison. That comparison used the same 27 B parameter model with different backends and quantizations. It does not establish live router priority. The homelab’s self-hosted autonomous agents run continuously and connect to that same endpoint as their model backend. A free hosted tier sits behind both local tiers in the fallback order. It is reached only when local capacity is unavailable, never in front of it. See Model routing for the general policy this follows.

​

Current shape, and what’s still gated

The Mac Studio is the capacity tier for serial service of large models that only fit in its 128 GB memory. This page does not assert a current resident model roster or multi-request saturation. The separate GPU tier had higher throughput in the tested one-user comparison. The same 27 B parameter model was measured with different backends and quantizations. That result does not establish live router priority. A boot warmup LaunchAgent (mlx-warmup) faults model weights into memory at boot so the first request does not pay the cold-start cost. For the recorded model roster, read What is actually loaded, a dated record. The re-measured, regime-stated figures live in Re-measured in isolation and the 2026-10-07 single-request comparison. The historical tuning report is in nix-darwin MACOS-LLM-PERFORMANCE-TUNING-REPORT.md. Note that this does not turn two machines into one 256 GB pool. Apple Silicon unified memory can’t be merged across a cable. The separate, still-gated direction is combined capacity by sharding. It means a model too big for either machine alone, run across both over a fast interconnect, unattended, by morning. That’s a capacity win, not a speed win, and it remains measurement-gated. See Distributed & multi-Mac for the honest picture.

​

The principles that hold across both stacks

• A small resident set, not a rotation. Swapping a multi-GB model evicts wired GPU memory and reloads another. That’s the slowest thing you can do. A laptop holds one resident model; the serving host holds two. Capability-role aliases (default, coding, quickest, tool-calling, …) resolve onto that resident set, so a role never triggers a swap. How many are resident is a property of the host, read from the registry, not a fixed number.
• The registry is the source of truth. Which physical model is resident lives in one place: the AI-stack registry that nix-ai writes at activation, read as ~/.config/ai-stack/registry.json. Docs describe the strategy; the current id is a registry value, never hard-coded here. See Models & quantization.
• Measure, don’t claim; then verify the instrument. No tuning change ships on a marketing number, only a measured one. But a measurement is only as good as proof of which weights the process loaded, so every number carries that provenance. See Verifying the instrument.
• MoE for throughput. A sparse mixture-of-experts model with a few billion active parameters decodes far faster than a dense model of the same total size. That’s the lever that makes a big model usable on a laptop.
• Capacity, not speed, across machines. Sharding one model over two Macs is communication-bound; reserve it for models that don’t fit, and run two independent workers for everything that does.

​

In this section

​

How it connects