Skip to main content
Two memory behaviours, one flag, and a host ceiling that never moves on its own. Tune them as a pair or the machine finds the edge for you.
A local MLX serving stack — a swapping proxy in front of one or more inference workers — has a single flag, --gpu-memory-utilization, that drives two different memory behaviours at once. Reading it as one knob is the usual reason a well-sized host still ends up paging.

​
The two behaviours

The flag is a fraction. It feeds two independent calculations:
  • allocation_limit is the per-worker Metal allocation cap. A worker that tries to allocate past it fails the allocation.
  • trip is the engine’s emergency KV-cache clear. When resident memory crosses it, the engine drops the cache to recover headroom.
The two use different denominators, and that is the whole story: Raising the wired ceiling raises allocation_limit but leaves trip exactly where it was.

​
Insight one: it is a cap, not a reservation

--gpu-memory-utilization does not pre-allocate anything. Setting it to a large fraction does not hand that memory to the worker, and setting it small does not hold memory back for the desktop. It only says how far an allocation is allowed to go before it is refused. The knob that actually decides how much of a host goes to inference is the OS wired ceiling. Treat --gpu-memory-utilization as a safety rail on top of that decision, not as the decision itself.

​
Insight two: the two knobs move together

The invariant to hold is:
The trip point is a soft valve. It has to fire before the host reaches its hard ceiling and starts paging, and it has to sit above the worker’s steady resident footprint so it does not fire during normal work. Move either knob alone and one side of that invariant breaks:

Lowering utilization alone

The trip point drops below the resident working set. The engine clears the KV cache constantly, every turn re-prefills from scratch, and multi-turn sessions degrade badly while the GPU looks busy.

Raising the ceiling alone

The trip point stays put while the hard ceiling climbs past it — or the trip lands above the ceiling entirely, so the soft valve never fires and the host reaches its hard limit first.
Neither knob is safe to change by itself. Any change to the wired ceiling needs a matching re-derivation of --gpu-memory-utilization, and the reverse.

​
Re-deriving the utilization

Rearranging the invariant gives a symbolic band for the flag:
The lower bound keeps the allocation cap above what the worker actually holds. The upper bound keeps the trip point below the hard ceiling, with the engine’s own 0.05 offset subtracted out. Pick a value inside the band with margin on both sides, and re-derive it whenever the ceiling, the model, or the batch shape changes. If the band is empty — the lower bound exceeds the upper — no value of the flag satisfies the invariant on that host. Raise the ceiling, shrink the footprint (smaller cache reservation, fewer concurrent sequences, a smaller model), or both.

​
Measuring the footprint: use wired, not RSS

On macOS, ps RSS does not attribute kernel-wired Metal allocations to the process that caused them. A worker holding tens of GB of wired GPU memory can look small in ps while top shows the real figure. Size against the wired number; RSS will quietly tell you the host has room it does not have.

​
Finding the real ceiling

The default wired ceiling on Apple Silicon is commonly quoted as about 75% of RAM. In practice it can be a substantially larger fraction than that. The rule of thumb is a guess; measure the machine instead.
1

Read the ceiling from MLX

Query max_recommended_working_set_size through MLX. That is the value the allocator actually enforces, whatever the sysctl is currently set to. Read it before changing anything — the default may already be higher than you were about to set.
2

Set the sysctl, then re-read

Change iogpu.wired_limit_mb and read max_recommended_working_set_size again to confirm the new ceiling took effect.
3

Re-derive utilization

Feed the confirmed ceiling back through the band above and set --gpu-memory-utilization to a value inside it.
4

Confirm the engine actually applied it

Read back the limit the engine reports at startup, not just the flag you passed. Serving stacks build their engine down more than one code path, and a flag honored on one path can be silently ignored on another — leaving the default in force while your command line says otherwise. Trust the engine’s own log line, then check the resulting trip point still satisfies the invariant.
Passing --gpu-memory-utilization is not proof it took effect. If an idle auto-unload or lazy-load feature routes model construction through a secondary path, the value can revert to the engine default there. The only reliable signal is the allocation limit the engine prints once the model loads — if it does not match your flag, the invariant you derived is not the one running.

​
The unit is MiB, and it is exact

iogpu.wired_limit_mb is in MiB, and MLX reports the ceiling as:
There is no rounding and no decimal conversion in that step. So a sysctl value read as “roughly N thousand MB” produces a ceiling about 7.4% larger than the same number read as decimal GB — because 1024² bytes per MiB is that much more than 1000² bytes per MB. That gap matters because it lands on the wrong side of safety: the ceiling you get is bigger than the one you thought you set, so util × ceiling leaves more allocation headroom than intended while trip — computed against total_ram, untouched by the sysctl — stays exactly where it was. Always re-derive from the value MLX reports back, never from the number you typed into the sysctl.

​
Where this applies

The same arithmetic holds in both serving modes. In standalone mode, one host serves itself and the footprint is a single worker plus whatever else is running on the desktop. In clustered mode, several hosts serve a shared endpoint, and each host is derived independently — a headless desktop-class member can carry a much higher ceiling than a laptop that also runs an interactive session, because the laptop must leave room for the compositor and the user’s own work. Host-specific values — real ceilings, measured footprints, and the utilization figures derived from them — live in the gated operational reference, alongside the rest of the per-host numbers.

Apple Silicon stack

The serving stack these knobs belong to, and the rest of the tuning playbook.

Models and quantization

What sets the footprint side of the invariant in the first place.

Distributed serving

How clustered mode spreads work across more than one host.

nix-ai

The module that generates the serving config these flags land in.