Backends and tool calling

The fast server isn’t the one with the biggest tok/s number. It’s the one that still emits a valid tool call on the fortieth turn of an agent loop. Any OpenAI-compatible server can answer a chat request. The ones that matter for agentic work hold up under sustained, structured, multi-turn pressure. That means continuous batching, a paged KV cache, and a tool-call parser that doesn’t drift. That was the reasoning for running vllm-mlx.

​

The landscape

​

Why vllm-mlx

Three properties decide it:

1. Prefix + paged KV cache. Multi-turn and tool-loop workloads re-send a growing, mostly unchanged context. Reusing the already-prefilled prefix instead of re-running it is the biggest real-world speedup there is. It’s what makes a local model tolerable as an agent backend.
2. Continuous batching. Concurrent callers fold into one GPU forward pass instead of serializing.
3. OpenAI-shaped tool calling. Every caller on the workstation already speaks the OpenAI API, so the model is a drop-in for the cloud providers in the same router.

​

Rapid-MLX: evaluate, don’t switch on hype

Rapid-MLX is real, actively developed, and, importantly, a hardened derivative of vllm-mlx, not a rival engine. Its headline speed multipliers are author-reported and should be discounted. Its genuinely interesting feature is tool-call auto-repair: it detects malformed tool output and reshapes it back into a valid tool_calls structure. It also adds a prompt cache. Because it shares DNA with the current backend, the right move is a scoped A/B on tool-calling reliability under multi-round load. It’s not a swap chasing a tokens-per-second headline.

​

Tool calling is the real failure mode

The number-one way a local agent breaks isn’t speed. It’s a tool call that doesn’t parse. Two things cause most of it:

• Parser mismatch. The server’s tool-call parser has to match the model’s chat template. The wrong parser produces correct-looking JSON that the server mangles, or silently drops the call. This stack defaults to the hermes parser, which in vllm-mlx reads the <tool_call> XML its resident Qwen-family models emit; a different model family needs its own parser.
• Quantization drift under long loops. A 4-bit model can format tool calls perfectly for the first several turns and then start emitting subtly invalid structure deep into an agentic trace. This is where auto-repair earns its keep.
• No parser at all for the model’s format. This is the one that gets misdiagnosed as a bad model. A large open-weights model emitted its own channel markup as raw text in the assistant content field, with finish_reason: stop and no tool_calls array. The call was semantically correct and the serving layer simply dropped it, because the server ships no parser for that format (verified 2026-07-26/27). It leaks into ordinary completions too. This is the same failure class as an agent runtime leaking raw <function=…> markup into a summary: the reasoning was fine, the plumbing lost it.

Speed has a second dimension here. A model that does return valid structured calls can still burn several hundred reasoning tokens before the first one. That was measured at roughly 400 on one candidate. It is latency an agent pays on every single action, not once per conversation. It belongs in the tool-calling verdict rather than in the raw tokens-per-second column. A parser gap also cuts the other way: writing the missing parser can unlock a model that the plumbing, not the model, was dropping. Check whether the failure is the model or the server before you retire the model. That is what happened with the channel-markup case described earlier. The parser was written as a declarative patch rather than the model being written off.

​

Offloading to RAM and NVMe

Offloading can extend usable model capacity on small-VRAM GPUs. A runtime can keep active MoE experts in a GPU cache while storing the wider expert set in system RAM. For PLE or embedding tables, llama.cpp supports CPU tensor overrides and lazy mmap-backed row reads. Strata accepts a separate PLE GGUF and reads it directly from SSD or through mmap. See the llama.cpp server options for --moe-cache-mib, --override-tensor, and --lazy-mode. Strata documents its PLE input and I/O options. These placements can clearly help models run on smaller GPUs. Speed depends on cache hits, page faults, storage latency, and headroom for runtime buffers and the KV cache. One measured configuration stalled when a 51.2 GB PLE table competed for a 48 GiB RAM budget. A mapped file does not make storage latency equivalent to RAM.

​

Related