Backends and tool calling

The fast server isn’t the one with the biggest tok/s number. It’s the one that still emits a valid tool call on the fortieth turn of an agent loop.

Any OpenAI-compatible server can answer a chat request. The ones that matter for agentic work are the ones that hold up under sustained, structured, multi-turn pressure — continuous batching, a paged KV cache, and a tool-call parser that doesn’t drift. That’s why this stack runs vllm-mlx.

​

The landscape

​

Why vllm-mlx

Three properties decide it:

1. Prefix + paged KV cache. Multi-turn and tool-loop workloads re-send a growing, mostly-unchanged context. Reusing the already-prefilled prefix instead of re-running it is the biggest real-world speedup there is — and it’s what makes a local model tolerable as an agent backend.
2. Continuous batching. Concurrent callers fold into one GPU forward pass instead of serializing.
3. OpenAI-shaped tool calling. Every caller on the workstation already speaks the OpenAI API, so the model is a drop-in for the cloud providers in the same router.

​

Rapid-MLX: evaluate, don’t switch on hype

Rapid-MLX is real, actively developed, and — importantly — a hardened derivative of vllm-mlx, not a rival engine. Its headline speed multipliers are author-reported and should be discounted. Its genuinely interesting feature is tool-call auto-repair: it detects malformed tool output and reshapes it back into a valid tool_calls structure, plus a prompt cache. Because it shares DNA with the current backend, the right move is a scoped A/B on tool-calling reliability under multi-round load — not a swap chasing a tokens-per-second headline.

​

Tool calling is the real failure mode

The number-one way a local agent breaks isn’t speed — it’s a tool call that doesn’t parse. Two things cause most of it:

• Parser mismatch. The server’s tool-call parser has to match the model’s chat template. The wrong parser produces correct-looking JSON that the server mangles, or silently drops the call. This stack defaults to the hermes parser, which in vllm-mlx reads the <tool_call> XML its resident Qwen-family models emit; a different model family needs its own parser.
• Quantization drift under long loops. A 4-bit model can format tool calls perfectly for the first several turns and then start emitting subtly invalid structure deep into an agentic trace. This is where auto-repair earns its keep.
• No parser at all for the model’s format. This is the one that gets misdiagnosed as a bad model. A large open-weights model emitted its own channel markup as raw text in the assistant content field, with finish_reason: stop and no tool_calls array — the call was semantically correct and the serving layer simply dropped it, because the server ships no parser for that format (verified 2026-07-26/27). It leaks into ordinary completions too. This is the same failure class as an agent runtime leaking raw <function=…> markup into a summary: the reasoning was fine, the plumbing lost it.

Speed has a second dimension here. A model that does return valid structured calls can still burn several hundred reasoning tokens before the first one — measured at roughly 400 on one candidate. That is latency an agent pays on every single action, not once per conversation, so it belongs in the tool-calling verdict rather than in the raw tokens-per-second column. A parser gap also cuts the other way: writing the missing parser can unlock a model that the plumbing, not the model, was dropping. Check whether the failure is the model or the server before you retire the model. That is what happened with the channel-markup case above, and the parser was written as a declarative patch rather than the model being written off.

​

Related