Before you record a benchmark number, prove which weights the process actually loaded. Nothing in the response tells you.
How a server ends up serving the wrong weights
There are two routes to it. Both were observed in a single deployment; treat the specifics as one estate’s experience and the shape as general.Mechanism A — a collapse that grafts physical ids as aliases
A proxy config generator collapses a multi-model catalog down to one resident model to fit a memory budget. Every other entry is disabled — kept intact, each still carrying its own correct launch command, simply not served. Then the generator does one more thing: it grafts every disabled model’s physical identifier onto the surviving entry as an alias, so that “any caller naming any known model still gets an answer.” That is the defect. The surviving entry now answers to names that do not identify it. Ask for a 120B model, get the 30B resident — HTTP 200, and a response labelled with the 120B you asked for. In one captured config, 11 of 12 model ids resolved to the same resident. The distinction the generator lost is between two kinds of name:
Treating them as interchangeable is what makes the failure invisible. Nothing
downstream caught it, because the two obvious checks are not checks.
Mechanism B — a router alias that falls through instead of failing
The second route needs no bad config generator. A router carries an alias pointing at a model the backend does not actually serve. Instead of returning an error, the request falls through to whatever is served and comes back 200. This one is worse than Mechanism A in a specific way: nothing anywhere is misconfigured in a way you can look at. The alias is spelled correctly, the backend is healthy, the response is well-formed. The only artifact of the defect is that the answer came from different weights than the name says — which is invisible in the response, by construction. The cost, observed in one deployment: an agent ran on a different model than its configuration named, for an unknown period. Nobody noticed, because there was nothing to notice. Every request succeeded. That is the real severity argument. A crash gets fixed in an hour. A silent substitution runs until someone independently verifies the instrument — and “until someone thinks to check” is not a bounded interval.Two things that look like verification and verify nothing
Both hand back exactly what you hoped to see while the server runs something
else. A label is not evidence.
The only check that works
Walk from the listening port to the process command line. Two hops, no trust:The error this hid: a 7x gap that inverted the conclusion
Taken through the proxy, a large (80B-class) mixture-of-experts model measured about 72 tok/s. Re-measured on an isolated worker with the loaded weights proven by process inspection, the same model decoded at 9–11 tok/s — medians 11.05 and 8.96, replicated across two independent sequences, with prefill at 97–152 tok/s and time-to-first-token at 0.58–0.91 s (verified 2026-07-26/27). A 7x gap. About 72 tok/s is 30B-class speed, and the grafted alias pointed at a 30B-class resident. Storage and memory pressure were both ruled out. The damage did not stay local. Because several grafted ids all resolved to the same resident, the numbers reached a public dataset attributing one model’s throughput to two others — one wrong measurement became several wrong published rows, each carrying a model name that had never been loaded. A silent substitution does not produce one bad number; it produces as many as there are aliases pointing at the survivor. On the headline cumulative metric the corrected model runs 10–13 tok/s — the correction survives either metric, because the defect was never about which rate you report. The wrong number was never implausible, and that is the whole problem. A plausible number from an unverified instrument is worse than no number: no number prompts a measurement, while a wrong one ends the inquiry. “It looks about right” is exactly the reasoning that lets a fabricated measurement survive review. So every published result has to carry its weight provenance, and any row that cannot prove which checkpoint was loaded is unverified — not a baseline.Three rules that fall out
Never let a name resolve to something it does not identify. An alias is an assertion that two names mean the same thing. When a generator points a physical identifier at a different object so that “every request gets an answer,” it has chosen a confident wrong answer over an honest error. Drop the alias, or return an explicit 404 — either beats silence. Convenience routing is for names that express intent, never for names that express identity. The corollary is a debugging one: when this happens, nothing is corrupt. The disabled entries were intact the whole time and would serve real weights the moment they were enabled. Only the names were wrong. Look at the routing table before you go looking for damage. Resolve exactly, or fail loudly — never fall through. A request for a name the backend does not serve must return an error, not the nearest available weights. Silent fallback converts a configuration bug into a data-quality bug that no amount of downstream checking can recover, because the evidence that anything went wrong is never written down. Log the resolved physical model next to the requested name. This whole failure class is defined by being silent, so the cheapest possible fix is to make it speak: one log line per load, or per request, carrying both names. When they differ, that is the entire defect, visible ingrep. Any serving layer that
resolves a name owes you that line — if yours does not emit it, that is the first
thing to add, ahead of any other guard, because every other check on this page is
something a human has to remember to run.
Mutation-test every new check. A check nobody has watched fail is not a
check. After you add a guard, deliberately reinstate the defect it guards
against and confirm the build breaks. The classic dud is test.sh; touch $out
— the semicolon creates the output file even when the test fails, so the check
can never fail. A compatibility check added for the
shell-parser bugs was mutation-tested this way:
putting either defect back fails the build.
The general shape of the problem
The recurring failure is an instrument returning something indistinguishable from its opposite:
Two habits cover most of it: require three independent tools to agree before
acting on a negative signal, and prove a check can fail before you trust
what it reports.
Related
Benchmarking
The envelope and public dataset every published number lands in.
Backends & tool calling
The other thing the serving layer can silently drop: a valid tool call.
Apple Silicon stack
Concurrency measured per model, including where more load makes it worse.
Distributed & multi-Mac
Silent-success shell bugs, and debt that a retry only deepens.