> ## Documentation Index > Fetch the complete documentation index at: https://docs.jacobpevans.com/llms.txt > Use this file to discover all available pages before exploring further. # Verifying the instrument > A model server can serve the wrong weights silently. The two checks that verify nothing, the one check that works, and the 7x measurement error that proves why it matters. > Before you record a benchmark number, prove which weights the process actually > loaded. Nothing in the response tells you. ## How a server ends up serving the wrong weights There are two routes to it. Both were observed in a single deployment; treat the specifics as one estate's experience and the shape as general. ### Mechanism A — a collapse that grafts physical ids as aliases A proxy config generator collapses a multi-model catalog down to **one resident model** to fit a memory budget. Every other entry is disabled — kept intact, each still carrying its own correct launch command, simply not served. Then the generator does one more thing: it grafts every disabled model's **physical identifier** onto the surviving entry as an **alias**, so that "any caller naming any known model still gets an answer." That is the defect. The surviving entry now answers to names that do not identify it. Ask for a 120B model, get the 30B resident — HTTP 200, and a response labelled with the 120B you asked for. In one captured config, 11 of 12 model ids resolved to the same resident. The distinction the generator lost is between two kinds of name: | Kind of name | What it asserts | Repointing it is | | ------------------------------------------------- | -------------------------------- | ---------------------------- | | a **role alias** (`default`, `coding`, `fastest`) | intent — "something suitable" | correct, and the whole point | | a **physical model id** | identity — "these exact weights" | a silent lie | Treating them as interchangeable is what makes the failure invisible. Nothing downstream caught it, because the two obvious checks are not checks. ### Mechanism B — a router alias that falls through instead of failing The second route needs no bad config generator. A router carries an alias pointing at a model the backend does not actually serve. Instead of returning an error, the request **falls through to whatever is served** and comes back 200. This one is worse than Mechanism A in a specific way: nothing anywhere is misconfigured in a way you can look at. The alias is spelled correctly, the backend is healthy, the response is well-formed. The only artifact of the defect is that the answer came from different weights than the name says — which is invisible in the response, by construction. The cost, observed in one deployment: an agent ran on a **different model than its configuration named**, for an unknown period. Nobody noticed, because there was nothing to notice. Every request succeeded. That is the real severity argument. A crash gets fixed in an hour. A silent substitution runs until someone independently verifies the instrument — and "until someone thinks to check" is not a bounded interval. ## Two things that look like verification and verify nothing | What you check | What it actually reports | | --------------------------------------------- | ------------------------------------------------------------------------------- | | the `model` field in an OpenAI-style response | the value echoed back from your request — never read from the loaded checkpoint | | a `/v1/models` listing | the local model cache — not what any running process has loaded | Both hand back exactly what you hoped to see while the server runs something else. A label is not evidence. ## The only check that works Walk from the listening port to the process command line. Two hops, no trust: ```bash theme={null} lsof -i : -sTCP:LISTEN -t # port -> pid ps -p -o command= # pid -> the weights it was launched with ``` If the command line does not name the model you think you are measuring, the number is void. Do this before the run, not after. Run it against the **worker**, never the proxy. A dedicated single-model worker resolves the id it was asked for and returns 404 on an id it does not hold — so once a worker's weights are proven by process inspection, "HTTP 200 plus a matching label" from *that worker* is trustworthy. Put a proxy in front of it and the same 200 and the same label prove nothing again. The guarantee belongs to the worker, not to the response shape. ## The error this hid: a 7x gap that inverted the conclusion Taken through the proxy, a large (80B-class) mixture-of-experts model measured about **72 tok/s**. Re-measured on an isolated worker with the loaded weights proven by process inspection, the same model decoded at **9–11 tok/s** — medians 11.05 and 8.96, replicated across two independent sequences, with prefill at 97–152 tok/s and time-to-first-token at 0.58–0.91 s (verified 2026-07-26/27). A 7x gap. About 72 tok/s is 30B-class speed, and the grafted alias pointed at a 30B-class resident. Storage and memory pressure were both ruled out. The damage did not stay local. Because several grafted ids all resolved to the same resident, the numbers reached a **public dataset attributing one model's throughput to two others** — one wrong measurement became several wrong published rows, each carrying a model name that had never been loaded. A silent substitution does not produce one bad number; it produces as many as there are aliases pointing at the survivor. On the [headline cumulative metric](/tools/mlx-benchmarks#the-headline-metric-is-cumulative-not-decode-only) the corrected model runs 10–13 tok/s — the correction survives either metric, because the defect was never about which rate you report. The wrong number was never implausible, and that is the whole problem. **A plausible number from an unverified instrument is worse than no number**: no number prompts a measurement, while a wrong one ends the inquiry. "It looks about right" is exactly the reasoning that lets a fabricated measurement survive review. So every published result has to carry its weight provenance, and any row that cannot prove which checkpoint was loaded is unverified — not a baseline. ## Three rules that fall out **Never let a name resolve to something it does not identify.** An alias is an assertion that two names mean the same thing. When a generator points a *physical identifier* at a different object so that "every request gets an answer," it has chosen a confident wrong answer over an honest error. Drop the alias, or return an explicit 404 — either beats silence. Convenience routing is for names that express intent, never for names that express identity. The corollary is a debugging one: when this happens, **nothing is corrupt**. The disabled entries were intact the whole time and would serve real weights the moment they were enabled. Only the names were wrong. Look at the routing table before you go looking for damage. **Resolve exactly, or fail loudly — never fall through.** A request for a name the backend does not serve must return an error, not the nearest available weights. Silent fallback converts a configuration bug into a data-quality bug that no amount of downstream checking can recover, because the evidence that anything went wrong is never written down. **Log the resolved physical model next to the requested name.** This whole failure class is defined by being silent, so the cheapest possible fix is to make it speak: one log line per load, or per request, carrying both names. When they differ, that is the entire defect, visible in `grep`. Any serving layer that resolves a name owes you that line — if yours does not emit it, that is the first thing to add, ahead of any other guard, because every other check on this page is something a human has to remember to run. **Mutation-test every new check.** A check nobody has watched fail is not a check. After you add a guard, deliberately reinstate the defect it guards against and confirm the build breaks. The classic dud is `test.sh; touch $out` — the semicolon creates the output file even when the test fails, so the check can never fail. A compatibility check added for the [shell-parser bugs](/local-llm/distributed) was mutation-tested this way: putting either defect back fails the build. ## The general shape of the problem The recurring failure is an instrument returning something indistinguishable from its opposite: | Instrument | Reads as | Actually means | | ------------------------------------------- | ------------------- | ---------------------------------------------- | | a `pgrep` no-match | the process is gone | your pattern is stale | | a proxy 502 or 503 | the backend is dead | the route is broken, or a dependency is | | `nc -z` with stdout redirected | reachable | can report success against an unreachable host | | an `https://` probe to a plain-HTTP backend | the host is down | TLS terminates in front of it | | a cluster membership list | the peers are alive | they are members | | a derived `1` | a hardcoded `1` | only the source line distinguishes them | Two habits cover most of it: **require three independent tools to agree before acting on a negative signal**, and prove a check *can* fail before you trust what it reports. ## Related The envelope and public dataset every published number lands in. The other thing the serving layer can silently drop: a valid tool call. Concurrency measured per model, including where more load makes it worse. Silent-success shell bugs, and debt that a retry only deepens.