Skip to main content
One envelope schema, every upstream eval tool, one public HF dataset.
mlx-benchmarks is the result-envelope contract and publisher for benchmarking MLX-quantized and locally hosted LLMs on Apple Silicon. It is the thin glue between upstream evaluation tools (lm-eval, vllm benchmark_serving, agent-framework harnesses) and a single public HuggingFace dataset, with a Gradio viewer on top.

​
What it does

  • Defines envelope v1 in schema.json: the authoritative, versioned contract every published shard validates against.
  • Provides mlx-bench-publish, a command-line tool that converts raw tool output into the envelope, validates it, and uploads to the HF dataset with content-addressed filenames (data/run-<timestamp>-<git_sha>-<suite>-<model_slug>.parquet).
  • Owns converters for lm-eval, vllm benchmark_serving, and framework-eval (OpenAI / Qwen-Agent / smolagents / ADK).
  • Auto-detects runtime metadata (OS, chip, memory, Python, MLX, lm-eval versions) via detect_system() so envelopes are fully reproducible without hand-curation.
  • Deploys a Gradio viewer to HF Spaces on every main push touching space/.

​
How it fits

A row is only as good as its weight provenance. A serving proxy was found launching a fallback model’s weights under other models’ labels. This put rows in the dataset measuring a model that was never loaded. One of them was off by 7x (verified 2026-07-26/27). Prove the loaded checkpoint from the running process before you publish. Treat any historical row that cannot prove it as unverified rather than as a baseline. See Verifying the instrument.

​
The headline metric is cumulative, not decode-only

Cumulative tokens/second = (prompt tokens + completion tokens) ÷ total wall-clock. That is the headline figure; decode-only rate is a secondary diagnostic. Decode-only throughput hides prefill. A model that reads its prompt several times faster delivers a visibly faster answer, and a decode-only number reports none of that gain. So tuning against it tunes for something the user never experiences. Cumulative counts every token the request touched against the time the user actually waited. The two metrics can rank models differently, which is the point. Report both columns; lead with cumulative.

​
Cumulative is meaningless without its regime

Cumulative tokens/second is highly sensitive to prompt size, because prompt tokens are counted in the numerator and prefill is far faster than decode. The same model, on the same hardware, scored 551 cumulative at a 2291-token prompt and 140.9 at a ~190-token prompt. Neither is wrong; they are answers to different questions. So a cumulative figure without its regime is not a fast or slow result. It is an uninterpretable one:
Publish the regime with the number, every time. At minimum, note prompt size and max_tokens. Also note cache state (assert cached_tokens=0 per run if you are not deliberately measuring cache hits). Never compare two cumulative figures taken in different regimes, including your own earlier ones.
A stated regime is not sufficient on its own, because a regime can match on paper and still differ in practice. On 2026-08-14 a candidate read 37.1 cumulative against the incumbent’s recorded 115.2. That regime looked identical on every published field. Re-measuring the incumbent with the same harness, in the same session put it at 138.8, not 115.2. The harness ran hot, and the naive comparison had understated the real gap by roughly 20%. It happened to understate; it could as easily have manufactured a win.
Re-measure the incumbent alongside the candidate, every time. A control run costs minutes. It’s the only thing that separates a model effect from a harness effect. Also match the probe’s concurrency to the endpoint’s. A probe wider than the deployed decode concurrency collects HTTP 429 responses and silently averages the failures into the aggregate.
Decode-only is the most regime-stable of the three rates. That makes it the honest basis for a cross-model comparison even though cumulative is the headline. When two numbers disagree about which model is faster, check whether they were taken in the same regime before you believe either.

​
Long-context results need two context values

A benchmark’s requested prompt length is not its serving limit. Publish both values so a reader can tell what the worker was configured to admit and what it actually processed:
  • Configured total context: the request budget admitted by the serving profile. It includes input and reserved output.
  • Actual prompt tokens: the input tokens reported by the server for this request.
  • Output reservation: the completion tokens reserved before admission.
  • Actual total context: actual prompt tokens plus the completion tokens returned.
The admission check is always:
For example, a 32k prompt sent to a 192k-enabled profile is still a 32k input measurement. It belongs in a separate window-sensitivity row because the configured 192k limit may change memory allocation or scheduling. It is not a 192k prompt result. Compare it with a 32k prompt under a 32k-enabled profile to answer whether enabling a larger window changes short-request speed.

​
Cache state and campaign results

Cache provenance is several independent facts, not one cache-busted label:
  • Weights: exact model revision and quantization loaded by the worker.
  • Model load: whether the process was already resident or started for the cell.
  • Prompt prefix: whether a shared prompt prefix was reused; cache-cold rows use a unique full prefix and assert zero reported cached prompt tokens when the server exposes that telemetry.
  • Operating-system and evaluation caches: state that can affect loading or evaluation even when prompt-prefix reuse is absent.
Each campaign row also carries a cell status. success contains valid measured rates. failed preserves a completed but invalid run. capacity_gated records a safety refusal. unsupported records a missing backend capability. aborted records a run stopped by a safety guard. not_applicable means the requested window or profile does not exist for that model. None of those status rows is a zero-throughput result. Published results are immutable shards. A new campaign appends a new shard and run-index entry; it does not rewrite a historical row to make a comparison look cleaner. The viewer distinguishes historical evidence from current campaign evidence. Comparisons must name the run, serving profile, context values, concurrency, cache state, and model revision that produced them.

​
Fresh 2026-08 context evidence

The current isolated campaign uses a 64-token output reservation, one request at a time, and a cache-busted full prompt for each measured repetition. It runs four warmed repetitions per successful cell. The first request is archived under readiness with an explicit initial_model_state; it is never included in the warmed median.

​
Clean-reset 2026-08-30 128k matrix

This separate immutable campaign began after a Studio reboot. The production model lifecycle was quiesced before each cell, and the preflight swap value was zero. The server used a 131,072-token window with a 128,000-token prompt and a 64-token output reservation. Each row has four resident repetitions. The cold first request is retained as readiness evidence and is excluded from the warm medians. The clean-reset base and MTP c1 rows show comparable 128k prompt throughput. The c2 result admitted two requests without swap, but its aggregate rate stayed at the c1 rate. In this native MTP configuration, the long-prefill path queued rather than increasing aggregate throughput. The c4 attempt is preserved as transport-limited evidence, not a capacity or quality verdict. Full-attention KV math informed the preflight only. 64 KiB per token per sequence is about 7.816 GiB at 128,000 plus 64 tokens, or 31.266 GiB for four sequences. Hybrid linear-attention state, weights, allocator behavior, and scratch space are outside that calculation, so runtime telemetry remains the fit result. The selected Qwen3.8 profile has a 131k configured window, so 192k is not_applicable, not a zero-speed result.

​
Runtime capability status

The MTP drafter was proven by live completions in an isolated native-MTP c1 profile. It is still not a base-model fallback or a serving default. The base and MTP rows use different serving backends, so their rate difference is not an MTP-only speed multiplier. That runtime status remains separate from the immutable measurement table. A production activation requires its own verified route and adds a labeled MTP row without changing any existing base-model evidence.

​
A worked comparison, one stated regime

Isolated workers with loaded weights proven from the running command line, one model resident at a time, never through a proxy. Identical regime for all four rows: a unique filler prompt of 170–190 tokens with cached_tokens=0 asserted per run, max_tokens=300, streaming with usage included. Cumulative is the median of three runs, with the observed range in brackets. All four returned a genuine tool_calls array with finish_reason: "tool_calls" and correctly parsed arguments. The reasoning-first model is the cautionary row. It is competitive on every rate and never exits thinking at max_tokens=300: zero answer characters across all three runs, against 1022–1193 characters of reasoning. Fast, and produces no answer. A throughput table cannot see that; a tool-call and answer-length assertion can. Measure what the caller receives, not only how quickly tokens arrive.

​
Getting started

1

Bring up the inference stack

From the nix-darwin flake: darwin-rebuild switch --flake .. This starts vllm-mlx + llama-swap on localhost:11434 via nix-ai. Or run vllm-mlx serve directly if you’re not on the Nix stack.
2

Install and authenticate

git clone https://github.com/JacobPEvans/mlx-benchmarks && cd mlx-benchmarks && uv sync. Then export HF_TOKEN=... with write scope on the dataset namespace.
3

Run a smoke benchmark

Point lm-eval at the local endpoint:
4

Publish (dry-run first)

.venv/bin/mlx-bench-publish ./run-output/<model-dir>/results_*.json --kind lm-eval --suite reasoning --dry-run validates the envelope locally against schema.json. Drop --dry-run to push to the HF dataset.
5

View results

Open the HF Space viewer. It auto-loads every published shard. Or cd space && python app.py for a local copy.

nix-ai

Packages the inference stack: vllm-mlx LaunchAgent, llama-swap, MLX module derivations. Where models actually run.

nix-darwin

macOS host config. Composes nix-ai into the system flake so benchmarks have a reproducible environment.

ai-assistant-instructions

Model routing + permission policy. Tells AI clients which models to benchmark.

Local LLM

The serving stack, tuning, and model strategy these benchmarks measure.

Source on GitHub

Schema, publisher, converters, full README, docs/architecture.md.