Skip to main content
One envelope schema, every upstream eval tool, one public HF dataset.
mlx-benchmarks is the result-envelope contract and publisher for benchmarking MLX-quantized and locally-hosted LLMs on Apple Silicon. It is the thin glue between upstream evaluation tools (lm-eval, vllm benchmark_serving, agent-framework harnesses) and a single public HuggingFace dataset, with a Gradio viewer on top.

​
What it does

  • Defines envelope v1 in schema.json — the authoritative, versioned contract every published shard validates against.
  • Provides mlx-bench-publish, a CLI that converts raw tool output into the envelope, validates it, and uploads to the HF dataset with content-addressed filenames (data/run-<timestamp>-<git_sha>-<suite>-<model_slug>.parquet).
  • Owns converters for lm-eval, vllm benchmark_serving, and framework-eval (OpenAI / Qwen-Agent / smolagents / ADK).
  • Auto-detects runtime metadata (OS, chip, memory, Python, MLX, lm-eval versions) via detect_system() so envelopes are fully reproducible without hand-curation.
  • Deploys a Gradio viewer to HF Spaces on every main push touching space/.

​
How it fits

A row is only as good as its weight provenance. A serving proxy was found launching a fallback model’s weights under other models’ labels, which put rows in the dataset measuring a model that was never loaded — one of them off by 7x (verified 2026-07-26/27). Prove the loaded checkpoint from the running process before you publish, and treat any historical row that cannot prove it as unverified rather than as a baseline. See Verifying the instrument.

​
The headline metric is cumulative, not decode-only

Cumulative tokens/second = (prompt tokens + completion tokens) ÷ total wall-clock. That is the headline figure; decode-only rate is a secondary diagnostic. Decode-only throughput hides prefill. A model that reads its prompt several times faster delivers a visibly faster answer, and a decode-only number reports none of that gain — so tuning against it tunes for something the user never experiences. Cumulative counts every token the request touched against the time the user actually waited. The two metrics can rank models differently, which is the point. Report both columns; lead with cumulative.

​
Cumulative is meaningless without its regime

Cumulative tokens/second is highly sensitive to prompt size, because prompt tokens are counted in the numerator and prefill is far faster than decode. The same model, on the same hardware, scored 551 cumulative at a 2291-token prompt and 140.9 at a ~190-token prompt. Neither is wrong; they are answers to different questions. So a cumulative figure without its regime is not a fast or slow result — it is an uninterpretable one:
Publish the regime with the number, every time. At minimum: prompt size, max_tokens, and cache state (assert cached_tokens=0 per run if you are not deliberately measuring cache hits). Never compare two cumulative figures taken in different regimes — including your own earlier ones.
A stated regime is not sufficient on its own, because a regime can match on paper and still differ in practice. On 2026-08-14 a candidate read 37.1 cumulative against the incumbent’s recorded 115.2 — a regime that looked identical on every published field. Re-measuring the incumbent with the same harness, in the same session put it at 138.8, not 115.2: the harness ran hot, and the naive comparison had understated the real gap by roughly 20%. It happened to understate; it could as easily have manufactured a win.
Re-measure the incumbent alongside the candidate, every time. A control run costs minutes and is the only thing that separates a model effect from a harness effect. Also match the probe’s concurrency to the endpoint’s — a probe wider than the deployed decode concurrency collects HTTP 429 responses and silently averages the failures into the aggregate.
Decode-only is the most regime-stable of the three rates, which makes it the honest basis for a cross-model comparison even though cumulative is the headline. When two numbers disagree about which model is faster, check whether they were taken in the same regime before you believe either.

​
A worked comparison, one stated regime

Isolated workers with loaded weights proven from the running command line, one model resident at a time, never through a proxy. Identical regime for all four rows: a unique filler prompt of 170–190 tokens with cached_tokens=0 asserted per run, max_tokens=300, streaming with usage included. Cumulative is the median of three runs, with the observed range in brackets. All four returned a genuine tool_calls array with finish_reason: "tool_calls" and correctly parsed arguments. The reasoning-first model is the cautionary row: it is competitive on every rate and never exits thinking at max_tokens=300 — zero answer characters across all three runs, against 1022–1193 characters of reasoning. Fast, and produces no answer. A throughput table cannot see that; a tool-call and answer-length assertion can. Measure what the caller receives, not only how quickly tokens arrive.

​
Getting started

1

Bring up the inference stack

From the nix-darwin flake: darwin-rebuild switch --flake .. This starts vllm-mlx + llama-swap on localhost:11434 via nix-ai. Or run vllm-mlx serve directly if you’re not on the Nix stack.
2

Install and authenticate

git clone https://github.com/JacobPEvans/mlx-benchmarks && cd mlx-benchmarks && uv sync. Then export HF_TOKEN=... with write scope on the dataset namespace.
3

Run a smoke benchmark

Point lm-eval at the local endpoint:
4

Publish (dry-run first)

.venv/bin/mlx-bench-publish ./run-output/<model-dir>/results_*.json --kind lm-eval --suite reasoning --dry-run validates the envelope locally against schema.json. Drop --dry-run to push to the HF dataset.
5

View results

Open the HF Space viewer — it auto-loads every published shard. Or cd space && python app.py for a local copy.

nix-ai

Packages the inference stack: vllm-mlx LaunchAgent, llama-swap, MLX module derivations. Where models actually run.

nix-darwin

macOS host config. Composes nix-ai into the system flake so benchmarks have a reproducible environment.

ai-assistant-instructions

Model routing + permission policy. Tells AI clients which models to benchmark.

Local LLM

The serving stack, tuning, and model strategy these benchmarks measure.

Source on GitHub

Schema, publisher, converters, full README, docs/architecture.md.