How we test
Speed: mlx_lm.benchmark, 1,024 prompt tokens, 512 generated, 5 trials. Quality: EvalPlus HumanEval+ v0.1.10, greedy, pass@1, scored in a sandbox with no network. Power: macmon while the GPU is generating. Every review video reruns the tests on camera.
Corrections
- 2026-11-03: Speed remeasured on a clean machine. Part of the first batch ran while macOS had 18 to 37 GB in swap. Devstral Small 2 read 8.1 tok/s with the GPU near 19% busy; that value is withdrawn. After a reboot every speed benchmark was rerun alone on the machine, and only those values are published. gpt-oss-20b moved from 79.1 to 82.6 tok/s; the others moved by 2% or less. Quality scores do not depend on speed and were kept.
- 2026-11-03: Server crash during code generation. The MLX server ran out of GPU memory twice. Generation was resumed with EvalPlus resume (finished answers kept) under a watchdog that restarts the server. All 164 answers were produced for every model.
- 2026-11-02: E2B and E4B speed reruns. The first E2B speed run was disturbed by other work on the Mac (119.6 tok/s, spread 104 to 133). Clean reruns replaced it.
- 2026-11-02: EvalPlus scored 0.000 on macOS. macOS rejects RLIMIT_AS, so every test worker crashed before running. The zero was a harness failure, fixed with EVALPLUS_MAX_MEMORY_BYTES=-1; tests and timeouts unchanged.
- 2026-11-02: Token limit raised to 8,192. EvalPlus defaults to 768 new tokens, which cut answers short. Raised for every model; prompts, tests and scoring unchanged.