Devstral Small 2 on a 64 GB Mac: speed, quality, memory
Mistral AI, released Dec 2025. 24B parameters, dense. 4-bit MLX build, 15.1 GB download. License Apache-2.0.
- HumanEval+ pass@1: 135 of 164 (82.3%), rank 6 of 8
- HumanEval pass@1: 145 of 164 (88.4%)
- Generation speed: 18.1 tokens per second (5 trials, 18.1 to 18.3); prompt 464 tokens per second
- Peak memory: 15.1 GB · power while generating: 26 W · GPU peak 83 °C
Run it yourself
pip install mlx-lm mlx_lm.chat --model mlx-community/Devstral-Small-2-24B-Instruct-2512-4bit
HumanEval/74
Return the list with fewer total characters. If both are equal, return the first list.
It forgot the tie rule. Both lists had 7 characters, and it returned the second one.
Problems it failed
HumanEval/32, HumanEval/54, HumanEval/65, HumanEval/74, HumanEval/75, HumanEval/83, HumanEval/96, HumanEval/99, HumanEval/102, HumanEval/108, HumanEval/115, HumanEval/116, HumanEval/121, HumanEval/126, HumanEval/127, HumanEval/130, HumanEval/132, HumanEval/145, HumanEval/160
Raw files: speed.log, evaluate.log, eval_results.json