Engineering · Release 26.9.5
mlx-serve is a native inference server for Apple silicon, written in Zig on top of MLX's C API. One binary, no Python, more than 20 language model architectures. In release 26.9.5 on an M4 Max it decodes Qwen3.6-35B-A3B at 239 tokens a second, answers a continued chat in 56 ms, and runs four Qwen3.8-27B chats at 119 tokens a second combined, up from 72 before this release. On the same Mac it decodes 1.1 to 3.2 times faster than llama.cpp without speculation on either side, and 3.2 times faster on Qwen3.8-27B with MTP on both.
Every number on this page comes from one Mac, an M4 Max with 128 GB. Decoding a token means streaming the model's weights from unified memory through the GPU, and no engine streams them faster than the memory bus allows. So the whole game on Apple silicon is how many tokens you get out of each read. mlx-serve gets more than one in three ways: it guesses ahead and checks the guesses in one pass, it checks several conversations in that same pass, and it never reads a prompt twice if it already has the answer cached. Everything else in the engine exists to make those three cheap.
One read of the weights
A 27B model at 4 bits is about 15 GB. Reading it once takes tens of milliseconds on an M4 Max, and one token per read tops out near 29 tokens a second in practice. mlx-serve verifies 16 rows per read: four chats, each with the next token plus three guesses.
Five models from the release benchmark, one M4 Max, the fastest configuration each ships with, speculation named where it engaged. Decode is llmprobe's own code-completion prompt, warmup discarded, median of three runs. Prefill is a 2k-token prompt. First-token times are a 1.4k-token conversation sent cold, then sent again with one more message.
| Model, 4-bit | Spec | Decode | Prefill | Cold | Warm | Sooner |
|---|---|---|---|---|---|---|
| Qwen3.6 35B-A3B | MTP | 239 | 1,871 | 850 ms | 68 ms | 12.5× |
| Gemma 4 E4B | PLD | 116 | 1,467 | 990 ms | 53 ms | 18.7× |
| Gemma 4 26B-A4B | PLD | 115 | 1,596 | 929 ms | 56 ms | 16.6× |
| Qwen3.8 Flash-Next* | MTP | 83 | 727 | 2,127 ms | 145 ms | 14.7× |
| Qwen3.8 27B | MTP | 70 | 257 | 6,835 ms | 202 ms | 33.8× |
Release 26.9.5, Apple M4 Max 128 GB, ReleaseFast, llmprobe --bench-only, medians of three. PLD is prompt lookup, MTP is the model's own multi-token-prediction head. Decode and prefill in tokens per second; cold and warm are the time to the first token for the same 1.4k-token conversation. *Flash-Next is a 125B-A6B mixture of experts in a mixed 4/8-bit pack.
mlx-serve is general on purpose: one server runs Gemma, Qwen, Llama, Mistral, DeepSeek, LFM, Nemotron and more, next to image, video, music and speech models. The speed comes from the parts around the math.
Qwen3.8-27B at 4 bits is about 15 GB. The M4 Max moves 546 GB a second, so reading it once takes about 28 ms, and one token per read can never beat about 36 tokens a second. Plain decode measures about 29. mlx-serve serves one chat at 70 and four at 119 on the same weights. None of that is a faster read. It is more tokens kept per read: about 2.4 for one chat with MTP, about 12.6 for four.
That is why this post is mostly about speculation and batching. The trunk forward on a 27B is at parity with every other MLX-based engine we have measured; nobody reads 15 GB faster than the bus. What separates engines is what they do with each read.
Before 26.9.5, four MTP chats on the 27B took turns. Each chat ran its own draft and its own verify, so four chats added up to 72 tokens a second, barely more than one. Now the four lanes draft as rows of one head forward and verify together in one 16-row pass, and every lane is drafted to the full width so the pass never runs half empty.
Three changes made it work. A matrix tile that Apple's M5 runs in hardware turned out to run well as plain shader code on the M4, 11.2 TFLOP/s at 8, 16 and 24 rows, so a 16-row verify costs about three plain reads, not sixteen. The draft head now runs once for all four lanes instead of twelve small serial steps. And past 1k tokens of context each chat attends over its own KV cache instead of the engine padding and stacking all four every step, which is the 26 to 64 at 28k.
An 8-bit KV cache halves the memory a long conversation takes, but reading it used to cost speed: at 32k the 27B decoded at 37 tokens a second with 8-bit KV against 46 with bf16. The packed kernels were spilling registers once a verify step had more than a dozen query rows. The new kernel dequantizes each 32-token page into threadgroup memory and does both attention matmuls with Apple's matmul2d tensor op, straight from our affine cache.
Apple M4 Max, 128 GB, macOS 26. mlx-serve 26.9.5, ReleaseFast. Single-stream rows are tests/bench.sh, which runs llmprobe --bench-only: warmup discarded, median of three, each engine at its fastest shipped configuration with the speculative mode named. The earlier columns in Fig 2 come from the same harness in benchmarks.md. Concurrency and 8-bit KV numbers come from our own harness: greedy, one engine per boot, repeated boots per arm, same prompts on both engines; the 8-bit KV runs used a 36 GB GPU memory limit. llmprobe's own four-stream cell on the 27B reads 86, because it includes the four cold prefills. Speculative numbers vary by prompt, so we compare only same-session, same-methodology cells. The llama.cpp comparison is its own session on 22 September: llama.cpp master f46bc30 built with Metal, llama-server -ngl 99 -fa on -c 65536 -np 4 --kv-unified --jinja (plus --spec-type draft-mtp on the MTP row), against mlx-serve 26.9.5. Its Flash-Next cell is the best of two llama.cpp runs, the default one and one with a 32k context and the prompt cache off, because the 100 GB model left the default run short of memory.
It speaks the OpenAI, Anthropic and Ollama APIs, so Claude Code, Codex, pi and anything else that talks to those can run on your Mac at these speeds.
mlx-serve