Install
Three ways to get the mlx-serve command:
| Method | How |
|---|
| From the app | The MLX Core welcome screen has a one-click Install button that puts mlx-serve on your PATH — into ~/.local/bin or ~/bin if you have one (no password), else /usr/local/bin after a single admin prompt. Never touches your shell config. New in v26.7.2. |
| Homebrew | brew tap ddalcu/mlx-serve https://github.com/ddalcu/mlx-serve && brew install mlx-serve |
| From source | zig build -Doptimize=ReleaseFast (needs brew install mlx-c webp). |
Commands
# Download if needed, serve, and chat right here
mlx-serve run gemma4
# Download a model into ~/.mlx-serve/models
mlx-serve pull qwen3.6:27b
# Show downloaded models
mlx-serve list
# Headless server over everything you've pulled — loads on demand by name
mlx-serve serve
| Command | What it does |
|---|
run <model> | Pull-if-missing, start the server, and open an interactive chat REPL in the terminal. Ctrl-C leaves the server behind nothing — it exits with you. |
pull <model> | Resumable download into ~/.mlx-serve/models/<org>/<repo> — interrupted pulls continue where they left off. |
list | Everything on disk, with sizes. |
serve | Serve the whole library on demand: any API request naming a pulled model loads it on the fly. This is the drop-in Ollama replacement mode — add --port 11434 and Ollama clients connect unchanged. |
Model names
Anywhere a <model> is accepted: a short alias (gemma4, qwen3.6:27b — sizes pick quantized builds curated for Apple Silicon), a HuggingFace org/repo, or an hf.co/org/repo URL. GGUF files work too — they route automatically to the embedded llama.cpp engine (DeepSeek-V4-Flash quants to the ds4 engine).
Flag reference
From mlx-serve --help, grouped. Flags compose with serve and run.
Core serving
| Flag | Meaning |
|---|
--model <dir> | Path to a model directory (alternative to the pulled-library workflow). |
--serve | HTTP server mode (with --model; the serve subcommand implies it). |
--host <ip> / --port <n> | Bind address and port (defaults 0.0.0.0 : 11234). |
--model-dir <dir> | Directory of models to discover at startup — they appear in /v1/models and load on demand. |
--ctx-size <n> | Max context length (default: auto-budgeted from the model max and available GPU memory). |
--timeout <n> | Stall timeout: abort after n seconds without a new token (default 300, 0 = off). A request that keeps generating never times out. |
--log-level <lvl> | error · warn · info (default) · debug. |
--log-file <path> | Persist the server log. Default ~/.mlx-serve/logs/mlx-serve-<port>.log (32 MB, rotating, one file per port); off disables. This is the file to read after a crash — the app's in-memory buffer dies with the process. |
Observability & access
| Flag | Meaning |
|---|
--metrics | Serve Prometheus metrics at GET /metrics (plus /metrics.json) and add a live panel to the index page — tokens/sec, requests in flight, time-to-first-token, cache hit rate, GPU load and memory. Names follow the vllm: convention, so stock Grafana dashboards work. Opt-in; zero cost when off. |
--api-key <token> | Require this key on every request from another machine — the OpenAI, Anthropic, and Ollama APIs, the index page, and the metrics endpoints. Accepts Authorization: Bearer, x-api-key, HTTP Basic (key as the password), or ?api_key=. Loopback stays trusted and /health stays open, so local tools need no credentials. Unset means no auth. |
Sampling & generation defaults
| Flag | Meaning |
|---|
--temp / --top-p / --top-k | Serve-mode defaults for requests that omit them (request body → these flags → the model's generation_config.json → engine defaults). |
--max-tokens <n> | Generation cap for --prompt mode (default 100). |
--reasoning-budget <n> | Max thinking tokens per request on reasoning models. |
--prompt <text> / --stream | One-shot offline generation, no server. |
--no-vision | Skip loading the vision encoder (saves memory on multimodal checkpoints). |
--no-safety | Disable the image-generation NSFW filter (or per request with "safety": false). |
Speed: speculative decoding
All three paths produce output equivalent to plain decoding — see the deep dive.
| Flag | Meaning |
|---|
--pld / --no-pld | Prompt Lookup Decoding — on by default, big wins on echo-heavy work (code edits, RAG, agents). --pld-draft-len, --pld-key-len tune it. |
--drafter <dir> | Gemma 4 assistant-drafter checkpoint; --draft-block-size overrides the per-target auto block size. |
--no-mtp / --mtp-depth <n> | Qwen native multi-token-prediction head (auto-loaded when the checkpoint ships one; priority MTP > drafter > PLD). |
Memory: KV cache & residency
| Flag | Meaning |
|---|
--kv-quant <mode> | KV-cache quantization: off (default) · 4 · 8 · turbo2 · turbo4 (Hadamard-rotated). Up to ~4× longer contexts in the same memory; per-request override via the kv_quant body field. |
--kv-attn-mode | dense (default) or fused — attention path for quantized KV. |
--prefix-cache-entries / --prefix-cache-mem | Hot prompt-prefix cache: entry count and byte budget (default 2 GB, LRU-evicted). |
--prefix-cache-disk <n>{KB,MB,GB} | SSD tier for the prefix cache (default off). Prefixes persist under ~/.mlx-serve/kv-cache and are restored across restarts and memory evictions instead of recomputed — a 40-second re-read of a long chat becomes under 3 seconds, byte-identical. Opt-in because it can hold many GB; e.g. 10GB. 0/off disables. |
--max-resident-models <n> | How many models stay loaded at once (default 3, LRU). |
--max-resident-mem <n>|auto | Byte cap across all loaded models (default: 80% of the MLX wired limit). |
--idle-evict-secs <n> | Evict idle models after n seconds (default: off). |
--max-concurrent <n> | Request submit-queue size for concurrent batched decoding. |
--skip-mem-preflight | Bypass the conservative free-RAM check before a model load. |
--tokenize-cache-entries <n> | Per-model LRU of chat-template render + tokenize results (default 4). |
GGUF engines
| Flag | Meaning |
|---|
--engine auto|ds4|llama | Engine for .gguf inputs only — auto reads the file's architecture metadata (safetensors always run the native MLX engine). |
--llama-cache-entries <n> | Resident KV sessions for llama.cpp-served GGUFs (default 4). |
--ssd-streaming | DeepSeek-V4-Flash: stream expert weights from SSD instead of holding the full model in RAM. |
Recipes
# Drop-in Ollama replacement — same port, your Ollama apps connect unchanged
mlx-serve serve --port 11434
# Claude Code against your local model
mlx-serve run gemma4 # in one terminal
export ANTHROPIC_BASE_URL=http://localhost:11234
export ANTHROPIC_API_KEY=unused
claude # in another
# Long-context agent box: 8-bit KV cache + tight timeout logging
mlx-serve serve --kv-quant 8 --log-level debug
# Serve a folder of models you manage yourself
mlx-serve --serve --model-dir ~/models --port 11234
The full, always-current list is mlx-serve --help — this page tracks it release to release.