mlx-serve: Native LLM Inference for Apple Silicon
Run any LLM on your Mac with OpenAI, Anthropic & Ollama compatible APIs, no Python, and faster than LM Studio.
- Source repo
- ddalcu/mlx-serve
- Stars
- ★ 2k
- Last updated
- today
- License
- NOASSERTION
- Primary language
- Zig
- FA score
- 48/100 · Major gaps
At a glance
- How it runs
- Works with
- Platform-specificClaude Code · OpenAI API · Claude API
- Cost
- Free, no paid service needed
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- A macOS developer on Apple Silicon wants a local LLM backend for coding assistants like Claude Code, Cursor, or aider without cloud costs or latency.
- Not a fit if
- Users on Intel Macs or non-macOS systems
- Machines with under 16GB RAM; largest models need 128GB
- Teams needing production-grade media generation quality
- Source review
- 48/100 · Major gaps 1 safety controls not found
What does this agent do, and when should you use it?
mlx-serve is a native Zig inference server for Apple Silicon that runs both MLX-format models and the entire GGUF universe from Hugging Face, covering architectures like Qwen, Llama, Mistral, Gemma, DeepSeek V4 Flash, and more. It exposes OpenAI-compatible, Anthropic-compatible, and Ollama-compatible HTTP APIs on the same port, allowing tools like Claude Code, OpenAI SDK, Cursor, and Open WebUI to connect seamlessly. Beyond text, it offers native MLX-based generation for images, video, music, speech (including voice cloning), and 3D models. The repository also includes MLX Core, a macOS menu-bar app with chat, agent mode, MCP tool calling, and model management, all licensed under MIT.
It runs a local HTTP server on port 11234 that serves endpoints like /v1/chat/completions, /v1/messages, and /api/chat, supporting streaming, tool calls, JSON-schema constrained decoding, vision, and reasoning content. It also provides media endpoints (/v1/images, /v1/audio, /v1/video, /v1/3d) for generating images, speech, video, and 3D models natively on MLX. Features like speculative decoding (PLD, drafter, MTP), continuous batching, KV-cache quantization, and prefix caching optimize speed and concurrency. Developers can interact via CLI commands (mlx-serve run, pull, serve) or use the MLX Core app for model browsing and one-click launching of coding agents like Claude Code. It also supports LAN model sharing via Bonjour for multi-Mac setups.
- A macOS developer on Apple Silicon wants a local LLM backend for coding assistants like Claude Code, Cursor, or aider without cloud costs or latency.
- A researcher needs to quickly test a variety of MLX and GGUF models offline, including large ones like DeepSeek V4 Flash, on a powerful Mac.
- A content creator wants to generate images, videos, music, or voice clones locally for creative projects, avoiding per-use API fees.
- A team of developers wants to share a single Mac's models across the office via LAN, reducing redundant downloads and storage.
- An existing Ollama user wants to switch to a faster, more capable engine without changing their current tools, by only updating the base URL.
How do you install or deploy this agent?
Requirements: macOS 26.2+ on Apple Silicon. Recommended via Homebrew: run 'brew tap ddalcu/mlx-serve https://github.com/ddalcu/mlx-serve' then 'brew install --cask mlx-core' (app) or 'brew install mlx-serve' (CLI). Alternatively, download the signed/notarized MLX Core.app from the latest release and drag it to Applications.
How do you use this agent?
Launch MLX Core and use the model browser to download a model; the server auto-starts at http://localhost:11234. For CLI, run 'mlx-serve run gemma4' to download & run Gemma 4 E4B interactively. To use with Claude Code, use 'mlx-serve launch claude-code' or configure your agent's base URL to http://localhost:11234. For OpenAI-API clients, set the base URL to http://localhost:11234/v1. For Ollama tools, point them to the same host.
What are this agent's strengths and limitations?
- Supports both MLX and GGUF models, providing broad model compatibility without Python dependencies.
- Implements four API surfaces (OpenAI, Anthropic, Ollama, and native Responses with WebSockets) on one port.
- Performance optimizations yield ~26% faster decode than LM Studio on identical MLX weights (per internal benchmarks).
- Bundles speculative decoding, continuous batching, and KV-cache quantization, delivering lower latency for interactive coding.
- All-in-one solution that also handles image, video, audio, and 3D generation, reducing need for separate tools.
- Hard requirement for macOS 26.2+ and Apple Silicon; no support for Intel Macs or other OSs.
- Heavy workloads may require substantial RAM (e.g., 128GB for largest models), limiting usability on lower-spec machines.
- Advanced features rely on specialized kernels and MTP support that may not be available on older Apple Silicon chips.
- Media generation capabilities are newer and may have quality gaps compared to dedicated commercial services.
- The project is rapidly evolving; adoption may require frequent updates and some debugging.
How does this agent compare with similar options?
The README includes a comparison table with LM Studio, Ollama, and mlx-lm. Unlike Ollama, mlx-serve natively runs MLX weights and offers Anthropic/Ollama API compatibility; it also adds image/video/music generation, which neither LM Studio nor Ollama provides. Benchmarks claim a 26% decode speed advantage over LM Studio on identical MLX weights.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| mlx-serve: Native LLM Inference for Apple Silicon This agent | 48 · Major gaps | Desktop appFree | ★ 2k | today | Zig | Claude Code · OpenAI API · Claude API |
| ReadmeAI - AI-Powered README Generator | 51 · Major gaps | CLIFree + model costs | ★ 3k | 2mo ago | Python | OpenAI API · Claude API |
| Webwright Browser Coding Agent | 66 · Some gaps | CLIFree + model costs | ★ 6.1k | 17d ago | Python | Codex · Claude Code · OpenAI API · Claude API |
| hax | 62 · Some gaps | CLIFree + model costs | ★ 932 | 1d ago | C | OpenAI API · Claude API |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows: SECURITY.md clearly states no authentication, default bind to 127.0.0.1, file tools confined to working directory, but shell commands are not confined; MLX Core has per-tool approvals and sandbox option, but no details on default enforcement or user confirmation. Data flow transparency is limited; README mentions FAQ on data location but no content. Sensitive data handling not detailed. Dependency security: CI pins Zig and mlx versions, but no evidence of vulnerability scanning. External effects: agent mode can execute shell commands, but SECURITY.md warns only for trusted models. Rollback mechanism not mentioned. Source attribution is good; README acknowledges third-party code and includes NOTICE. Deductions: lack of default least privilege, unclear user confirmation, insufficient data flow and sensitive data details, insufficient dependency security evidence, external effects control relies on user discretion, no rollback mechanism.
Evidence shows: README and test scripts are consistent; feature list matches test coverage, but no actual run results. Dependency availability: CI scripts pin Zig, mlx, llama.cpp versions, but no guarantee of dependency availability. Failure messages: test scripts have error output, but server error handling details not provided. Deductions: static review cannot verify actual runs, insufficient failure message evidence.
Evidence shows: README clearly targets developers on Apple Silicon, scenarios include chat, agent, media generation. Capability boundaries are described in SECURITY.md and README, such as no authentication, local only. Trigger precision: tool calling tests cover various scenarios, but no precise description of trigger conditions. Environment fit: explicitly requires macOS 26.2+ and Apple Silicon, provides Homebrew and source build. Deductions: insufficient trigger precision evidence, environment fit limited to specific hardware.
Evidence shows: README is well-structured, provides doc links and FAQ. Install notes are detailed, including Homebrew and source build. Naming stability: version numbers appear in README and CI, but no naming conventions. Examples and FAQ are rich, but FAQ content not provided. Known limitations are mentioned in SECURITY.md and README, such as no authentication, local only. License is MIT, but metadata is NOASSERTION, and LICENSE file includes third-party license notices. Changelog exists as CHANGELOG.md, but content not provided. Maintenance responsibility: SECURITY.md and CI exist, but maintainer not clearly identified. Deductions: insufficient naming stability evidence, changelog content not provided, maintenance responsibility unclear.
Evidence shows: Output usability: provides multiple API compatibility and media generation; test scripts verify basic functionality. Marginal value: comparison table shows advantages over LM Studio etc., but no independent verification. Cost-benefit: free and open source, but no performance data or resource consumption details. Deductions: performance claims unverified, insufficient cost-benefit evidence.
Evidence shows: Claims in README have doc links, but no specific data sources. Cross-source corroboration: test scripts and CI provide some verification, but no independent sources. Fact-inference separation: README distinguishes performance claims and speculation, but not clearly marked. Deductions: claims lack traceable data, insufficient cross-source corroboration, unclear fact-inference separation.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Static review cannot verify actual runs; all performance claims and feature availability are untested independently.
- SECURITY.md explicitly states no authentication, local use only; do not expose to network.
- Agent mode can execute shell commands without workspace confinement; use with caution.
- License metadata is NOASSERTION, actual is MIT, but be aware of third-party component licenses.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
v26.10.3 v26.10.3 — New Models, Transcripts, RDMA, Faster
Release notes excerpt
Highlights - Chat on the Neural Engine. Dense Qwen models, from Qwen 3 to Qwen3.8, can run on the Neural Engine instead of the GPU, which leaves the GPU free for images, video or another model. Turn on Neural Engine in the model's settings; it is text only, and the first load compiles the model for a few minutes. …View on GitHub ↗ -
v26.10.2 v26.10.2 — Songs, Films & Sound Effects - The Newest Frontier Models - Linux - Rock Solid Under Load
Release notes excerpt
Click for video: [](https://www.youtube.com/watch?v=ffYDzIaR_vU) <img width="1411" height="814" alt="image" src="https://github.com/user-attachments/assets/4eacea5b-81e8-4899-b5d9-911658946934" /> <img width="1665" height="1035" alt="image" src="https://github.com/user-attachments/assets/5f1cb9d3-aff0-4566-8bfa-6210871de81e" /> <img width="1220" height="697" alt="image" src="https://github.com/user-attachments/assets/60bf1696-45a0-45c0-827c-e1a3cf848979" /> Highlights …View on GitHub ↗ -
v26.10.1 v26.10.1 - Speed Across the Board - GGUF on Our MLX Engine - Sushi Flash Next EXL3 🍣
Release notes excerpt
Highlights - Faster across the board, with identical results. From Gemma 4 E4B to Qwen3.8 27B and Flash Next, models decode up to 66% faster and read prompts up to 51% faster, and the words that come out are exactly the same as in 26.9.6 (checked on 18 models: Gemma 4, Llama, Mistral, Qwen, Muse, LFM2.5, Ling, Bonsai and more). Measured against the shipped 26.9.6 in the same session, arms alternating, llmprobe taking every number; decode is the headline tokens per second, then the context rungs, prefill last: | Model | Mac | Decode | Prefill | |---|---|---|---| …View on GitHub ↗