mlx-serve: Native LLM Inference for Apple Silicon
Run any LLM on your Mac with OpenAI, Anthropic & Ollama compatible APIs, no Python, and faster than LM Studio.
Evidence shows: SECURITY.md clearly states no authentication, default bind to 127.0.0.1, file tools confined to working directory, but shell commands are not confined; MLX Core has per-tool approvals and sandbox option, but no details on default enforcement or user confirmation. Data flow transparency is limited; README mentions FAQ on data location but no content. Sensitive data handling not detailed. Dependency security: CI pins Zig and mlx versions, but no evidence of vulnerability scanning. External effects: agent mode can execute shell commands, but SECURITY.md warns only for trusted models. Rollback mechanism not mentioned. Source attribution is good; README acknowledges third-party code and includes NOTICE. Deductions: lack of default least privilege, unclear user confirmation, insufficient data flow and sensitive data details, insufficient dependency security evidence, external effects control relies on user discretion, no rollback mechanism.
Evidence shows: README and test scripts are consistent; feature list matches test coverage, but no actual run results. Dependency availability: CI scripts pin Zig, mlx, llama.cpp versions, but no guarantee of dependency availability. Failure messages: test scripts have error output, but server error handling details not provided. Deductions: static review cannot verify actual runs, insufficient failure message evidence.
Evidence shows: README clearly targets developers on Apple Silicon, scenarios include chat, agent, media generation. Capability boundaries are described in SECURITY.md and README, such as no authentication, local only. Trigger precision: tool calling tests cover various scenarios, but no precise description of trigger conditions. Environment fit: explicitly requires macOS 26.2+ and Apple Silicon, provides Homebrew and source build. Deductions: insufficient trigger precision evidence, environment fit limited to specific hardware.
Evidence shows: README is well-structured, provides doc links and FAQ. Install notes are detailed, including Homebrew and source build. Naming stability: version numbers appear in README and CI, but no naming conventions. Examples and FAQ are rich, but FAQ content not provided. Known limitations are mentioned in SECURITY.md and README, such as no authentication, local only. License is MIT, but metadata is NOASSERTION, and LICENSE file includes third-party license notices. Changelog exists as CHANGELOG.md, but content not provided. Maintenance responsibility: SECURITY.md and CI exist, but maintainer not clearly identified. Deductions: insufficient naming stability evidence, changelog content not provided, maintenance responsibility unclear.
Evidence shows: Output usability: provides multiple API compatibility and media generation; test scripts verify basic functionality. Marginal value: comparison table shows advantages over LM Studio etc., but no independent verification. Cost-benefit: free and open source, but no performance data or resource consumption details. Deductions: performance claims unverified, insufficient cost-benefit evidence.
Evidence shows: Claims in README have doc links, but no specific data sources. Cross-source corroboration: test scripts and CI provide some verification, but no independent sources. Fact-inference separation: README distinguishes performance claims and speculation, but not clearly marked. Deductions: claims lack traceable data, insufficient cross-source corroboration, unclear fact-inference separation.
- Static review cannot verify actual runs; all performance claims and feature availability are untested independently.
- SECURITY.md explicitly states no authentication, local use only; do not expose to network.
- Agent mode can execute shell commands without workspace confinement; use with caution.
- License metadata is NOASSERTION, actual is MIT, but be aware of third-party component licenses.
What does this agent do, and when should you use it?
mlx-serve is a native Zig inference server for Apple Silicon that runs both MLX-format models and the entire GGUF universe from Hugging Face, covering architectures like Qwen, Llama, Mistral, Gemma, DeepSeek V4 Flash, and more. It exposes OpenAI-compatible, Anthropic-compatible, and Ollama-compatible HTTP APIs on the same port, allowing tools like Claude Code, OpenAI SDK, Cursor, and Open WebUI to connect seamlessly. Beyond text, it offers native MLX-based generation for images, video, music, speech (including voice cloning), and 3D models. The repository also includes MLX Core, a macOS menu-bar app with chat, agent mode, MCP tool calling, and model management, all licensed under MIT.
It runs a local HTTP server on port 11234 that serves endpoints like /v1/chat/completions, /v1/messages, and /api/chat, supporting streaming, tool calls, JSON-schema constrained decoding, vision, and reasoning content. It also provides media endpoints (/v1/images, /v1/audio, /v1/video, /v1/3d) for generating images, speech, video, and 3D models natively on MLX. Features like speculative decoding (PLD, drafter, MTP), continuous batching, KV-cache quantization, and prefix caching optimize speed and concurrency. Developers can interact via CLI commands (mlx-serve run, pull, serve) or use the MLX Core app for model browsing and one-click launching of coding agents like Claude Code. It also supports LAN model sharing via Bonjour for multi-Mac setups.
- A macOS developer on Apple Silicon wants a local LLM backend for coding assistants like Claude Code, Cursor, or aider without cloud costs or latency.
- A researcher needs to quickly test a variety of MLX and GGUF models offline, including large ones like DeepSeek V4 Flash, on a powerful Mac.
- A content creator wants to generate images, videos, music, or voice clones locally for creative projects, avoiding per-use API fees.
- A team of developers wants to share a single Mac's models across the office via LAN, reducing redundant downloads and storage.
- An existing Ollama user wants to switch to a faster, more capable engine without changing their current tools, by only updating the base URL.
What are this agent's strengths and limitations?
- Supports both MLX and GGUF models, providing broad model compatibility without Python dependencies.
- Implements four API surfaces (OpenAI, Anthropic, Ollama, and native Responses with WebSockets) on one port.
- Performance optimizations yield ~26% faster decode than LM Studio on identical MLX weights (per internal benchmarks).
- Bundles speculative decoding, continuous batching, and KV-cache quantization, delivering lower latency for interactive coding.
- All-in-one solution that also handles image, video, audio, and 3D generation, reducing need for separate tools.
- Hard requirement for macOS 26.2+ and Apple Silicon; no support for Intel Macs or other OSs.
- Heavy workloads may require substantial RAM (e.g., 128GB for largest models), limiting usability on lower-spec machines.
- Advanced features rely on specialized kernels and MTP support that may not be available on older Apple Silicon chips.
- Media generation capabilities are newer and may have quality gaps compared to dedicated commercial services.
- The project is rapidly evolving; adoption may require frequent updates and some debugging.
How do you install or deploy this agent?
Requirements: macOS 26.2+ on Apple Silicon. Recommended via Homebrew: run 'brew tap ddalcu/mlx-serve https://github.com/ddalcu/mlx-serve' then 'brew install --cask mlx-core' (app) or 'brew install mlx-serve' (CLI). Alternatively, download the signed/notarized MLX Core.app from the latest release and drag it to Applications.
How do you use this agent?
Launch MLX Core and use the model browser to download a model; the server auto-starts at http://localhost:11234. For CLI, run 'mlx-serve run gemma4' to download & run Gemma 4 E4B interactively. To use with Claude Code, use 'mlx-serve launch claude-code' or configure your agent's base URL to http://localhost:11234. For OpenAI-API clients, set the base URL to http://localhost:11234/v1. For Ollama tools, point them to the same host.
How does this agent compare with similar options?
The README includes a comparison table with LM Studio, Ollama, and mlx-lm. Unlike Ollama, mlx-serve natively runs MLX weights and offers Anthropic/Ollama API compatibility; it also adds image/video/music generation, which neither LM Studio nor Ollama provides. Benchmarks claim a 26% decode speed advantage over LM Studio on identical MLX weights.