oMLX: Open-Source MLX Inference Server Revolutionizes Local LLMs on Apple Silicon

oMLX: Open-Source MLX Inference Server Revolutionizes Local LLMs on Apple Silicon

A quietly groundbreaking development in the local LLM ecosystem has emerged from the Reddit community of Apple Silicon enthusiasts: oMLX, an open-source inference server designed specifically for macOS devices with Apple Silicon chips. Created by developer cryingneko and shared on r/LocalLLaMA, oMLX combines a native macOS menubar application with a sophisticated paged SSD caching system that fundamentally changes how large language models operate on personal machines.

Unlike Ollama — which, despite its popularity, relies on a more generalized backend and lacks deep integration with Apple’s MLX framework — oMLX is built from the ground up to leverage Apple’s native machine learning libraries. This enables unprecedented efficiency, particularly in memory-constrained environments like laptops. The most compelling innovation is its paged SSD caching mechanism, which persists key-value (KV) cache blocks to the device’s solid-state drive. This means that when a coding agent or note-taking tool revisits a previously processed context, the system doesn’t recompute it from scratch — it restores the cached block in milliseconds.

For developers using local LLMs with tools like Claude Code or Obsidian Copilot, this is a revelation. Traditional inference servers suffer from cache thrashing when context prefixes shift frequently — a common scenario in iterative coding sessions. Each new prefix invalidates the GPU’s KV cache, forcing the model to reprocess long sequences of text. This results in frustrating delays, often exceeding several seconds per turn. oMLX eliminates this bottleneck by storing every cache block on SSD, enabling near-instantaneous retrieval even after the server is restarted. The system uses a copy-on-write architecture inspired by vLLM and vllm-mlx, ensuring memory efficiency while supporting hybrid cache types for complex models like Gemma3 and DeepSeek.

Beyond caching, oMLX offers a comprehensive suite of enterprise-grade features in a consumer-friendly package. It supports continuous batching via mlx-lm, allowing multiple concurrent requests to be processed efficiently. Users can simultaneously load and serve an LLM, embedding model, and reranker — all managed through an LRU eviction policy that optimizes memory usage. The server is fully compatible with OpenAI’s API endpoints (/v1/chat/completions, /v1/embeddings, etc.) and also supports Anthropic’s /v1/messages format, making it a drop-in replacement for cloud-based services. Tool calling, structured JSON output, and MCP protocol support further enhance its utility for automation workflows.

Perhaps most impressively, oMLX comes with a fully native macOS menubar application built using PyObjC — not Electron — ensuring low resource consumption and seamless integration with macOS system services. The dashboard includes a built-in chat interface, real-time GPU/SSD monitoring, and an integrated Hugging Face model downloader, eliminating the need for command-line interactions. The entire application is distributed as a signed and notarized DMG, meeting Apple’s security standards and making installation as simple as dragging an icon to the Applications folder.

With requirements limited to Apple Silicon (M1 or newer) and macOS 14.0+, oMLX democratizes high-performance local AI for a vast user base. Its open-source nature invites community contributions, and its focus on practical, real-world use cases — particularly for developers and knowledge workers — positions it as a potential successor to Ollama for Apple users. As the demand for private, low-latency LLMs grows, oMLX represents a rare fusion of technical innovation and user-centric design.

Source: Reddit r/LocalLLaMA, post by /u/cryingneko