A research team says it has squeezed more LLM performance out of Apple Silicon than any existing runtime — by writing directly to the metal, literally.
BaseRT is a new inference runtime built natively on Apple's Metal GPU API rather than on top of existing frameworks like llama.cpp or MLX. The authors argue that those tools carry abstraction overhead not designed for Metal's execution model or Apple Silicon's unified memory layout. By writing chip-specific kernel fusion, memory-aware optimisation, and custom dispatch logic from scratch, they report decode throughput up to 1.56x faster than llama.cpp and up to 1.35x faster than MLX on M3 and M4 Pro devices. The gains are reportedly larger still on prefill for mixture-of-experts models. BaseRT covers model families including Qwen3, Llama 3.2, and Gemma 4, across eight quantisation formats from Q2 to FP16, and spans sub-1B to 30B parameter models. The figures come from the BaseRT authors and have not been independently replicated.
The broader argument here is that Apple Silicon as an inference platform has been underrated — not because the hardware is weak, but because the software layer was leaving performance unclaimed. That matters as edge inference becomes a real engineering priority: rising cloud costs, privacy regulations, and latency demands are pushing more model execution onto local devices, and the runtime layer is a real bottleneck.
BaseRT is publicly available on GitHub, which invites independent benchmarking — and independent benchmarking is exactly what these claims need before anyone rewrites their local inference stack around them.