Quantization Formats for Apple Silicon LLM Inference

Every token an Apple Silicon model produces has to pull the entire set of weights through memory once, and that single fact decides more about quantization strategy than any benchmark chart. Tokens per second comes down to bandwidth divided by the size of the model sitting in memory, and once that ratio is fixed, the game becomes about shrinking the model without shrinking what it knows. On a laptop or a Mac Studio, the CPU, GPU, and Neural Engine all draw from the same pool of high-bandwidth memory. There's no PCIe bus to cross, no separate VRAM pool that has to hold a copy of the weights before the GPU can touch them, the way there is on a discrete-GPU desktop rig. That single design choice, unified memory architecture, is why quantization format selection on Apple hardware is a bandwidth-matching problem rather than a generic size-versus-quality tradeoff. It's a bandwidth-matching problem, and On Apple Silicon generally, the memory bandwidth gains across generations have driven inference speed improvements more than core count increases alone.

How GGUF became the cross-platform baseline and what its sub-format hierarchy means

GGUF showed up on August 21, 2023, as a clean break from GGML, formalized in pull request #302 on the ggerganov/ggml repository. It's backward-incompatible by design, and the tradeoff paid off: one file now holds the weights, the metadata, the tokenizer, and often a chat template, all packaged together so a model can move between tools without extra setup. That portability is the reason GGUF became the default. It's the only major format that runs on a CPU with no GPU at all, on consumer NVIDIA and AMD cards, and on Apple Silicon, and it's native to llama.cpp, Ollama, and LM Studio, three of the most-used local inference tools around.

"GGUF" isn't one format, it's a family, and the internal split affects which quant a model actually loads as and how it performs. The legacy quants, Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, use a flat, single-level block structure: every layer gets the same bit budget regardless of whether that layer actually needs it. The two differ in how they handle weight distributions, a distinction that affects accuracy on models whose weights skew away from zero.

K-quants (Q2_K, the Q3_K and Q4_K and Q5_K families in S/M/L variants, Q6_K) replaced that flat structure with something hierarchical: grouped blocks of values, each with its own quantized scale and minimum. That structure lets the format apply mixed precision within a single model, giving more sensitive layers more bits than layers where the accuracy hit matters less. The "K" in the name refers to exactly this, a mixed-precision class rather than a single fixed bit-width. The quality gap this produces isn't subtle. Perplexity data compiled by one research group shows legacy Q4_0 with a perplexity delta of +0.2499 against the full-precision baseline, while Q4_K_M is +0.0535, roughly five times better, at only a slightly larger file size. Within the K-quant family itself, Q4_K_S trims precision further than Q4_K_M and gives up measurable quality for it. Q4_K_M keeps getting cited as the practical default for anyone who isn't optimizing for one specific constraint.

Beyond K-quants sits a newer class: IQ quants and imatrix-based quantization, which use a calibration pass to figure out which tensors are sensitive to precision loss before the model gets quantized at all. Various IQ quant variants show up in this class, and the naming scheme keeps expanding as the approach matures.

What Unsloth's Dynamic GGUFs reveal about calibration-driven quantization on Apple Silicon hardware

Unsloth's Dynamic 2.0 approach takes the K-quant idea and pushes it further: instead of picking a handful of layers to treat specially, it adjusts the quantization type of every layer individually, and it does this for both mixture-of-experts and dense architectures. Dynamic v3.0, released in August 2026 for Qwen3.8-27B, claims better than a 10% improvement in top-1% accuracy over every other provider at the same file size. That's a specific, falsifiable claim, and it rests on a specific evaluation method: Divergence-300 @32, which runs 300 examples pulled from Terminal-Bench 2.1, DeepSWE, Harbor, MathArena 2025-26, plus non-Latin-script and long-document prompts, through decoding for 32 tokens, then comparing the output against a full-precision reference. The evaluation method is designed to be a specific, reproducible test rather than a marketing number.

A broader benchmark measured 64 different GGUF quants from six separate uploaders, all scored by KL divergence against a BF16 reference across roughly 250,000 tokens spanning coding, chat, tool calling, science, non-Latin scripts, and long documents. Q8_0 hit a KL divergence of 0.069, which set the ceiling for how good any quant in the comparison could reasonably get. What's notable is where the model struggled even at that high bit-width: tool calling posted a KL divergence of 0.177 at Q8_0, worse than long-document handling at 0.121, an odd result for a model family that's supposed to be tool-call-competent. Across the full field of 64 quants, Unsloth's uploads claimed a majority of the frontier positions among quants not strictly beaten on both quality and size, though a meaningful share of those spots still belonged to other uploaders, so no one lab's quantization should be treated as uniformly superior.

Where MLX pulls ahead of GGUF on Apple Silicon

MLX is Apple's own answer to this problem, open-sourced in late 2023, and it took roughly two years to reach real production maturity. By 2026 it had moved decisively ahead of llama.cpp's Metal backend on its home turf. Where GGUF is a file format that many runtimes read, MLX is a framework built around Apple Silicon's memory model from the start, and its main quantization path is native 4-bit and 8-bit, with mixed-precision options available. Converting a Hugging Face model over takes one command: mlx_lm.convert --hf-path X --q-bits 4, no separate quantization tool required.

The quality numbers hold up: 4-bit quantization on models above 7B parameters keeps 97.3% of the full-precision MMLU score while cutting memory use by several times over. That ratio doesn't hold at the small end, though. Sub-3B models take a visible quality hit at 4-bit, and 8-bit is the better call for anything that size. On speed, MLX runs 15 to 30% faster than GGUF at equivalent quantization levels. For models under roughly 14B parameters, the gap widens further, with MLX beating llama.cpp on decode throughput by 20 to 87%, depending on the model and quant combination, wide enough to change which framework someone chooses. That's not a marginal win. The gap changes which framework someone chooses before they've even looked at quality benchmarks.

What native Metal kernel design adds beyond framework-level optimization in Ollama 0.19 and BaseRT

Ollama shipped version 0.19 on March 31, 2026, with a new inference backend built on MLX, and rather than adapting a cross-platform engine to run acceptably on Apple hardware, it treats unified memory as the starting assumption. That backend carries a hard requirement of 32GB or more of unified memory to even activate. Base M4 machines at 8 or 16GB simply don't get it; that's a floor, not a soft recommendation.

The performance delta on qualifying hardware is large. On qualifying hardware, decode throughput roughly doubled on the new MLX backend compared to the old llama.cpp backend, and prefill throughput improved substantially alongside it. Those numbers change what feels usable in an actual conversation with a local model.

BaseRT goes a step further still. It's a native Metal inference runtime, built specifically around Apple Silicon's memory topology rather than adapted to it, and it supports everything from Q2 up to FP16 across eight quantization formats on any M-series device. Tested against Qwen3, Llama 3.2, and Gemma 4 at Q4 and Q8 on M3 and M4 Pro hardware, BaseRT posted decode throughput meaningfully higher than llama.cpp and higher still than MLX on Q4 models on the M4 Pro. Prefill throughput on Qwen3-30B-A3B at 4-bit reached a large multiple over MLX, with the gap growing substantially wider specifically on MoE prefill, a workload that punishes generic kernel design more than dense models do. Chip-specific kernel fusion combined with unified-memory-aware optimization and custom dispatch logic produces those numbers, recovering throughput that a framework built for portability across many chips inevitably leaves on the table. BaseRT is public at github.com/basecompute/baseRT for anyone who wants to check the numbers directly.

Set the frameworks side by side and the split becomes clear. llama.cpp ships a built-in quantizer for GGUF across several bit-widths, but it's tied to that format without a conversion step. Ollama 0.19 added the MLX path as a new option, gated behind that 32GB memory requirement. MLC-LLM supports a range of quantization formats including mixed precision, with a production-grade ingestion pipeline for Hugging Face models. MLX itself offers Apple-tuned mixed-precision quantization, with a growing library of pre-quantized MLX models available directly. PyTorch's MPS backend, by contrast, has lagged behind on quantization support and stability with larger models, and is not yet a serious option for anyone running production inference locally.

What BaseRT's numbers establish, more than any single benchmark, is that the quantization format and the kernel that executes it are not independent choices. A Q4 model run through a kernel that doesn't understand unified memory leaves real throughput on the table, throughput that a Metal-native kernel recovers without touching the quantization itself. Choosing a bit-width and choosing a runtime are the same decision, not two separate ones made in sequence.

MoE architecture as a special case where quantization format interacts with routing sparsity

Mixture-of-experts models complicate this picture in a specific way. Sparse activation, where only a fraction of total parameters fire on any given token, already reduces the bytes that need to move through memory, and on unified memory hardware that reduction stacks directly on top of whatever savings quantization already provides. Fewer active parameters times fewer bits per parameter compounds faster than either number alone suggests.

Offloading that computation to Apple's Neural Engine runs into a catch. Dynamic expert routing produces tensor shapes that change from token to token, and that's a poor match for the ANE's static execution pipeline, which expects to know its computation graph in advance. Top-k selection and scatter/gather operations, both central to how MoE routing actually works, aren't operations the ANE handles well, and launching small expert kernels one at a time adds dispatch overhead that eats into whatever gain the sparsity was supposed to deliver.

A runtime called NPUMoE was built specifically to work around this. It offloads the dense, predictable parts of computation to the ANE while keeping dynamic routing operations on the CPU or GPU where they belong, using an offline calibration pass to estimate how much capacity each expert needs and how often each one gets called. Three techniques do the heavy lifting: static capacity tiers assigned per expert, grouped execution of experts rather than one-by-one dispatch, and load-aware residency in the compute graph that cuts down on synchronization overhead between the CPU and the ANE. Tested across three MoE language models and four long-context workloads on Apple M-series hardware, NPUMoE cut latency by a wide range depending on the workload, from a modest reduction up to several times over, and improved energy efficiency by a similarly wide range, a spread wide enough to suggest the gains depend heavily on how sparse the specific model's routing is. The ANE is built around FP16 compute specifically, so any quantization scheme aiming to use it has to account for that constraint rather than assuming the same bit-width choices that work well on GPU will transfer cleanly.