Cover illustration for “Quantization Formats for Apple Silicon LLM Inference”

Essay On-Device Inference

Quantization Formats for Apple Silicon LLM Inference

How to pick the right quantization format for running large language models on Apple Silicon.

Mireille Okonkwo9 min read

Personal AI Journal

On-Device Inference

GGUF showed up on August 21, 2023, as a clean break from GGML, formalized in pull request #302 on the ggerganov/ggml repository.
From Quantization Formats for Apple Silicon LLM Inference