Size reduction (vs FP16)    Quality (perplexity / benchmark)
   Q2  ~88% smaller   ---->   noticeable degradation; use only when desperate
   Q3  ~84% smaller   ---->   measurable degradation; acceptable for some tasks
   Q4  ~75% smaller   ---->   SWEET SPOT; minimal perceptible loss
   Q5  ~70% smaller   ---->   near-lossless; modest size premium
   Q8  ~58% smaller   ---->   effectively lossless; large
   FP16 (reference)   ---->   baseline

# 1. Convert the HF checkpoint to a base (F16) GGUF
python convert_hf_to_gguf.py ./my-finetuned-model --outfile model-f16.gguf

# 2. Quantize to the sweet spot
llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

model.save_pretrained_gguf("model-gguf", quantization_method="q4_k_m")

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = "./my-finetuned-model"
quant_path = "./my-model-awq"
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }

model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

python -m mlx_lm.convert \
    --hf-path ./my-finetuned-model \
    --quantize --q-bits 4 \
    --mlx-path ./my-model-mlx

# Module FT19 — Quantization Formats

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT19 — Quantization Formats
**Duration**: 75 minutes
**Level**: Senior Engineer and above
**Prerequisites**: FT08 (QLoRA — you have already trained quantized; now you *export* quantized). FT11 (the training loop — you need a fine-tuned checkpoint to compress).

---

## Learning Objectives

After completing this module, you will be able to:

1. State the **format decision matrix**: for a given deployment target (local CPU, Mac, NVIDIA production server, frontier Hopper/Blackwell), name the right quantization format and defend the choice.
2. Explain the **quality/size trade-off** — why Q4 (~4 bits/parameter) is the general sweet spot (~75% size reduction vs. FP16 with minimal quality loss), and why Q2/Q3 degrade noticeably while Q8 is near-lossless.
3. Run the three primary **conversion workflows**: GGUF (llama.cpp / Unsloth), AWQ (AutoAWQ), and MLX (mlx-lm).
4. Explain **Unsloth Dynamic 2.0** and why intelligent per-layer quantization (sensitive layers stay higher precision) beats uniform quants at the same size.
5. Place quantization correctly in the stack: **Layer 4, downstream of Layer 3 (training)**. It compresses what training produced; it does not change learned behavior, only compactness and speed.

---

# 19.1 — Where Quantization Sits: Layer 4, Downstream of Training

*The first thing to get right is the position. Quantization is not a training step. It is an export step.*

You finished FT11. You have a fine-tuned checkpoint — a model whose behavior has been steered (Layer 3) on top of a base (Layer 1) via an adapter or full-parameter update (Layer 2). The weights are (almost always) in FP16 or BF16 — sixteen bits per parameter. That is the format training produced. It is also far too large to deploy cheaply. A 7B model in FP16 is ~14 GB; a 70B model is ~140 GB. Those numbers gate who can run your model and where.

**Layer 4 — the Export — is where you compress.** Quantization takes the trained weights and represents them in fewer bits: 8-bit, 4-bit, sometimes as low as 2-bit. The model that comes out the other side has the *same architecture and the same learned behavior* — it just occupies less memory and runs faster. The defining property, restated from FT00:

> **Layer 4 is downstream of training.** You quantize *after* you fine-tune. Quantization compresses what Layer 3 produced; it does not change what the model learned, only how compactly and quickly it runs.

This is why QLoRA (FT08) is a special case: QLoRA quantizes the base *during* training so you can afford the forward/backward passes, then leaves the adapter in high precision. That is a training-time trick. FT19 is about the *deployment-time* compression you apply to the final, fully-merged checkpoint. The two are related but distinct: QLoRA is "quantize to train"; FT19 is "quantize to serve."

There is one subtle but important caveat. Aggressive quantization *does* introduce a small amount of error — a quantized 4-bit weight is an approximation of the 16-bit original. At sensible bitrates (Q4 and above) that error is small enough to be near-imperceptible on most benchmarks. At aggressive bitrates (Q2, Q3) it is not, and you will see quality drop. So: quantization does not *re-train* behavior, but a bad quantization choice can *degrade* it. The job of this module is to keep you on the right side of that line.

---

# 19.2 — The Format Decision Matrix

*Seven formats, one question: where is this model going to run?*

The single most important fact about quantization formats is this: **the format is determined by the inference engine that will run the model.** You do not pick a format in the abstract. You pick the runtime, then the format that runtime consumes.

Here is the matrix. Read it as: "If the deployment target is X, the format is Y."

| Deployment target | Format | Why |
| --- | --- | --- |
| **Local / CPU / mixed CPU+GPU / Mac (via Ollama, llama.cpp)** | **GGUF** | The universal local format. One file, runs anywhere llama.cpp runs. |
| **NVIDIA GPU production serving (vLLM, TGI)** | **AWQ** (esp. AWQ-Marlin) | First-class in vLLM with optimized kernels. Best speed/quality for server inference. |
| **NVIDIA GPU serving (mature, widely available)** | **GPTQ** (GPTQ-Marlin) | Mature, slightly worse quality than AWQ at 4-bit; showing its age but ubiquitous. |
| **Max quality at low bitrate (ExLlamaV2 ecosystem)** | **EXL2** | Variable-rate per-layer quantization. Best-in-class perplexity at small sizes, but ExLlamaV2-only. |
| **Apple Silicon native** | **MLX** | Best Mac inference perf via mlx-lm / LM Studio. 4-bit group quant is the default. |
| **High-quality serving when VRAM allows (Hopper/Blackwell)** | **FP8** | The new high-quality option. Minimal quality loss; needs the VRAM headroom. |
| **Blackwell-native 4-bit (2025 frontier)** | **MXFP4 / NVFP4** | Both E2M1 4-bit. MXFP4 block=32; NVFP4 block=16, finer-grained, better accuracy. |

Let me walk each one.

### GGUF — the universal local format

GGUF (GPT-Generated Unified Format) is the file format consumed by **llama.cpp** and everything built on it: **Ollama, LM Studio, llamafile, KoboldCpp.** If you are running a model on a laptop, a Raspberry Pi, a consumer desktop with mixed CPU/GPU offload, or a Mac using Metal, you are almost certainly running GGUF.

The defining property of GGUF is **maximum compatibility.** A single `.gguf` file is self-contained — weights, tokenizer, config, chat template, all in one. It runs on CPU-only machines (slowly), on CPU+GPU offload (the typical consumer setup), and on Mac via the Metal backend (which is fast — Apple Silicon has unified memory and GGUF exploits it well). No other format matches its portability.

**Q4_K_M is the sweet spot.** GGUF uses a "K-quant" naming scheme (`Q4_K_M`, `Q5_K_M`, `Q3_K_M`, etc.) where the number is the target bits-per-weight and `_M` means "medium" (a balance of layers quantized at different precisions). **Q4_K_M** — roughly 4 bits per parameter — is the community default because it hits ~75% size reduction with minimal perceptible quality loss. A 7B model that is 14 GB in FP16 becomes ~4.4 GB in Q4_K_M. That is the number to reach for first. Move to Q5 or Q8 only if you have VRAM to spare and want near-lossless; drop to Q3 or Q2 only if you are desperate for size and accept the quality cost.

### AWQ — production serving on NVIDIA GPUs

AWQ (Activation-aware Weight Quantization) is the format you reach for when the model is going to be **served at scale on NVIDIA GPUs via vLLM or TGI.** The variant that matters in 2025 is **AWQ-Marlin** — the Marlin kernel is a highly optimized GPU inference kernel, and vLLM has first-class support for it.

The reason AWQ beats naive 4-bit on servers is in the name: it is *activation-aware.* It uses a calibration set to identify the ~1% of "salient" weights that matter most for preserving output quality, and keeps those at higher precision while quantizing the rest. The result: 4-bit AWQ has noticeably better perplexity than 4-bit GPTQ at the same size. For a production serving stack where you want the best speed/quality ratio, AWQ-Marlin in vLLM is the default recommendation.

### GPTQ — mature, ubiquitous, slightly behind

GPTQ is the older 4-bit format, based on a one-shot post-training quantization algorithm that uses a small calibration set to minimize output error. It is **mature and widely available** — nearly every quantized model on Hugging Face has a GPTQ variant. The **GPTQ-Marlin** kernel (the same optimized kernel family AWQ uses) gives it good inference speed in vLLM.

The honest assessment: at 4-bit, GPTQ is **slightly worse in quality than AWQ** (higher perplexity, small benchmark drops) for the same model size. It is "showing its age." But it is everywhere, well-tested, and fine for many deployments. If you have a choice, prefer AWQ for new server deployments; if GPTQ is what your tooling supports or what's already published, it is a perfectly defensible choice.

### EXL2 — max quality at low bitrates

EXL2 (ExLlamaV2's format) does something the others do not: **variable-rate per-layer quantization.** Instead of choosing one bitrate for the whole model, EXL2 measures each layer's sensitivity and assigns it a different average bitrate — some layers at 8-bit, some at 4, some at 3 — such that the *total* model hits a target average (say, 4.5 bits/param) while keeping the sensitive layers high-precision.

The payoff: **best-in-class perplexity at small sizes.** If you absolutely need to squeeze a 70B model into the smallest possible footprint without wrecking quality, EXL2 is the format that does it. The cost: it runs in **ExLlamaV2** (and a few compatible backends). vLLM's EXL2 support is experimental. So EXL2 is the specialist's choice — superb quality, narrower runtime support.

### MLX — Apple Silicon native

MLX is Apple's own array framework, purpose-built for Apple Silicon's unified memory architecture. The **mlx-lm** library and **LM Studio** (which uses MLX under the hood on Mac) consume MLX-format models. If your deployment target is a Mac — an M-series MacBook, a Mac Studio — MLX gives you the best inference performance on that hardware, full stop. The unified memory means the CPU and GPU share the same RAM, so there is no expensive CPU-to-GPU copy; MLX exploits this directly.

The recommended MLX quant is **4-bit group quantization** (`mlx-community` publishes these as `*-4bit`). Like GGUF's Q4_K_M, it gives roughly 75% size reduction with minimal quality loss. For Mac deployment, MLX 4-bit is the default.

### FP8 — high-quality serving when VRAM allows

FP8 is the format you reach for when you have **VRAM headroom** and want to preserve as much quality as possible while still getting a 2x size/speed win over FP16. FP8 is exactly what it sounds like — 8-bit floating point — and on **Hopper (H100) and Blackwell (B100/B200)** GPUs it has hardware acceleration. The quality loss vs. FP16 is negligible on most benchmarks. If you are serving a model on H100s and the model fits in FP8, FP8 is often the right call — it is the highest-quality compressed option short of staying in FP16.

### MXFP4 / NVFP4 — the 2025 Blackwell frontier

This is the new frontier. Both MXFP4 and NVFP4 are **E2M1 4-bit floating-point formats** (1 sign, 2 exponent, 1 mantissa bit) — but they differ in the *block size* used for shared scaling factors:

- **MXFP4** (the OCP Microscaling format): block size of **32** elements sharing one scaling factor.
- **NVFP4** (NVIDIA's variant): block size of **16** elements sharing one scaling factor.

The finer-grained block (16 vs. 32) means NVFP4 can represent weight distributions more accurately — **better accuracy at the same 4-bit width.** Both are Blackwell-native, both support training and inference. These are the formats that make 4-bit *training* (not just inference) practical at frontier scale in 2025. You will encounter them most when working with the largest models on the newest hardware.

---

# 19.3 — The Quality/Size Trade-off

*One curve to memorize. The general sweet spot is Q4.*

Quantization is a trade between size/speed and quality. The curve is not linear, and the shape of the non-linearity is the whole story.

```
Size reduction (vs FP16)    Quality (perplexity / benchmark)
   Q2  ~88% smaller   ---->   noticeable degradation; use only when desperate
   Q3  ~84% smaller   ---->   measurable degradation; acceptable for some tasks
   Q4  ~75% smaller   ---->   SWEET SPOT; minimal perceptible loss
   Q5  ~70% smaller   ---->   near-lossless; modest size premium
   Q8  ~58% smaller   ---->   effectively lossless; large
   FP16 (reference)   ---->   baseline
```

The principle: **Q4 is the general sweet spot.** At roughly 4 bits per parameter you get about a 75% size reduction versus FP16, and on most benchmarks the quality loss is small — a fraction of a point on MMLU, a small perplexity uptick. For the overwhelming majority of deployments, Q4 is where you start and often where you stop.

Go lower — Q3, Q2 — and you save more memory, but the degradation becomes *noticeable.* Q2 models are dramatically smaller but you will see it in the output: more errors, worse reasoning, occasional incoherence. Reach for Q2 only when you are VRAM-constrained to the point that you cannot fit the model any other way, and accept that you are trading real quality for the size win.

Go higher — Q5, Q8 — and you approach lossless. Q8 is for most practical purposes indistinguishable from FP16. You pay for it in size: a Q8 model is roughly half the FP16 size, not a quarter. Use Q8 when you have the memory and want to be certain you are not losing anything (e.g., a high-stakes eval, or a deployment where a small perplexity difference matters).

The same logic applies *within* a format's K-quant ladder. GGUF's `Q4_K_M` is the medium-mix 4-bit; `Q4_K_S` is the small-mix (slightly smaller, slightly lower quality). When in doubt, take the `_M`.

---

# 19.4 — The Conversion Workflows

*Three formats you will actually convert to. Here is how.*

### GGUF via llama.cpp (or Unsloth)

The canonical GGUF conversion uses llama.cpp's `convert_hf_to_gguf.py` script. You point it at a Hugging Face checkpoint directory and it emits a `.gguf` file, which you then quantize to your target level with `llama-quantize`.

```bash
# 1. Convert the HF checkpoint to a base (F16) GGUF
python convert_hf_to_gguf.py ./my-finetuned-model --outfile model-f16.gguf

# 2. Quantize to the sweet spot
llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M
```

**Unsloth** provides a one-shot alternative that handles the whole pipeline — convert + quantize + even push to Ollama — in a few lines. If you trained with Unsloth (FT08/FT11), the `save_pretrained_gguf` path does this directly:

```python
model.save_pretrained_gguf("model-gguf", quantization_method="q4_k_m")
```

This is the lowest-friction path to a deployable local model.

### AWQ via AutoAWQ

AutoAWQ is the library for producing AWQ-format models. It loads your FP16 checkpoint, runs a calibration pass on a small dataset (to find the salient weights), and writes the quantized model back in a format vLLM can load directly.

```python
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = "./my-finetuned-model"
quant_path = "./my-model-awq"
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }

model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)
```

The output loads in vLLM as `--quantization awq` (and on Hopper/Blackwell, vLLM auto-routes to the Marlin kernel for speed).

### MLX via mlx-lm

For Apple Silicon, `mlx-lm` converts a Hugging Face checkpoint to MLX format with a single command:

```bash
python -m mlx_lm.convert \
    --hf-path ./my-finetuned-model \
    --quantize --q-bits 4 \
    --mlx-path ./my-model-mlx
```

The resulting directory loads directly in `mlx-lm` generate, in LM Studio, or anywhere the MLX runtime is used.

---

# 19.5 — Unsloth Dynamic 2.0: Intelligent Per-Layer Quantization

*The reason uniform quants leave quality on the table — and the fix.*

A *uniform* quantization (standard GGUF Q4_K_M, standard AWQ 4-bit) applies essentially the same bitrate across the whole model. But not all layers are equally sensitive. Some layers — often attention projection layers, or specific MLP blocks — have weight distributions that degrade badly when compressed to 4-bit. Others compress cleanly to 3-bit or below with no measurable effect.

**Unsloth Dynamic 2.0** does what EXL2 does at export time, but as a general-purpose quantization strategy: it *measures each layer's sensitivity* and assigns bitrates accordingly. Sensitive layers stay at higher precision (5-, 6-, or 8-bit). Less sensitive layers drop to 3-bit or lower. The total model hits the same average size as a uniform Q4, but the *quality* is higher because the bits went where they matter.

The empirical result: **Dynamic 2.0 beats uniform quants on quality at the same file size.** If you have the option, it is strictly preferable to a naive Q4 for the formats that support it. It is the same insight as EXL2's variable-rate quantization, packaged for the conversion workflow you already use.

---

# 19.6 — Why AFTER Training (Not During)

This is worth stating explicitly because it is the layer-assignment judgment the course keeps returning to.

You quantize **after** training because:

1. **It compresses what training produced.** The trained weights are the artifact you want to deploy. Quantization makes that artifact smaller and faster. It is a transform on the final weights, not a step in the optimization.
2. **It does not change learned behavior** (at sensible bitrates). The model's steering — the format, preferences, reasoning activation, compliance that Layers 2–3 produced — is preserved. Quantization at Q4+ perturbs the weights slightly but does not move the behavior across any decision boundary that matters.
3. **It is cheap and repeatable.** A quantization pass takes minutes, not the hours/days of a training run. You can quantize the same checkpoint to multiple formats (GGUF for local, AWQ for server, MLX for Mac) from one source checkpoint without retraining.

The one exception is QLoRA (FT08), which quantizes *during* training to fit the base in VRAM. But note: even QLoRA keeps the *adapter* in high precision and, at the end, you merge the adapter back into a dequantized or freshly-quantized checkpoint for deployment. The training quantization is a memory trick; the deployment quantization is the export step. QLoRA's training-time quant is not the same as the export-time quant this module is about.

---

## Anti-Patterns

### Quantizing to Q2 for size and being surprised by quality loss

The most common deployment mistake. A team needs the model smaller, reaches for the smallest available quant, and is then confused when the model that scored well in FP16 now produces worse output. Q2 is a desperate measure for VRAM-constrained situations, not a default. **Start at Q4.** Only go lower with eyes open and a benchmark comparing the quantized model to the original.

### Mixing formats carelessly

Loading an AWQ checkpoint into a GGUF-only runtime, or trying to serve a GPTQ model where vLLM expects AWQ-Marlin. **The format must match the runtime.** Decide the deployment target first (FT20), then the format that target consumes, then convert. Do not pick a format because it produced the smallest file and then discover it does not run where you need it to.

### Not benchmarking the quantized model against the original

Quantization is lossy in principle. The only way to know *how* lossy for your model and your workload is to measure. Run the same eval suite (GSM8K, MMLU, your domain benchmark, a perplexity check) on the FP16 original *and* the quantized variant. If the delta is within your tolerance, ship. If it is not, step up a bitrate (Q4 -> Q5 -> Q8) or change formats. Shipping a quantized model you never measured is flying blind.

### Treating quantization as part of training

Trying to "quantize-aware train" when you do not need to, or folding the deployment quant into the training loop unnecessarily. Layer 4 is downstream of Layer 3. Train in full precision (or QLoRA), merge, *then* quantize the merged checkpoint for export. Keep the layers separate — the modularity of the stack is the point.

---

## Key Terms

| Term | Definition |
| --- | --- |
| **Quantization** | Layer 4 export step; representing trained weights in fewer bits (8/4/3/2) to reduce size and increase speed |
| **GGUF** | The universal local format consumed by llama.cpp / Ollama / LM Studio; runs on CPU, mixed CPU/GPU, Mac Metal |
| **Q4_K_M** | The GGUF K-quant sweet spot (~4 bits/param); ~75% size reduction with minimal quality loss |
| **AWQ** | Activation-aware Weight Quantization; the 4-bit format for vLLM/TGI production serving on NVIDIA GPUs (AWQ-Marlin uses optimized kernels) |
| **GPTQ** | Mature one-shot 4-bit post-training quantization; slightly worse quality than AWQ at 4-bit; ubiquitous |
| **EXL2** | ExLlamaV2 format; variable-rate per-layer quantization; best-in-class perplexity at small sizes; ExLlamaV2-only |
| **MLX** | Apple Silicon-native format; best Mac inference perf via mlx-lm / LM Studio; 4-bit group quant recommended |
| **FP8** | 8-bit floating point; high-quality serving option on Hopper/Blackwell with hardware acceleration; needs VRAM headroom |
| **MXFP4 / NVFP4** | 2025 Blackwell-native 4-bit (E2M1) formats; MXFP4 block=32, NVFP4 block=16 (finer-grained, better accuracy) |
| **Unsloth Dynamic 2.0** | Intelligent per-layer quantization; sensitive layers stay higher precision; beats uniform quants at same size |
| **K-quant** | GGUF's naming scheme (`Q4_K_M`); the number is target bits/weight, `_M` is the medium layer-mix |

---

## Lab Exercise

See `07-lab-spec.md`. The "Quantize Three Ways" lab: take your fine-tuned model from FT11, convert it to GGUF Q4_K_M, AWQ 4-bit, and MLX 4-bit. Measure size, inference speed, and quality (perplexity + a benchmark) for each. Build the per-target recommendation table. Consumer-GPU/Mac-runnable in an afternoon.

---

## References

1. **llama.cpp** — Gerganov et al. The GGUF format and the reference local inference engine. <https://github.com/ggerganov/llama.cpp>. The `convert_hf_to_gguf.py` script and the K-quant ladder.
2. **AutoAWQ** — The library for producing AWQ-format models for vLLM/TGI serving. <https://github.com/casper-hansen/AutoAWQ>. Lin et al., *AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration*, arXiv:2306.00978.
3. **Unsloth Dynamic 2.0** — Intelligent per-layer quantization that beats uniform quants on quality at the same size. <https://unsloth.ai/blog/dynamic-2.0>. The dynamic quantization strategy that keeps sensitive layers at higher precision.
4. **NVIDIA NVFP4 blog** — NVIDIA's writeup of the NVFP4 format (block size 16) and its accuracy advantage over MXFP4 (block size 32) on Blackwell. The frontier 4-bit training+inference formats.
5. **kubesimplify — "Quantization demystified"** — A practitioner's walkthrough of the major formats (GGUF, AWQ, GPTQ, EXL2, MLX) and when to use each. <https://github.com/kubesimplify>.
6. **Frantar et al. (2022)** — *GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers*, arXiv:2210.17323. The GPTQ algorithm.
7. **Course 3, FT08** — *QLoRA*. The training-time quantization trick. FT19 is the deployment-time complement: you trained with QLoRA, you export with the formats in this module.
8. **Course 3, FT20** — *Serving Stacks*. The next module: the runtimes (vLLM, Ollama, TGI) that consume the formats you produce here.