Course: Course 3 — LLM Fine-Tuning Masterclass
Module: FT19 — Quantization Formats
Duration: ~45 minutes (spoken at ~140 wpm)
Format: Verbatim transcript with [SLIDE N] cues. Read aloud or use as speaker notes.
[SLIDE 1 — Title]
Welcome to module FT nineteen — Quantization Formats. This is the first module of pillar six, Deploy. We have spent five pillars steering the model — base, data, adapter, alignment. Now we cross into the export layer. You have a fine-tuned checkpoint. It works. It is also, in its native sixteen-bit format, far too large to deploy cheaply. A seven-billion-parameter model in FP16 is fourteen gigabytes. A seventy-billion model is one hundred and forty. Those numbers gate who can run your model and where. This module is how you compress it without wrecking what you spent five pillars building.
The thesis in one sentence: the format is determined by where the model will run. Pick the runtime, then the format. Default to four-bit. Always benchmark against the original.
[SLIDE 2 — Where it sits: Layer 4]
First, the position. Quantization is Layer four — the Export. It sits downstream of Layer three, the Steer, which is everything you just finished in pillars three through five. The defining property, from FT zero zero: Layer four is downstream of training. You quantize AFTER you fine-tune. It compresses what training produced. It does not change learned behavior at sensible bitrates — it changes how compactly and quickly the model runs.
This is related to but distinct from QLoRA, which you met in FT zero eight. QLoRA quantizes the base DURING training as a memory trick, so the forward and backward passes fit in VRAM. The adapter stays high-precision. That is quantize-to-train. FT nineteen is quantize-to-serve: you take the merged checkpoint and compress it for deployment. Even a QLoRA-trained model gets a fresh deployment-time quantization at export.
One caveat to be honest about. Aggressive quantization does introduce a small amount of error. A four-bit weight is an approximation of the sixteen-bit original. At sensible bitrates — four-bit and above — that error is small enough to be near-imperceptible on most benchmarks. At aggressive bitrates — two and three bit — it is not, and you will see quality drop. So quantization does not re-train behavior, but a bad quantization choice can degrade it. The job of this module is to keep you on the right side of that line.
[SLIDE 3 — The format decision matrix]
Now the matrix. The single most important fact about quantization formats: the format is determined by the inference engine that will run the model. You do not pick a format in the abstract. You pick the runtime, then the format that runtime consumes. Read the slide as: if the deployment target is X, the format is Y.
Local, CPU, mixed CPU plus GPU, Mac via Ollama — that is GGUF. The universal local format. NVIDIA GPU production serving via vLLM or TGI — that is AWQ, specifically AWQ-Marlin. Maximum quality at low bitrate in the ExLlamaV2 ecosystem — that is EXL2. Apple Silicon native — that is MLX. VRAM-rich Hopper or Blackwell where you want high quality — FP8. And the twenty twenty-five Blackwell-native frontier, four-bit — that is MXFP4 or NVFP4.
Let me walk each one.
[SLIDE 4 — GGUF and the local formats]
GGUF is the file format consumed by llama dot cpp and everything built on it — Ollama, LM Studio, llamafile, KoboldCpp. If you are running a model on a laptop, a consumer desktop with mixed CPU-GPU offload, or a Mac via Metal, you are almost certainly running GGUF. Its defining property is maximum compatibility. A single dot-g-g-u-f file is self-contained — weights, tokenizer, config, chat template, all in one. No other format matches its portability.
The GGUF sweet spot is Q4 underscore K underscore M. The K-quant scheme: the number is target bits per weight, the underscore M means medium. Q4 K M is roughly four bits per parameter, about seventy-five percent size reduction with minimal perceptible quality loss. A seven-billion model goes from fourteen gigabytes to about four point four. That is the number to reach for first. Move to Q5 or Q8 only with VRAM to spare; drop to Q3 or Q2 only if desperate.
[SLIDE 5 — The NVIDIA serving formats]
For NVIDIA GPU production serving, AWQ is the format. Activation-aware Weight Quantization, and the variant that matters is AWQ-Marlin — Marlin is a highly optimized kernel, first-class in vLLM. AWQ beats naive four-bit because it is activation-aware: a calibration set identifies the one percent of salient weights that matter most for output quality, and keeps those high-precision while quantizing the rest. The result: four-bit AWQ has noticeably better perplexity than four-bit GPTQ at the same size. AWQ-Marlin in vLLM is the default recommendation for new server deployments.
GPTQ is the older four-bit format — mature, ubiquitous, slightly worse quality than AWQ at four-bit, showing its age, but everywhere and well-tested. If GPTQ is what your tooling supports, it is defensible. If you have a choice, prefer AWQ.
[SLIDE 6 — EXL2, MLX, and the frontier formats]
Three more, quickly. EXL2 is ExLlamaV2's format, and it does variable-rate per-layer quantization — each layer gets a different average bitrate based on its measured sensitivity, so the total hits a target average while sensitive layers stay high-precision. The payoff: best-in-class perplexity at small sizes. If you must squeeze a seventy-billion model into the smallest footprint without wrecking quality, EXL2 does it. The cost: ExLlamaV2-only runtime. Specialist's choice.
MLX is Apple Silicon-native. It exploits unified memory — no CPU-to-GPU copy — giving the best inference performance on a Mac. mlx-lm and LM Studio load it. The recommended quant is four-bit group, same sweet spot as GGUF's Q4 K M.
FP8 is the high-quality option when VRAM allows — a two-times size and speed win over FP16 with negligible quality loss, hardware-accelerated on Hopper and Blackwell. If the model fits in FP8 on H one hundreds, it is often the right call.
And the twenty twenty-five frontier: MXFP4 and NVFP4. Both are E two M one four-bit floating-point — one sign, two exponent, one mantissa. The difference is block size: MXFP4 uses thirty-two elements per shared scaling block; NVFP4 uses sixteen, which is finer-grained and gives better accuracy at the same four-bit width. Both are Blackwell-native, both support training and inference. These make four-bit training, not just inference, practical at frontier scale this year.
[SLIDE 7 — The quality/size trade-off]
Now the curve. One chart to memorize. Quantization is a trade between size and speed on one side, quality on the other. The curve is not linear, and the shape of the non-linearity is the whole story.
Read left to right. Q2 — about eighty-eight percent smaller — noticeable degradation. More errors, worse reasoning, occasional incoherence. Reach for it only when desperate. Q3 — about eighty-four percent smaller — measurable degradation, acceptable for some tasks. Q4 — about seventy-five percent smaller — the sweet spot. Minimal perceptible loss. This is where you start and usually where you stop. Q5 — seventy percent smaller — near-lossless. Q8 — fifty-eight percent smaller — effectively lossless, indistinguishable from FP16 for most purposes. Then FP16, the baseline.
The principle: Q4 is the general sweet spot. At roughly four bits per parameter you get about seventy-five percent size reduction versus FP16, and on most benchmarks the quality loss is small — a fraction of a point on MMLU, a small perplexity uptick. Go lower and you save more memory but the degradation becomes noticeable. Go higher and you approach lossless, but you pay in size. When in doubt, take the K-quant underscore M.
[SLIDE 8 — The conversion workflows]
Three formats you will actually convert to. The load-bearing point: one source checkpoint, three export artifacts. You train once in FP16, then convert the same merged checkpoint to as many formats as your targets require.
GGUF. The canonical path uses llama dot cpp's convert script — point it at your checkpoint, it emits a dot-g-g-u-f, then you quantize with llama-quantize. Two commands. Unsloth provides a one-shot alternative that handles convert, quantize, and even push to Ollama in a few lines. Lowest-friction path to a local model.
AWQ. AutoAWQ is the library. It loads your FP16 checkpoint, runs a calibration pass to find the salient weights, and writes the quantized model in a format vLLM loads directly — on Hopper and Blackwell it auto-routes to the Marlin kernel.
MLX. mlx-lm converts a Hugging Face checkpoint to MLX format with a single command. The result loads in mlx-lm, in LM Studio, or anywhere the MLX runtime is used.
[SLIDE 9 — Unsloth Dynamic 2.0]
A refinement worth knowing. A uniform quantization — standard GGUF Q4 K M, standard AWQ four-bit — applies essentially the same bitrate across the whole model. But not all layers are equally sensitive. Some layers — often attention projection layers, or specific MLP blocks — have weight distributions that degrade badly when compressed to four-bit. Others compress cleanly to three-bit or below with no measurable effect.
Unsloth Dynamic two-point-oh does what EXL2 does at export time, but as a general-purpose strategy: it measures each layer's sensitivity and assigns bitrates accordingly. Sensitive layers stay at higher precision — five, six, or eight-bit. Less sensitive layers drop to three-bit or lower. The total model hits the same average size as a uniform Q4, but the quality is higher because the bits went where they matter. The empirical result: Dynamic two-point-oh beats uniform quants on quality at the same file size. It is the same insight as EXL2's variable-rate quantization, packaged for the conversion workflow you already use. If you have the option, it is strictly preferable to a naive Q4 for the formats that support it.
[SLIDE 10 — Why after training]
Let me state the layer-assignment judgment explicitly, because it is the one the course keeps returning to. You quantize after training for three reasons. One: it compresses what training produced. The trained weights are the artifact you want to deploy. Quantization makes that artifact smaller and faster. It is a transform on the final weights, not a step in the optimization. Two: it does not change learned behavior at sensible bitrates. The model's steering — the format, preferences, reasoning activation, compliance that layers two and three produced — is preserved. Quantization at Q4 and above perturbs the weights slightly but does not move the behavior across any decision boundary that matters. Three: it is cheap and repeatable. A quantization pass takes minutes, not the hours or days of a training run. You can quantize the same checkpoint to GGUF for local, AWQ for server, MLX for Mac — all from one source checkpoint, no retraining. That modularity is the point.
[SLIDE 11 — Anti-patterns]
Four anti-patterns to leave with.
First, quantizing to Q2 for size and being surprised by quality loss. The most common deployment mistake. A team needs the model smaller, reaches for the smallest available quant, and is then confused when the model that scored well in FP16 now produces worse output. Q2 is a desperate measure, not a default. Start at Q4. Only go lower with eyes open and a benchmark comparing the quantized model to the original.
Second, mixing formats carelessly. Loading an AWQ checkpoint into a GGUF-only runtime, or trying to serve a GPTQ model where vLLM expects AWQ-Marlin. The format must match the runtime. Decide the deployment target first — that is FT twenty — then the format that target consumes, then convert. Do not pick a format because it produced the smallest file and then discover it does not run where you need it to.
Third, not benchmarking the quantized model against the original. Quantization is lossy in principle. The only way to know how lossy for your model and your workload is to measure. Run the same eval suite — GSM8K, MMLU, your domain benchmark, a perplexity check — on the FP16 original AND the quantized variant. If the delta is within your tolerance, ship. If it is not, step up a bitrate or change formats. Shipping a quantized model you never measured is flying blind.
Fourth, treating quantization as part of training. Trying to quantize-aware train when you do not need to, or folding the deployment quant into the training loop unnecessarily. Layer four is downstream of layer three. Train in full precision or QLoRA, merge, THEN quantize the merged checkpoint for export. Keep the layers separate.
[SLIDE 12 — What you can now do]
You can now state the format decision matrix and, for a given deployment target, name the right format and defend it. You can explain the quality-size trade-off and why Q4 is the sweet spot. You can run the three conversion workflows — GGUF, AWQ, MLX — from one source checkpoint. You can explain Unsloth Dynamic two-point-oh and why per-layer quantization beats uniform. And you can place quantization correctly in the stack: Layer four, downstream of training, compressing what you built without changing what it learned.
That is the export half. Next, FT twenty — Serving Stacks. We pick up where this module leaves off: the runtimes — vLLM, Ollama, TGI — that consume the formats you just learned to produce. Because a quantized file is useless until something serves it. Let's find out what.
End of module FT19. Duration: approximately forty-five minutes at one-hundred-forty words per minute.
# Teaching Script — Module FT19: Quantization Formats **Course**: Course 3 — LLM Fine-Tuning Masterclass **Module**: FT19 — Quantization Formats **Duration**: ~45 minutes (spoken at ~140 wpm) **Format**: Verbatim transcript with `[SLIDE N]` cues. Read aloud or use as speaker notes. --- [SLIDE 1 — Title] Welcome to module FT nineteen — Quantization Formats. This is the first module of pillar six, Deploy. We have spent five pillars steering the model — base, data, adapter, alignment. Now we cross into the export layer. You have a fine-tuned checkpoint. It works. It is also, in its native sixteen-bit format, far too large to deploy cheaply. A seven-billion-parameter model in FP16 is fourteen gigabytes. A seventy-billion model is one hundred and forty. Those numbers gate who can run your model and where. This module is how you compress it without wrecking what you spent five pillars building. The thesis in one sentence: the format is determined by where the model will run. Pick the runtime, then the format. Default to four-bit. Always benchmark against the original. [SLIDE 2 — Where it sits: Layer 4] First, the position. Quantization is Layer four — the Export. It sits downstream of Layer three, the Steer, which is everything you just finished in pillars three through five. The defining property, from FT zero zero: Layer four is downstream of training. You quantize AFTER you fine-tune. It compresses what training produced. It does not change learned behavior at sensible bitrates — it changes how compactly and quickly the model runs. This is related to but distinct from QLoRA, which you met in FT zero eight. QLoRA quantizes the base DURING training as a memory trick, so the forward and backward passes fit in VRAM. The adapter stays high-precision. That is quantize-to-train. FT nineteen is quantize-to-serve: you take the merged checkpoint and compress it for deployment. Even a QLoRA-trained model gets a fresh deployment-time quantization at export. One caveat to be honest about. Aggressive quantization does introduce a small amount of error. A four-bit weight is an approximation of the sixteen-bit original. At sensible bitrates — four-bit and above — that error is small enough to be near-imperceptible on most benchmarks. At aggressive bitrates — two and three bit — it is not, and you will see quality drop. So quantization does not re-train behavior, but a bad quantization choice can degrade it. The job of this module is to keep you on the right side of that line. [SLIDE 3 — The format decision matrix] Now the matrix. The single most important fact about quantization formats: the format is determined by the inference engine that will run the model. You do not pick a format in the abstract. You pick the runtime, then the format that runtime consumes. Read the slide as: if the deployment target is X, the format is Y. Local, CPU, mixed CPU plus GPU, Mac via Ollama — that is GGUF. The universal local format. NVIDIA GPU production serving via vLLM or TGI — that is AWQ, specifically AWQ-Marlin. Maximum quality at low bitrate in the ExLlamaV2 ecosystem — that is EXL2. Apple Silicon native — that is MLX. VRAM-rich Hopper or Blackwell where you want high quality — FP8. And the twenty twenty-five Blackwell-native frontier, four-bit — that is MXFP4 or NVFP4. Let me walk each one. [SLIDE 4 — GGUF and the local formats] GGUF is the file format consumed by llama dot cpp and everything built on it — Ollama, LM Studio, llamafile, KoboldCpp. If you are running a model on a laptop, a consumer desktop with mixed CPU-GPU offload, or a Mac via Metal, you are almost certainly running GGUF. Its defining property is maximum compatibility. A single dot-g-g-u-f file is self-contained — weights, tokenizer, config, chat template, all in one. No other format matches its portability. The GGUF sweet spot is Q4 underscore K underscore M. The K-quant scheme: the number is target bits per weight, the underscore M means medium. Q4 K M is roughly four bits per parameter, about seventy-five percent size reduction with minimal perceptible quality loss. A seven-billion model goes from fourteen gigabytes to about four point four. That is the number to reach for first. Move to Q5 or Q8 only with VRAM to spare; drop to Q3 or Q2 only if desperate. [SLIDE 5 — The NVIDIA serving formats] For NVIDIA GPU production serving, AWQ is the format. Activation-aware Weight Quantization, and the variant that matters is AWQ-Marlin — Marlin is a highly optimized kernel, first-class in vLLM. AWQ beats naive four-bit because it is activation-aware: a calibration set identifies the one percent of salient weights that matter most for output quality, and keeps those high-precision while quantizing the rest. The result: four-bit AWQ has noticeably better perplexity than four-bit GPTQ at the same size. AWQ-Marlin in vLLM is the default recommendation for new server deployments. GPTQ is the older four-bit format — mature, ubiquitous, slightly worse quality than AWQ at four-bit, showing its age, but everywhere and well-tested. If GPTQ is what your tooling supports, it is defensible. If you have a choice, prefer AWQ. [SLIDE 6 — EXL2, MLX, and the frontier formats] Three more, quickly. EXL2 is ExLlamaV2's format, and it does variable-rate per-layer quantization — each layer gets a different average bitrate based on its measured sensitivity, so the total hits a target average while sensitive layers stay high-precision. The payoff: best-in-class perplexity at small sizes. If you must squeeze a seventy-billion model into the smallest footprint without wrecking quality, EXL2 does it. The cost: ExLlamaV2-only runtime. Specialist's choice. MLX is Apple Silicon-native. It exploits unified memory — no CPU-to-GPU copy — giving the best inference performance on a Mac. mlx-lm and LM Studio load it. The recommended quant is four-bit group, same sweet spot as GGUF's Q4 K M. FP8 is the high-quality option when VRAM allows — a two-times size and speed win over FP16 with negligible quality loss, hardware-accelerated on Hopper and Blackwell. If the model fits in FP8 on H one hundreds, it is often the right call. And the twenty twenty-five frontier: MXFP4 and NVFP4. Both are E two M one four-bit floating-point — one sign, two exponent, one mantissa. The difference is block size: MXFP4 uses thirty-two elements per shared scaling block; NVFP4 uses sixteen, which is finer-grained and gives better accuracy at the same four-bit width. Both are Blackwell-native, both support training and inference. These make four-bit training, not just inference, practical at frontier scale this year. [SLIDE 7 — The quality/size trade-off] Now the curve. One chart to memorize. Quantization is a trade between size and speed on one side, quality on the other. The curve is not linear, and the shape of the non-linearity is the whole story. Read left to right. Q2 — about eighty-eight percent smaller — noticeable degradation. More errors, worse reasoning, occasional incoherence. Reach for it only when desperate. Q3 — about eighty-four percent smaller — measurable degradation, acceptable for some tasks. Q4 — about seventy-five percent smaller — the sweet spot. Minimal perceptible loss. This is where you start and usually where you stop. Q5 — seventy percent smaller — near-lossless. Q8 — fifty-eight percent smaller — effectively lossless, indistinguishable from FP16 for most purposes. Then FP16, the baseline. The principle: Q4 is the general sweet spot. At roughly four bits per parameter you get about seventy-five percent size reduction versus FP16, and on most benchmarks the quality loss is small — a fraction of a point on MMLU, a small perplexity uptick. Go lower and you save more memory but the degradation becomes noticeable. Go higher and you approach lossless, but you pay in size. When in doubt, take the K-quant underscore M. [SLIDE 8 — The conversion workflows] Three formats you will actually convert to. The load-bearing point: one source checkpoint, three export artifacts. You train once in FP16, then convert the same merged checkpoint to as many formats as your targets require. GGUF. The canonical path uses llama dot cpp's convert script — point it at your checkpoint, it emits a dot-g-g-u-f, then you quantize with llama-quantize. Two commands. Unsloth provides a one-shot alternative that handles convert, quantize, and even push to Ollama in a few lines. Lowest-friction path to a local model. AWQ. AutoAWQ is the library. It loads your FP16 checkpoint, runs a calibration pass to find the salient weights, and writes the quantized model in a format vLLM loads directly — on Hopper and Blackwell it auto-routes to the Marlin kernel. MLX. mlx-lm converts a Hugging Face checkpoint to MLX format with a single command. The result loads in mlx-lm, in LM Studio, or anywhere the MLX runtime is used. [SLIDE 9 — Unsloth Dynamic 2.0] A refinement worth knowing. A uniform quantization — standard GGUF Q4 K M, standard AWQ four-bit — applies essentially the same bitrate across the whole model. But not all layers are equally sensitive. Some layers — often attention projection layers, or specific MLP blocks — have weight distributions that degrade badly when compressed to four-bit. Others compress cleanly to three-bit or below with no measurable effect. Unsloth Dynamic two-point-oh does what EXL2 does at export time, but as a general-purpose strategy: it measures each layer's sensitivity and assigns bitrates accordingly. Sensitive layers stay at higher precision — five, six, or eight-bit. Less sensitive layers drop to three-bit or lower. The total model hits the same average size as a uniform Q4, but the quality is higher because the bits went where they matter. The empirical result: Dynamic two-point-oh beats uniform quants on quality at the same file size. It is the same insight as EXL2's variable-rate quantization, packaged for the conversion workflow you already use. If you have the option, it is strictly preferable to a naive Q4 for the formats that support it. [SLIDE 10 — Why after training] Let me state the layer-assignment judgment explicitly, because it is the one the course keeps returning to. You quantize after training for three reasons. One: it compresses what training produced. The trained weights are the artifact you want to deploy. Quantization makes that artifact smaller and faster. It is a transform on the final weights, not a step in the optimization. Two: it does not change learned behavior at sensible bitrates. The model's steering — the format, preferences, reasoning activation, compliance that layers two and three produced — is preserved. Quantization at Q4 and above perturbs the weights slightly but does not move the behavior across any decision boundary that matters. Three: it is cheap and repeatable. A quantization pass takes minutes, not the hours or days of a training run. You can quantize the same checkpoint to GGUF for local, AWQ for server, MLX for Mac — all from one source checkpoint, no retraining. That modularity is the point. [SLIDE 11 — Anti-patterns] Four anti-patterns to leave with. First, quantizing to Q2 for size and being surprised by quality loss. The most common deployment mistake. A team needs the model smaller, reaches for the smallest available quant, and is then confused when the model that scored well in FP16 now produces worse output. Q2 is a desperate measure, not a default. Start at Q4. Only go lower with eyes open and a benchmark comparing the quantized model to the original. Second, mixing formats carelessly. Loading an AWQ checkpoint into a GGUF-only runtime, or trying to serve a GPTQ model where vLLM expects AWQ-Marlin. The format must match the runtime. Decide the deployment target first — that is FT twenty — then the format that target consumes, then convert. Do not pick a format because it produced the smallest file and then discover it does not run where you need it to. Third, not benchmarking the quantized model against the original. Quantization is lossy in principle. The only way to know how lossy for your model and your workload is to measure. Run the same eval suite — GSM8K, MMLU, your domain benchmark, a perplexity check — on the FP16 original AND the quantized variant. If the delta is within your tolerance, ship. If it is not, step up a bitrate or change formats. Shipping a quantized model you never measured is flying blind. Fourth, treating quantization as part of training. Trying to quantize-aware train when you do not need to, or folding the deployment quant into the training loop unnecessarily. Layer four is downstream of layer three. Train in full precision or QLoRA, merge, THEN quantize the merged checkpoint for export. Keep the layers separate. [SLIDE 12 — What you can now do] You can now state the format decision matrix and, for a given deployment target, name the right format and defend it. You can explain the quality-size trade-off and why Q4 is the sweet spot. You can run the three conversion workflows — GGUF, AWQ, MLX — from one source checkpoint. You can explain Unsloth Dynamic two-point-oh and why per-layer quantization beats uniform. And you can place quantization correctly in the stack: Layer four, downstream of training, compressing what you built without changing what it learned. That is the export half. Next, FT twenty — Serving Stacks. We pick up where this module leaves off: the runtimes — vLLM, Ollama, TGI — that consume the formats you just learned to produce. Because a quantized file is useless until something serves it. Let's find out what. --- *End of module FT19. Duration: approximately forty-five minutes at one-hundred-forty words per minute.*