{ "module": "FT19 — Quantization Formats", "course": "3 — LLM Fine-Tuning Masterclass", "version": "1.0.0", "duration_minutes": 45, "total_questions": 15, "bloom_distribution": { "target": "20% recall / 40% application / 40% analysis-design", "actual": { "recall": 3, "application": 6, "analysis": 6 } }, "passing_score_percent": 70, "questions": [ { "id": "Q01", "bloom": "recall", "type": "multiple_choice", "prompt": "Which layer of the Steering Stack is quantization, and what is its defining property?", "options": [ "Layer 3 (the Steer). It is part of training and changes learned behavior.", "Layer 4 (the Export). It is downstream of training — it compresses what training produced; it does not change learned behavior at sensible bitrates.", "Layer 2 (the Adapter). It runs inside the LoRA matrices.", "Layer 5 (the Boundary). It is a harness concern." ], "answer_index": 1, "rationale": "Quantization is Layer 4 (Export), downstream of Layer 3 (training). You quantize AFTER you fine-tune. It is a transform on the final weights that changes compactness and speed, not the learned behavior (at Q4+ bitrates). Options A, C, and D misplace it in the stack." }, { "id": "Q02", "bloom": "recall", "type": "multiple_choice", "prompt": "What is the GGUF sweet-spot quant, and roughly what size reduction does it give versus FP16?", "options": [ "Q8_0 — about 50% smaller, lossless.", "Q2_K — about 88% smaller, near-lossless.", "Q4_K_M — about 75% smaller, minimal perceptible quality loss.", "FP8 — about 50% smaller, hardware-accelerated." ], "answer_index": 2, "rationale": "Q4_K_M is the community default: roughly 4 bits/parameter, ~75% size reduction vs FP16 with minimal perceptible quality loss (e.g., a 7B model goes from ~14GB to ~4.4GB). Q8 is larger (only ~58% smaller); Q2 degrades noticeably; FP8 is a different format family, not a GGUF quant." }, { "id": "Q03", "bloom": "recall", "type": "multiple_choice", "prompt": "What do MXFP4 and NVFP4 have in common, and how do they differ?", "options": [ "Both are 8-bit integer formats; they differ only in endianness.", "Both are E2M1 4-bit floating-point formats; they differ in block size for shared scaling — MXFP4 uses 32 elements/block, NVFP4 uses 16 (finer-grained, better accuracy).", "Both are GGUF variants; MXFP4 is for CPU, NVFP4 for GPU.", "Both are Apple-native; MXFP4 is older, NVFP4 is the 2025 replacement." ], "answer_index": 1, "rationale": "MXFP4 and NVFP4 are both 2025 Blackwell-native E2M1 4-bit floating-point formats (1 sign, 2 exponent, 1 mantissa). The difference is the block size for shared scaling factors: MXFP4 = 32 elements/block (the OCP Microscaling format); NVFP4 = 16 elements/block (NVIDIA's variant). NVFP4's finer-grained blocks yield better accuracy at the same 4-bit width." }, { "id": "Q04", "bloom": "application", "type": "multiple_choice", "prompt": "You are deploying a fine-tuned model to a local developer laptop that must run on CPU with mixed CPU/GPU offload, via Ollama. Which format do you convert to?", "options": [ "AWQ 4-bit — it has the best serving kernels.", "GGUF Q4_K_M — it is the universal local format consumed by llama.cpp/Ollama and runs on CPU/mixed offload.", "MLX 4-bit — Apple Silicon native.", "FP8 — the high-quality option." ], "answer_index": 1, "rationale": "The format is determined by the runtime. Ollama/llama.cpp consumes GGUF, and GGUF is the only one of the four that runs well on CPU and mixed CPU/GPU offload. AWQ needs CUDA kernels for production speed; MLX needs Apple Silicon; FP8 needs Hopper/Blackwell and is a server format. GGUF's portability is exactly its strength for local CPU targets." }, { "id": "Q05", "bloom": "application", "type": "multiple_choice", "prompt": "You are deploying to an H100 vLLM server serving many concurrent users. You want the best speed/quality ratio. Which format?", "options": [ "GGUF Q4_K_M — maximum compatibility.", "AWQ 4-bit (AWQ-Marlin) — first-class in vLLM with optimized kernels, best speed/quality for server inference.", "EXL2 — best perplexity at small sizes.", "GPTQ — the most mature format." ], "answer_index": 1, "rationale": "AWQ-Marlin is the default recommendation for NVIDIA production serving in vLLM/TGI. AWQ's activation-aware quantization gives better perplexity than GPTQ at 4-bit, and the Marlin kernel is highly optimized for server throughput. GGUF's portability is irrelevant on a fixed vLLM box; EXL2's vLLM support is experimental; GPTQ is fine but slightly worse quality than AWQ." }, { "id": "Q06", "bloom": "application", "type": "multiple_choice", "prompt": "You are deploying an on-device assistant to a Mac Studio (M-series) and want the best inference performance on that hardware. Which format?", "options": [ "GGUF Q4_K_M via llama.cpp's Metal backend.", "AWQ 4-bit via vLLM.", "MLX 4-bit via mlx-lm / LM Studio.", "FP8 via the Mac's GPU." ], "answer_index": 2, "rationale": "MLX is Apple Silicon-native and exploits the unified memory architecture (no CPU-to-GPU copy), giving the best inference performance on a Mac. mlx-lm and LM Studio load MLX-format models directly. GGUF-on-Metal works but is slower than MLX on Apple hardware. AWQ and FP8 are NVIDIA-server formats." }, { "id": "Q07", "bloom": "application", "type": "multiple_choice", "prompt": "You need to squeeze a 70B model into the smallest possible footprint while preserving quality, and you fully control the deployment runtime. Which format do you reach for?", "options": [ "GGUF Q2_K — smallest file size.", "AWQ 4-bit — server-friendly.", "EXL2 — variable-rate per-layer quantization gives best-in-class perplexity at small sizes.", "FP8 — highest quality." ], "answer_index": 2, "rationale": "EXL2's variable-rate per-layer quantization measures each layer's sensitivity and assigns bitrates accordingly, yielding best-in-class perplexity at small total sizes — exactly what you need to fit a 70B into the smallest footprint without wrecking quality. Because you control the runtime, EXL2's ExLlamaV2-only limitation is acceptable. GGUF Q2_K is smaller but degrades noticeably; AWQ and FP8 are not size-optimized." }, { "id": "Q08", "bloom": "application", "type": "multiple_choice", "prompt": "You have an H100 with VRAM to spare and want to preserve maximum quality while still getting a size/speed win over FP16. Which format?", "options": [ "GGUF Q2_K — smallest.", "FP8 — 2x size/speed win over FP16 with negligible quality loss and hardware acceleration on Hopper.", "GPTQ 4-bit — most mature.", "EXL2 3-bit — max compression." ], "answer_index": 1, "rationale": "FP8 is the high-quality serving option when VRAM allows: a 2x size/speed win over FP16 with negligible quality loss, and H100 has hardware FP8 acceleration. It is the highest-quality compressed option short of staying in FP16. The Q4/Q2/EXL2 options trade more quality for smaller size, which you do not need given the VRAM headroom." }, { "id": "Q09", "bloom": "application", "type": "multiple_choice", "prompt": "A team quantized their model to Q2_K to fit a small edge device and is surprised the output quality dropped noticeably. What went wrong, and what should they do first?", "options": [ "Nothing — Q2 is always near-lossless; the model must be buggy.", "Q2 is in the 'noticeable degradation' zone — not a default. Step up to Q4_K_M (the sweet spot), re-measure against FP16, and only go below Q4 with eyes open.", "Re-train the model at Q2 to recover quality.", "Switch to AWQ, which is always lossless at any bitrate." ], "answer_index": 1, "rationale": "Q2 is a desperate measure for VRAM-constrained situations, not a default — it sits in the 'noticeable degradation' zone of the quality/size curve (more errors, worse reasoning). The fix is to step up to Q4_K_M and benchmark the quant against FP16. No format is lossless at every bitrate; re-training at Q2 conflates training and export (wrong layer); AWQ at 4-bit is near-lossless but not at any bitrate." }, { "id": "Q10", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why is quantization described as 'downstream of training' rather than part of it? What does it preserve at sensible bitrates?", "options": [ "It is part of training — quantization re-optimizes the weights to a new behavior.", "It is a TRANSFORM on the final trained weights, applied after training. At Q4+ it perturbs weights slightly but does not move behavior across any decision boundary that matters — the model's steering (format, preferences, compliance) is preserved. It changes compactness/speed, not learned behavior.", "It preserves the LoRA adapter separately from the base.", "It is downstream because it runs on a different GPU than training." ], "answer_index": 1, "rationale": "Layer 4 compresses the artifact Layer 3 produced; it is a transform on the final weights, not a step in the optimization. At sensible bitrates (Q4+) the perturbation is small enough that the model's learned behavior — the steering — is preserved. It changes how compactly and quickly the model runs, not what it learned. Option A conflates it with training; C describes adapter modularity (Layer 2); D is incidental, not definitional." }, { "id": "Q11", "bloom": "analysis", "type": "multiple_choice", "prompt": "A team loaded an AWQ checkpoint into an Ollama instance and it failed to run. What is the root cause, and what is the correct fix?", "options": [ "The AWQ file was corrupted — re-export it.", "Format/runtime mismatch. AWQ is consumed by vLLM/TGI on NVIDIA GPUs; Ollama runs GGUF. The format must match the runtime. Fix: convert the checkpoint to GGUF Q4_K_M for the Ollama target (and keep the AWQ copy for the vLLM server).", "Ollama does not support quantized models — use the FP16.", "AWQ requires a newer Ollama version." ], "answer_index": 1, "rationale": "This is the 'mixing formats carelessly' anti-pattern. The format is determined by the runtime: Ollama/llama.cpp consumes GGUF; vLLM/TGI consume AWQ. Decide the deployment target first, then convert to the format that target consumes. The fix is to produce a GGUF copy for the Ollama instance (cheap and repeatable — no retraining). Corruption, FP16 fallback, and version bumps do not address the real mismatch." }, { "id": "Q12", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why does per-layer (variable-rate) quantization — EXL2, Unsloth Dynamic 2.0 — beat uniform quantization at the same average file size?", "options": [ "Because per-layer quantization uses a newer algorithm.", "Because not all layers are equally sensitive. Some layers (often attention projections, certain MLP blocks) degrade badly at 4-bit; others compress cleanly to 3-bit with no measurable effect. Per-layer allocation puts bits where the quality cost of compression is highest, so the same total bitrate yields lower perplexity than uniform.", "Because per-layer quantization always uses 8-bit for every layer.", "Because uniform quantization is mathematically incorrect." ], "answer_index": 1, "rationale": "The mechanism is sensitivity heterogeneity: layers differ in how much quality they lose when compressed. Uniform quantization wastes bits on tolerant layers and starves sensitive ones. Per-layer (variable-rate) quantization measures each layer's sensitivity and assigns bitrates so the bits go where compression is most costly — yielding lower perplexity at the same total size. It is not about recency, universal 8-bit, or uniform being wrong; it is about allocating the bitrate budget where it matters." }, { "id": "Q13", "bloom": "analysis", "type": "multiple_choice", "prompt": "How does QLoRA's quantization (FT08) differ from FT19's deployment quantization? Why are they different concerns?", "options": [ "They are the same — both quantize for deployment.", "QLoRA quantizes the base DURING training as a memory trick to fit forward/backward passes in VRAM (the adapter stays high-precision). FT19 quantizes the merged checkpoint AFTER training as the deployment export. Different points in the pipeline: QLoRA is 'quantize to train'; FT19 is 'quantize to serve.'", "QLoRA produces GGUF; FT19 produces AWQ.", "QLoRA is lossless; FT19 is lossy." ], "answer_index": 1, "rationale": "QLoRA's training-time quant is a memory trick: the frozen base is 4-bit so the optimizer state fits, while the adapter trains in high precision. FT19's deployment quant is the export step applied to the merged checkpoint after training. They are different concerns at different points in the pipeline. Even a QLoRA-trained model gets a fresh deployment-time quantization at export. Options A, C, D mischaracterize the relationship." }, { "id": "Q14", "bloom": "analysis", "type": "multiple_choice", "prompt": "AWQ and GPTQ are both 4-bit post-training quantization formats. Why is AWQ generally preferred over GPTQ for new NVIDIA server deployments, while GPTQ remains 'defensible'?", "options": [ "AWQ is newer, so it must be better.", "AWQ is activation-aware: it uses a calibration set to keep the ~1% of salient weights at higher precision, yielding lower perplexity than GPTQ at the same 4-bit size. GPTQ is more mature and ubiquitous (nearly every model has a GPTQ variant) and is fine where it is what tooling supports — but at 4-bit AWQ has the quality edge.", "GPTQ is illegal on H100s.", "AWQ produces smaller files than GPTQ by a factor of 2." ], "answer_index": 1, "rationale": "AWQ's activation-awareness (calibration to find and preserve salient weights) gives it a quality edge over GPTQ at the same 4-bit size — that is the mechanistic reason to prefer it for new deployments. GPTQ remains defensible because it is mature, ubiquitous, and well-tested; where it is what existing tooling supports or what is already published, it is fine. Recency, legality, and 2x size differences are not the reasons." }, { "id": "Q15", "bloom": "analysis", "type": "multiple_choice", "prompt": "A team ships a quantized model without benchmarking it against the FP16 original. Users report subtle quality regressions the team did not anticipate. What is the core error, and what is the correct process?", "options": [ "The error is quantizing at all — they should have shipped FP16.", "The error is the 'not benchmarking the quantized model against the original' anti-pattern. Quantization is lossy in principle; the only way to know how lossy for your model is to measure. Correct process: run the same eval (GSM8K, MMLU, domain benchmark, perplexity) on FP16 AND the quant; if the delta exceeds tolerance, step up a bitrate (Q4 -> Q5 -> Q8) or change format.", "The error is using Q4 — Q8 is always required.", "The error is not retraining after quantization." ], "answer_index": 1, "rationale": "Shipping an unmeasured quant is flying blind — quantization is lossy in principle and the loss is model/workload-specific. The correct process is to benchmark the quant against the FP16 original and step up bitrate or change format if the delta exceeds tolerance. Quantizing is correct (the point of FT19); Q4 is the sweet spot, not a bug; retraining is the wrong layer (Layer 4 does not retrain)." } ] }