Lab Specification — Module FT19: Quantization Formats

# Shared venv for measurement (perplexity, benchmark, size)
python3.11 -m venv ft19-measure-env && source ft19-measure-env/bin/activate
pip install -q transformers accelerate torch datasets evaluate
deactivate

# GGUF path: clone llama.cpp and build the quantize tool
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && pip install -r requirements/requirements-convert_hf_to_gguf.txt
# Build llama-quantize (macOS: make; Linux w/ CUDA: make GGML_CUDA=1)
make
cd ..

# AWQ path
python3.11 -m venv ft19-awq-env && source ft19-awq-env/bin/activate
pip install -q autoawq transformers
deactivate

# MLX path (Apple Silicon only; skip on CUDA boxes — you will use a prebuilt MLX instead)
python3.11 -m venv ft19-mlx-env && source ft19-mlx-env/bin/activate
pip install -q mlx-lm
deactivate

# merge_adapter.py — run in ft19-measure-env
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

BASE_ID = "Qwen/Qwen2.5-1.5B"                 # the base you fine-tuned from
ADAPTER_PATH = "./ft11-lora-out/checkpoint-XXX"  # your FT11 adapter
MERGED_PATH  = "./ft19-source-model"

base = AutoModelForCausalLM.from_pretrained(BASE_ID, torch_dtype=torch.bfloat16)
model = PeftModel.from_pretrained(base, ADAPTER_PATH)
model = model.merge_and_unload()                       # Layer 2 merges into Layer 1
model.save_pretrained(MERGED_PATH, safe_serialization=True)

tok = AutoTokenizer.from_pretrained(BASE_ID)
tok.save_pretrained(MERGED_PATH)
print("Merged checkpoint saved to", MERGED_PATH)

# measure_baseline.py — run in ft19-measure-env
import os, torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

SOURCE = "./ft19-source-model"
size_bytes = sum(os.path.getsize(os.path.join(SOURCE,f)) for f in os.listdir(SOURCE) if f.endswith(".safetensors"))
print(f"FP16 size: {size_bytes/1e9:.2f} GB")

tok = AutoTokenizer.from_pretrained(SOURCE)
model = AutoModelForCausalLM.from_pretrained(SOURCE, torch_dtype=torch.bfloat16, device_map="auto")

# Perplexity on a small slice of wikitext (proxy for quality)
data = load_dataset("wikitext","wikitext-2-raw-v1", split="test")["text"]
enc = tok("\n\n".join(data), return_tensors="pt").to(model.device)
with torch.no_grad():
    out = model(**enc, labels=enc["input_ids"])
print(f"FP16 perplexity: {torch.exp(out.loss).item():.3f}")

cd llama.cpp
# Step 1: HF checkpoint -> F16 GGUF
python convert_hf_to_gguf.py ../ft19-source-model --outfile ../model-f16.gguf

# Step 2: quantize to the sweet spot
./llama-quantize ../model-f16.gguf ../model-Q4_K_M.gguf Q4_K_M
cd ..
ls -lh model-Q4_K_M.gguf

# llama.cpp's bundled benchmark; reports tokens/sec
./llama.cpp/llama-bench -m model-Q4_K_M.gguf -p 128 -n 128

./llama.cpp/llama-perplexity -m model-Q4_K_M.gguf -f wikitext-2-raw-test.txt 2>&1 | tail -5

# awq_quantize.py — run in ft19-awq-env (CUDA required)
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

SOURCE = "./ft19-source-model"
OUT    = "./ft19-awq-model"

quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }

model = AutoAWQForCausalLM.from_pretrained(SOURCE)
tokenizer = AutoTokenizer.from_pretrained(SOURCE)

# AutoAWQ runs a calibration pass internally (default calibration set)
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(OUT)
tokenizer.save_pretrained(OUT)
print("AWQ model saved to", OUT)

pip install -q vllm   # in the awq env, or a fresh venv

# awq_measure.py
from vllm import LLM, SamplingParams
import os

OUT = "./ft19-awq-model"
llm = LLM(model=OUT, quantization="awq_marlin", dtype="float16", gpu_memory_utilization=0.8)
sp = SamplingParams(max_tokens=128, temperature=0.0)

# Speed: generate 128 tokens across a batch of prompts
prompts = ["The capital of France is"] * 8
import time
t0 = time.time()
outs = llm.generate(prompts, sp)
dt = time.time() - t0
total_tokens = sum(len(o.outputs[0].token_ids) for o in outs)
print(f"AWQ tokens/sec: {total_tokens/dt:.1f}")

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained("./ft19-awq-model", device_map="auto")
# ... reuse the Phase 1 perplexity block, pointing at the AWQ dir

source ft19-mlx-env/bin/activate
python -m mlx_lm.convert \
    --hf-path ./ft19-source-model \
    --quantize --q-bits 4 \
    --mlx-path ./ft19-mlx-model
deactivate
ls -lh ft19-mlx-model

# mlx_measure.py — run on the Mac, in the mlx env
import mlx_lm, time
model, tokenizer = mlx_lm.load("./ft19-mlx-model")

prompts = ["The capital of France is"] * 8
t0 = time.time()
for p in prompts:
    mlx_lm.generate(model, tokenizer, prompt=p, max_tokens=128, verbose=False)
dt = time.time() - t0
print(f"MLX tokens/sec (per-prompt avg): {128*8/dt:.1f}")

# Lab Specification — Module FT19: Quantization Formats

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT19 — Quantization Formats
**Lab title**: Quantize Three Ways
**Duration**: 3–5 hours (the core lab of Pillar 6 — produces a defensible per-target recommendation)
**Environment**: Python 3.11+. A consumer NVIDIA GPU (RTX 4090 / 24GB) OR a rented A10g/A100 (RunPod, Lambda, Colab Pro) for the AWQ path; Apple Silicon (M-series Mac) for the MLX path; CPU-only works for the GGUF path. ~25GB free disk for the base + three export artifacts.

---

## Learning objectives

By the end of this lab you will have:

1. **Taken the fine-tuned model you produced in FT11** and converted it to **three deployment formats**: GGUF Q4_K_M, AWQ 4-bit, and MLX 4-bit.
2. **Measured each format** on three axes: on-disk size, inference speed (tokens/sec), and quality (perplexity + a benchmark).
3. **Compared all three to the FP16 original** and confirmed (or refuted) the claim that Q4 is a near-lossless sweet spot.
4. **Produced a per-deployment-target recommendation** — given a local-CPU target, a Mac target, and an NVIDIA-server target, which format ships, and why, defended with your own numbers.

This lab turns the format decision matrix from a table you read into a table you *measured*. The deliverable is not "three quantized files" — it is a defensible recommendation grounded in your own measurements.

> **Prerequisite.** This lab assumes you have a fine-tuned checkpoint from FT11. If you do not, use a small openly-licensed instruct model as a stand-in (e.g., `Qwen/Qwen2.5-1.5B-Instruct`) and note that you are using an un-fine-tuned base. The mechanics are identical; only the "is this your steering preserved?" question differs.

---

## Phase 0 — Environment setup (20 min)

You need three separate environments because the format toolchains do not all coexist cleanly. Set up one venv per path; this also mirrors how you would do it in production (the conversion happens wherever the target toolchain lives, not on the training box necessarily).

```bash
# Shared venv for measurement (perplexity, benchmark, size)
python3.11 -m venv ft19-measure-env && source ft19-measure-env/bin/activate
pip install -q transformers accelerate torch datasets evaluate
deactivate
```

```bash
# GGUF path: clone llama.cpp and build the quantize tool
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && pip install -r requirements/requirements-convert_hf_to_gguf.txt
# Build llama-quantize (macOS: make; Linux w/ CUDA: make GGML_CUDA=1)
make
cd ..
```

```bash
# AWQ path
python3.11 -m venv ft19-awq-env && source ft19-awq-env/bin/activate
pip install -q autoawq transformers
deactivate
```

```bash
# MLX path (Apple Silicon only; skip on CUDA boxes — you will use a prebuilt MLX instead)
python3.11 -m venv ft19-mlx-env && source ft19-mlx-env/bin/activate
pip install -q mlx-lm
deactivate
```

> **Hardware note.** The AWQ path needs CUDA (AutoAWQ's GPU kernels). The MLX path needs Apple Silicon. The GGUF path runs anywhere. If you are on a single machine, do the paths it supports and use a published equivalent for the others (document this in the report). The point is to learn the *workflows* and *measurements*, not to own every piece of hardware.

---

## Phase 1 — Prepare the source checkpoint (15 min)

Ensure your FT11 checkpoint is a merged FP16/BF16 Hugging Face directory. If you trained with a LoRA adapter, merge it first.

```python
# merge_adapter.py — run in ft19-measure-env
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

BASE_ID = "Qwen/Qwen2.5-1.5B"                 # the base you fine-tuned from
ADAPTER_PATH = "./ft11-lora-out/checkpoint-XXX"  # your FT11 adapter
MERGED_PATH  = "./ft19-source-model"

base = AutoModelForCausalLM.from_pretrained(BASE_ID, torch_dtype=torch.bfloat16)
model = PeftModel.from_pretrained(base, ADAPTER_PATH)
model = model.merge_and_unload()                       # Layer 2 merges into Layer 1
model.save_pretrained(MERGED_PATH, safe_serialization=True)

tok = AutoTokenizer.from_pretrained(BASE_ID)
tok.save_pretrained(MERGED_PATH)
print("Merged checkpoint saved to", MERGED_PATH)
```

If you already merged in FT11, point everything at that directory and skip ahead. Record the path — it is `SOURCE_MODEL` for every conversion below.

### Measure the FP16 baseline (size + perplexity)

```python
# measure_baseline.py — run in ft19-measure-env
import os, torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

SOURCE = "./ft19-source-model"
size_bytes = sum(os.path.getsize(os.path.join(SOURCE,f)) for f in os.listdir(SOURCE) if f.endswith(".safetensors"))
print(f"FP16 size: {size_bytes/1e9:.2f} GB")

tok = AutoTokenizer.from_pretrained(SOURCE)
model = AutoModelForCausalLM.from_pretrained(SOURCE, torch_dtype=torch.bfloat16, device_map="auto")

# Perplexity on a small slice of wikitext (proxy for quality)
data = load_dataset("wikitext","wikitext-2-raw-v1", split="test")["text"]
enc = tok("\n\n".join(data), return_tensors="pt").to(model.device)
with torch.no_grad():
    out = model(**enc, labels=enc["input_ids"])
print(f"FP16 perplexity: {torch.exp(out.loss).item():.3f}")
```

**Record:** `FP16_SIZE_GB` and `FP16_PPL`. These are the reference numbers every quantized variant gets compared against.

---

## Phase 2 — Path A: GGUF Q4_K_M (45 min)

### 2a. Convert + quantize

```bash
cd llama.cpp
# Step 1: HF checkpoint -> F16 GGUF
python convert_hf_to_gguf.py ../ft19-source-model --outfile ../model-f16.gguf

# Step 2: quantize to the sweet spot
./llama-quantize ../model-f16.gguf ../model-Q4_K_M.gguf Q4_K_M
cd ..
ls -lh model-Q4_K_M.gguf
```

Record `GGUF_SIZE_GB`.

### 2b. Measure speed

```bash
# llama.cpp's bundled benchmark; reports tokens/sec
./llama.cpp/llama-bench -m model-Q4_K_M.gguf -p 128 -n 128
```

Record `GGUF_TOKENS_PER_SEC` (the `tg128` column is generation speed).

### 2c. Measure quality (perplexity)

llama.cpp can compute perplexity directly on the GGUF:

```bash
./llama.cpp/llama-perplexity -m model-Q4_K_M.gguf -f wikitext-2-raw-test.txt 2>&1 | tail -5
```

(Download a plain-text wikitext-2 test file, or extract it from the dataset in Phase 1.) Record `GGUF_PPL`.

> **Teaching moment:** GGUF perplexity should be very close to the FP16 baseline (typically within 1–3% at Q4_K_M). If you see a large gap, you likely hit a known-bad layer; try Q5_K_M and re-measure. This is the "Q4 is near-lossless" claim, verified on your own model.

---

## Phase 3 — Path B: AWQ 4-bit (60 min)

### 3a. Quantize with AutoAWQ

```python
# awq_quantize.py — run in ft19-awq-env (CUDA required)
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

SOURCE = "./ft19-source-model"
OUT    = "./ft19-awq-model"

quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }

model = AutoAWQForCausalLM.from_pretrained(SOURCE)
tokenizer = AutoTokenizer.from_pretrained(SOURCE)

# AutoAWQ runs a calibration pass internally (default calibration set)
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(OUT)
tokenizer.save_pretrained(OUT)
print("AWQ model saved to", OUT)
```

Record `AWQ_SIZE_GB`.

### 3b. Measure speed + quality (via vLLM or transformers)

The cleanest speed measurement is through vLLM (which is where AWQ-Marlin runs in production). If vLLM is not available, fall back to the transformers AWQ loader — slower, but the perplexity number is still valid.

```bash
pip install -q vllm   # in the awq env, or a fresh venv
```

```python
# awq_measure.py
from vllm import LLM, SamplingParams
import os

OUT = "./ft19-awq-model"
llm = LLM(model=OUT, quantization="awq_marlin", dtype="float16", gpu_memory_utilization=0.8)
sp = SamplingParams(max_tokens=128, temperature=0.0)

# Speed: generate 128 tokens across a batch of prompts
prompts = ["The capital of France is"] * 8
import time
t0 = time.time()
outs = llm.generate(prompts, sp)
dt = time.time() - t0
total_tokens = sum(len(o.outputs[0].token_ids) for o in outs)
print(f"AWQ tokens/sec: {total_tokens/dt:.1f}")
```

For perplexity, load via transformers in the measure env:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained("./ft19-awq-model", device_map="auto")
# ... reuse the Phase 1 perplexity block, pointing at the AWQ dir
```

Record `AWQ_TOKENS_PER_SEC` and `AWQ_PPL`.

---

## Phase 4 — Path C: MLX 4-bit (40 min)

> Skip this phase if you are not on Apple Silicon. Instead, download a published `mlx-community` 4-bit model of comparable size and measure *that* as a stand-in, noting in the report that you did not convert your own checkpoint for the MLX path.

### 4a. Convert with mlx-lm

```bash
source ft19-mlx-env/bin/activate
python -m mlx_lm.convert \
    --hf-path ./ft19-source-model \
    --quantize --q-bits 4 \
    --mlx-path ./ft19-mlx-model
deactivate
ls -lh ft19-mlx-model
```

Record `MLX_SIZE_GB`.

### 4b. Measure speed + quality

```python
# mlx_measure.py — run on the Mac, in the mlx env
import mlx_lm, time
model, tokenizer = mlx_lm.load("./ft19-mlx-model")

prompts = ["The capital of France is"] * 8
t0 = time.time()
for p in prompts:
    mlx_lm.generate(model, tokenizer, prompt=p, max_tokens=128, verbose=False)
dt = time.time() - t0
print(f"MLX tokens/sec (per-prompt avg): {128*8/dt:.1f}")
```

For perplexity, MLX does not ship a one-liner; the practical proxy is to compute loss on a held-out text slice via `mlx_lm.evaluate` or compare against the GGUF/FP16 numbers using a small eval set. Record `MLX_PPL` (or note "see wikitext eval, delta vs FP16 within X%").

---

## Phase 5 — The comparison table (30 min)

Assemble your numbers. This is the deliverable's centerpiece.

| Format | Size (GB) | Size vs FP16 | Tokens/sec | Perplexity | PPL vs FP16 | Runtime |
| --- | --- | --- | --- | --- | --- | --- |
| **FP16 (original)** | _ | 1.0x | _ | _ | baseline | transformers/vLLM |
| **GGUF Q4_K_M** | _ | _ | _ | _ | _ | Ollama / llama.cpp |
| **AWQ 4-bit (Marlin)** | _ | _ | _ | _ | _ | vLLM / TGI |
| **MLX 4-bit** | _ | _ | _ | _ | _ | mlx-lm / LM Studio (Mac) |

### The per-target recommendation

State three deployment targets and name the format for each, defended with your numbers:

- **Target 1 — Local developer laptop (CPU-only, mixed offload):** format = ______. Defense: ______
- **Target 2 — Mac Studio M-series (unified memory):** format = ______. Defense: ______
- **Target 3 — NVIDIA production server (vLLM, concurrent users):** format = ______. Defense: ______

Example defense: *"For Target 3 I ship AWQ 4-bit Marlin. My AWQ variant was 0.91x the size of GGUF Q4_K_M but delivered 3.2x the tokens/sec on the A100, and its perplexity was within 1.4% of FP16. GGUF's portability is irrelevant on a fixed vLLM server; AWQ-Marlin's kernel speed is the load-bearing property there."*

The quality of the defense — not the choice itself — is what is graded.

---

## Deliverables

Submit `ft19-lab-report.md` containing:

- [ ] **Source checkpoint**: path, base model, whether it is your FT11 output or a stand-in. FP16 size + perplexity (the baseline).
- [ ] **Path A (GGUF)**: the two commands (convert + quantize), `Q4_K_M` file size, tokens/sec, perplexity.
- [ ] **Path B (AWQ)**: the AutoAWQ config (`w_bit`, `q_group_size`, `version`), file size, tokens/sec, perplexity. Note whether you measured via vLLM (Marlin) or transformers.
- [ ] **Path C (MLX)**: the `mlx_lm.convert` command, file size, tokens/sec, perplexity (or eval delta). If you used a stand-in model on non-Mac hardware, say so.
- [ ] **The comparison table**: filled in with your numbers.
- [ ] **The per-target recommendation**: three targets, one format each, 2–4 sentence defense per target using your numbers.
- [ ] **One honest surprise**: something your measurements showed that the module's generalities did not prepare you for (e.g., "AWQ was actually slower than GGUF on my consumer GPU because Marlin needs Hopper+" or "MLX 4-bit perplexity was closer to FP16 than either server format").

---

## Solution key

- **Sizes.** All three 4-bit variants should land in a similar band: roughly 25–30% of the FP16 size (a ~70–75% reduction). Exact numbers depend on tokenizer/config overhead and group size. GGUF Q4_K_M is typically the smallest of the three for a given model; MLX 4-bit is comparable; AWQ with `q_group_size=128` is slightly larger due to zero-points and metadata.
- **Speed.** The *rank order depends on hardware.* On Hopper/Blackwell GPUs, AWQ-Marlin should be fastest (optimized kernel, batched). On Apple Silicon, MLX should beat GGUF-on-Metal for generation throughput thanks to unified-memory exploitation. On CPU or mixed CPU/GPU consumer hardware, GGUF (llama.cpp) is the only one of the three that runs well — and that is precisely its strength. A student who reports "AWQ was slow" on a consumer GPU may have hit the Marlin kernel's architecture requirement — a legitimate, instructive finding.
- **Perplexity.** All three 4-bit variants should be within a few percent of FP16 (the Q4 sweet-spot claim). GGUF Q4_K_M and MLX 4-bit typically within ~1–2%; AWQ 4-bit within ~1–3%. If a student sees a large gap (>5%), suspect a calibration issue (AWQ) or a bad K-quant choice — have them try Q5_K_M / 8-bit group and re-measure. A clean confirmation of the sweet spot is the expected result.
- **The recommendation.** There is no single "best format" — the grade is whether the student mapped each target to the format that target's runtime consumes, defended with their own size/speed/quality numbers. Correct mapping: GGUF for local/CPU/Mac-via-Ollama; MLX for Mac-native; AWQ for vLLM/TGI on NVIDIA. A weak defense restates the matrix without the student's measurements; a strong defense cites specific cells.
- **Common failure modes to watch for:**
  - *AWQ perplexity is wildly off:* calibration failed or `version` mismatches the kernel. Check `version="GEMM"` and that the calibration set is reasonable. Try `GEMV` if `GEMM` is wrong for the target.
  - *GGUF is much slower than expected:* running on CPU without offload, or the wrong K-quant for the hardware. Check that GPU offload (`-ngl`) is set if a GPU is present.
  - *MLX path skipped on a non-Mac box:* that is fine, but the student must use a stand-in and say so. A blank MLX row is not acceptable.
  - *Perplexity identical across formats:* the student may have measured the FP16 by accident (pointed at the source dir). Check that each measurement points at the quantized artifact.

---

## Stretch goals

1. **Add a 5th column — Unsloth Dynamic 2.0 (or EXL2).** Convert the same checkpoint with a variable-rate quantizer and measure. Demonstrate (or refute) the claim that sensitivity-aware per-layer quant beats uniform Q4 on perplexity at the same file size.
2. **The K-quant sweep.** For the GGUF path, produce Q2_K, Q3_K_M, Q4_K_M, Q5_K_M, Q8_0 and plot size vs. perplexity. Find the knee of the curve on *your* model. This is the empirical version of the quality/size trade-off diagram.
3. **Cross-format quality probe.** Run a small task benchmark (GSM8K or MMLU via `lm-eval`) on FP16 vs each 4-bit variant. Confirm that perplexity deltas do (or do not) translate to benchmark deltas. Sometimes a small perplexity uptick is invisible on benchmarks; sometimes it is not.
4. **Round-trip your FT11 steering.** If your source checkpoint is a fine-tuned model with distinctive behavior (a format, a compliance profile), confirm that behavior survives quantization — generate from the FP16 and each quant on a prompt that exercises the steered behavior, and verify the quantized outputs preserve it. This is the "quantization preserves learned behavior" claim, tested directly.