Module: FT19 — Quantization Formats Diagram count: 5 Tool: Mermaid (primary). Each diagram validated in Mermaid Live Editor.
Type: Deployment-target decision map Purpose: The single diagram that anchors the module. The format is determined by the inference engine / deployment target, not chosen in the abstract. Read the left column first, then the format, then the rationale. Reading the diagram: Each row maps a deployment target to its format. The right column is the load-bearing "why."
flowchart LR
subgraph Target["DEPLOYMENT TARGET"]
T1["Local / CPU / mixed\n(Ollama, llama.cpp)"]
T2["NVIDIA GPU prod serving\n(vLLM, TGI)"]
T3["Max quality at low bitrate\n(ExLlamaV2)"]
T4["Apple Silicon\n(macs)"]
T5["VRAM-rich Hopper/Blackwell\n(high quality)"]
T6["Blackwell-native 4-bit\n(2025 frontier)"]
end
subgraph Format["FORMAT"]
F1["GGUF\n(Q4_K_M sweet spot)"]
F2["AWQ\n(AWQ-Marlin)"]
F3["EXL2\n(variable-rate)"]
F4["MLX\n(4-bit group)"]
F5["FP8"]
F6["MXFP4 / NVFP4"]
end
T1 --> F1
T2 --> F2
T3 --> F3
T4 --> F4
T5 --> F5
T6 --> F6
style Target fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
style Format fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
style T1 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
style T2 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
style T3 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
style T4 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
style T5 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
style T6 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
style F1 fill:#08080c,stroke:#5eead4,color:#5eead4
style F2 fill:#08080c,stroke:#5eead4,color:#5eead4
style F3 fill:#08080c,stroke:rgba(94,234,212,0.5),color:#e4e4e8
style F4 fill:#08080c,stroke:#5eead4,color:#5eead4
style F5 fill:#08080c,stroke:rgba(94,234,212,0.5),color:#e4e4e8
style F6 fill:#08080c,stroke:rgba(94,234,212,0.5),color:#e4e4e8
Type: Trade-off curve (discretized) Purpose: Make the non-linear shape of the trade-off concrete. Q4 is the sweet spot. Below Q4 the quality cost gets steep; above Q4 the size premium gets expensive for diminishing quality returns. Reading the diagram: Left to right is more bits/param (higher quality, larger size). The Q4 column is the recommended default; Q2/Q3 are the danger zone; Q8 is near-lossless.
flowchart LR
Q2["Q2\n~88% smaller\nNOTICEABLE\ndegradation"] --> Q3["Q3\n~84% smaller\nmeasurable\ndegradation"]
Q3 --> Q4["Q4 · SWEET SPOT\n~75% smaller\nminimal loss"]
Q4 --> Q5["Q5\n~70% smaller\nnear-lossless"]
Q5 --> Q8["Q8\n~58% smaller\neffectively\nlossless"]
Q8 --> FP["FP16\nbaseline\nreference"]
Q2 -.->|"desperate only"| Danger["DANGER ZONE\nbenchmark before shipping"]
Q4 -.->|"start here"| Sweet["DEFAULT\nreach for Q4_K_M first"]
Q8 -.->|"VRAM-rich"| Safe["LOSSLESS\nuse when certainty matters"]
style Q2 fill:#14141f,stroke:#f08080,color:#f08080
style Q3 fill:#14141f,stroke:#f0a868,color:#f0a868
style Q4 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
style Q5 fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
style Q8 fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
style FP fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
style Danger fill:#08080c,stroke:#f08080,stroke-dasharray: 4 2,color:#f08080
style Sweet fill:#08080c,stroke:#5eead4,stroke-dasharray: 4 2,color:#5eead4
style Safe fill:#08080c,stroke:rgba(94,234,212,0.5),stroke-dasharray: 4 2,color:#5eead4
Type: Pipeline / conversion flow Purpose: Show the three primary conversion paths from one trained checkpoint to three deployable formats. One source checkpoint -> three export artifacts. The modularity is the point. Reading the diagram: Top is the single trained FP16 checkpoint (the output of FT11). Three branches convert it to the three formats via their respective tools. Each artifact targets a different runtime.
flowchart TD
CKPT["YOUR FINE-TUNED CHECKPOINT\n(FT11 output · FP16/BF16)"]
CKPT --> GGUF["GGUF path\nllama.cpp convert_hf_to_gguf.py\nOR Unsloth save_pretrained_gguf"]
CKPT --> AWQ["AWQ path\nAutoAWQ.quantize\ncalibration -> 4-bit"]
CKPT --> MLX["MLX path\nmlx_lm.convert --quantize --q-bits 4"]
GGUF --> A1["model-Q4_K_M.gguf\n-> Ollama / llama.cpp / LM Studio"]
AWQ --> A2["model-awq/\n-> vLLM / TGI (Marlin kernel)"]
MLX --> A3["model-mlx/\n-> mlx-lm / LM Studio (Mac)"]
Note["ONE source checkpoint\nTHREE export artifacts\nQuantize AFTER training (Layer 4)"]
CKPT -.-> Note
style CKPT fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
style GGUF fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
style AWQ fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
style MLX fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
style A1 fill:#14141f,stroke:rgba(94,234,212,0.5),color:#5eead4
style A2 fill:#14141f,stroke:rgba(94,234,212,0.5),color:#5eead4
style A3 fill:#14141f,stroke:rgba(94,234,212,0.5),color:#5eead4
style Note fill:#08080c,stroke:rgba(94,234,212,0.4),stroke-dasharray: 4 2,color:#5eead4
Type: Comparison (uniform vs per-layer) Purpose: Show why uniform quantization leaves quality on the table and how per-layer sensitivity-aware quantization (Unsloth Dynamic 2.0 / EXL2's approach) beats it at the same total size. Reading the diagram: Two columns of the same model. Uniform gives every layer Q4. Dynamic measures each layer's sensitivity and varies the bitrate — sensitive layers stay high, tolerant layers drop low — hitting the same average with higher quality.
flowchart LR
subgraph Uniform["UNIFORM Q4 (one bitrate for all)"]
U1["Layer 1 · Q4"]
U2["Layer 2 · Q4 (sensitive! quality lost)"]
U3["Layer 3 · Q4"]
U4["Layer 4 · Q4 (could be Q3, wasted bits)"]
U5["Avg = Q4 · quality = baseline"]
end
subgraph Dynamic["DYNAMIC 2.0 (sensitivity-aware)"]
D1["Layer 1 · Q4"]
D2["Layer 2 · Q6 (sensitive -> keep high)"]
D3["Layer 3 · Q3"]
D4["Layer 4 · Q3 (tolerant -> compress)"]
D5["Avg = Q4 · quality HIGHER at same size"]
end
Uniform -.->|"measure each layer's\nsensitivity, re-allocate"| Dynamic
style Uniform fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
style Dynamic fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
style U1 fill:#08080c,stroke:rgba(255,255,255,0.08),color:#9494a0
style U2 fill:#08080c,stroke:#f08080,color:#f08080
style U3 fill:#08080c,stroke:rgba(255,255,255,0.08),color:#9494a0
style U4 fill:#08080c,stroke:#f0a868,color:#f0a868
style U5 fill:#08080c,stroke:rgba(255,255,255,0.12),color:#9494a0
style D1 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
style D2 fill:#08080c,stroke:#82e0aa,color:#82e0aa
style D3 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
style D4 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
style D5 fill:#08080c,stroke:#5eead4,color:#5eead4
Type: Stack position / pipeline Purpose: Place quantization correctly. It is Layer 4 (Export), downstream of Layer 3 (the Steer / training). It compresses what training produced; it does not change learned behavior. Compare with QLoRA's training-time quant (a memory trick, a different concern). Reading the diagram: Bottom to top is the stack. Layer 3 trains the behavior; Layer 4 exports it compressed. The dashed callout contrasts the deployment quant (this module) with QLoRA's training-time quant (FT08).
flowchart TD
L5["5. THE BOUNDARY · the harness (Courses 1, 2A)"]
L4["4. THE EXPORT · QUANTIZE + SERVE\nTHIS MODULE (FT19) + FT20"]
L3["3. THE STEER · fine-tuning (SFT/DPO/GRPO) (FT12-FT18)"]
L2["2. THE ADAPTER · LoRA / DoRA (FT08-FT11)"]
L1["1. THE BASE · pretrained weights"]
L1 --> L2 --> L3 --> L4 --> L5
L3 -.->|"outputs a merged\nFP16/BF16 checkpoint"| CKPT["trained checkpoint"]
CKPT -.->|"Layer 4 compresses it\n(GGUF/AWQ/MLX/etc.)\nbehavior PRESERVED"| L4
Note["QLoRA (FT08) quantizes DURING training\n(memory trick to fit base in VRAM).\nFT19 quantizes AFTER training\n(deployment export).\nDifferent concerns, different layers."]
L4 -.-> Note
style L1 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
style L2 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
style L3 fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
style L4 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
style L5 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
style CKPT fill:#08080c,stroke:rgba(94,234,212,0.4),color:#5eead4
style Note fill:#08080c,stroke:rgba(94,234,212,0.4),stroke-dasharray: 4 2,color:#5eead4
#14141f panel fill, #5eead4 accent for primary, rgba(255,255,255,0.12) for secondary borders, #e4e4e8 / #9494a0 for text, with #82e0aa (ok) and #f08080 (danger) and #f0a868 (warn) used semantically for quality/sensitivity and warnings.flowchart with subgraph, LR/TD direction, dashed -.-> for annotations) supported in current Mermaid (v10.4+).# Diagrams — Module FT19: Quantization Formats
**Module**: FT19 — Quantization Formats
**Diagram count**: 5
**Tool**: Mermaid (primary). Each diagram validated in [Mermaid Live Editor](https://mermaid.live).
---
## Diagram 1 — The Format Decision Matrix (format -> use case)
**Type**: Deployment-target decision map
**Purpose**: The single diagram that anchors the module. The format is determined by the inference engine / deployment target, not chosen in the abstract. Read the left column first, then the format, then the rationale.
**Reading the diagram**: Each row maps a deployment target to its format. The right column is the load-bearing "why."
```mermaid
flowchart LR
subgraph Target["DEPLOYMENT TARGET"]
T1["Local / CPU / mixed\n(Ollama, llama.cpp)"]
T2["NVIDIA GPU prod serving\n(vLLM, TGI)"]
T3["Max quality at low bitrate\n(ExLlamaV2)"]
T4["Apple Silicon\n(macs)"]
T5["VRAM-rich Hopper/Blackwell\n(high quality)"]
T6["Blackwell-native 4-bit\n(2025 frontier)"]
end
subgraph Format["FORMAT"]
F1["GGUF\n(Q4_K_M sweet spot)"]
F2["AWQ\n(AWQ-Marlin)"]
F3["EXL2\n(variable-rate)"]
F4["MLX\n(4-bit group)"]
F5["FP8"]
F6["MXFP4 / NVFP4"]
end
T1 --> F1
T2 --> F2
T3 --> F3
T4 --> F4
T5 --> F5
T6 --> F6
style Target fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
style Format fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
style T1 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
style T2 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
style T3 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
style T4 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
style T5 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
style T6 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
style F1 fill:#08080c,stroke:#5eead4,color:#5eead4
style F2 fill:#08080c,stroke:#5eead4,color:#5eead4
style F3 fill:#08080c,stroke:rgba(94,234,212,0.5),color:#e4e4e8
style F4 fill:#08080c,stroke:#5eead4,color:#5eead4
style F5 fill:#08080c,stroke:rgba(94,234,212,0.5),color:#e4e4e8
style F6 fill:#08080c,stroke:rgba(94,234,212,0.5),color:#e4e4e8
```
---
## Diagram 2 — The Quality/Size Trade-off Curve
**Type**: Trade-off curve (discretized)
**Purpose**: Make the non-linear shape of the trade-off concrete. Q4 is the sweet spot. Below Q4 the quality cost gets steep; above Q4 the size premium gets expensive for diminishing quality returns.
**Reading the diagram**: Left to right is more bits/param (higher quality, larger size). The Q4 column is the recommended default; Q2/Q3 are the danger zone; Q8 is near-lossless.
```mermaid
flowchart LR
Q2["Q2\n~88% smaller\nNOTICEABLE\ndegradation"] --> Q3["Q3\n~84% smaller\nmeasurable\ndegradation"]
Q3 --> Q4["Q4 · SWEET SPOT\n~75% smaller\nminimal loss"]
Q4 --> Q5["Q5\n~70% smaller\nnear-lossless"]
Q5 --> Q8["Q8\n~58% smaller\neffectively\nlossless"]
Q8 --> FP["FP16\nbaseline\nreference"]
Q2 -.->|"desperate only"| Danger["DANGER ZONE\nbenchmark before shipping"]
Q4 -.->|"start here"| Sweet["DEFAULT\nreach for Q4_K_M first"]
Q8 -.->|"VRAM-rich"| Safe["LOSSLESS\nuse when certainty matters"]
style Q2 fill:#14141f,stroke:#f08080,color:#f08080
style Q3 fill:#14141f,stroke:#f0a868,color:#f0a868
style Q4 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
style Q5 fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
style Q8 fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
style FP fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
style Danger fill:#08080c,stroke:#f08080,stroke-dasharray: 4 2,color:#f08080
style Sweet fill:#08080c,stroke:#5eead4,stroke-dasharray: 4 2,color:#5eead4
style Safe fill:#08080c,stroke:rgba(94,234,212,0.5),stroke-dasharray: 4 2,color:#5eead4
```
---
## Diagram 3 — The Conversion Workflows
**Type**: Pipeline / conversion flow
**Purpose**: Show the three primary conversion paths from one trained checkpoint to three deployable formats. One source checkpoint -> three export artifacts. The modularity is the point.
**Reading the diagram**: Top is the single trained FP16 checkpoint (the output of FT11). Three branches convert it to the three formats via their respective tools. Each artifact targets a different runtime.
```mermaid
flowchart TD
CKPT["YOUR FINE-TUNED CHECKPOINT\n(FT11 output · FP16/BF16)"]
CKPT --> GGUF["GGUF path\nllama.cpp convert_hf_to_gguf.py\nOR Unsloth save_pretrained_gguf"]
CKPT --> AWQ["AWQ path\nAutoAWQ.quantize\ncalibration -> 4-bit"]
CKPT --> MLX["MLX path\nmlx_lm.convert --quantize --q-bits 4"]
GGUF --> A1["model-Q4_K_M.gguf\n-> Ollama / llama.cpp / LM Studio"]
AWQ --> A2["model-awq/\n-> vLLM / TGI (Marlin kernel)"]
MLX --> A3["model-mlx/\n-> mlx-lm / LM Studio (Mac)"]
Note["ONE source checkpoint\nTHREE export artifacts\nQuantize AFTER training (Layer 4)"]
CKPT -.-> Note
style CKPT fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
style GGUF fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
style AWQ fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
style MLX fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
style A1 fill:#14141f,stroke:rgba(94,234,212,0.5),color:#5eead4
style A2 fill:#14141f,stroke:rgba(94,234,212,0.5),color:#5eead4
style A3 fill:#14141f,stroke:rgba(94,234,212,0.5),color:#5eead4
style Note fill:#08080c,stroke:rgba(94,234,212,0.4),stroke-dasharray: 4 2,color:#5eead4
```
---
## Diagram 4 — Unsloth Dynamic 2.0 vs Uniform Quantization
**Type**: Comparison (uniform vs per-layer)
**Purpose**: Show why uniform quantization leaves quality on the table and how per-layer sensitivity-aware quantization (Unsloth Dynamic 2.0 / EXL2's approach) beats it at the same total size.
**Reading the diagram**: Two columns of the same model. Uniform gives every layer Q4. Dynamic measures each layer's sensitivity and varies the bitrate — sensitive layers stay high, tolerant layers drop low — hitting the same average with higher quality.
```mermaid
flowchart LR
subgraph Uniform["UNIFORM Q4 (one bitrate for all)"]
U1["Layer 1 · Q4"]
U2["Layer 2 · Q4 (sensitive! quality lost)"]
U3["Layer 3 · Q4"]
U4["Layer 4 · Q4 (could be Q3, wasted bits)"]
U5["Avg = Q4 · quality = baseline"]
end
subgraph Dynamic["DYNAMIC 2.0 (sensitivity-aware)"]
D1["Layer 1 · Q4"]
D2["Layer 2 · Q6 (sensitive -> keep high)"]
D3["Layer 3 · Q3"]
D4["Layer 4 · Q3 (tolerant -> compress)"]
D5["Avg = Q4 · quality HIGHER at same size"]
end
Uniform -.->|"measure each layer's\nsensitivity, re-allocate"| Dynamic
style Uniform fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
style Dynamic fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
style U1 fill:#08080c,stroke:rgba(255,255,255,0.08),color:#9494a0
style U2 fill:#08080c,stroke:#f08080,color:#f08080
style U3 fill:#08080c,stroke:rgba(255,255,255,0.08),color:#9494a0
style U4 fill:#08080c,stroke:#f0a868,color:#f0a868
style U5 fill:#08080c,stroke:rgba(255,255,255,0.12),color:#9494a0
style D1 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
style D2 fill:#08080c,stroke:#82e0aa,color:#82e0aa
style D3 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
style D4 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
style D5 fill:#08080c,stroke:#5eead4,color:#5eead4
```
---
## Diagram 5 — Quantize-After-Training: Layer 4 in the Stack
**Type**: Stack position / pipeline
**Purpose**: Place quantization correctly. It is Layer 4 (Export), downstream of Layer 3 (the Steer / training). It compresses what training produced; it does not change learned behavior. Compare with QLoRA's training-time quant (a memory trick, a different concern).
**Reading the diagram**: Bottom to top is the stack. Layer 3 trains the behavior; Layer 4 exports it compressed. The dashed callout contrasts the deployment quant (this module) with QLoRA's training-time quant (FT08).
```mermaid
flowchart TD
L5["5. THE BOUNDARY · the harness (Courses 1, 2A)"]
L4["4. THE EXPORT · QUANTIZE + SERVE\nTHIS MODULE (FT19) + FT20"]
L3["3. THE STEER · fine-tuning (SFT/DPO/GRPO) (FT12-FT18)"]
L2["2. THE ADAPTER · LoRA / DoRA (FT08-FT11)"]
L1["1. THE BASE · pretrained weights"]
L1 --> L2 --> L3 --> L4 --> L5
L3 -.->|"outputs a merged\nFP16/BF16 checkpoint"| CKPT["trained checkpoint"]
CKPT -.->|"Layer 4 compresses it\n(GGUF/AWQ/MLX/etc.)\nbehavior PRESERVED"| L4
Note["QLoRA (FT08) quantizes DURING training\n(memory trick to fit base in VRAM).\nFT19 quantizes AFTER training\n(deployment export).\nDifferent concerns, different layers."]
L4 -.-> Note
style L1 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
style L2 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
style L3 fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
style L4 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
style L5 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
style CKPT fill:#08080c,stroke:rgba(94,234,212,0.4),color:#5eead4
style Note fill:#08080c,stroke:rgba(94,234,212,0.4),stroke-dasharray: 4 2,color:#5eead4
```
---
## Validation notes
- All five diagrams use the course design system colors: `#14141f` panel fill, `#5eead4` accent for primary, `rgba(255,255,255,0.12)` for secondary borders, `#e4e4e8` / `#9494a0` for text, with `#82e0aa` (ok) and `#f08080` (danger) and `#f0a868` (warn) used semantically for quality/sensitivity and warnings.
- Paste each into [Mermaid Live Editor](https://mermaid.live) to render. All use stable Mermaid syntax (`flowchart` with `subgraph`, `LR`/`TD` direction, dashed `-.->` for annotations) supported in current Mermaid (v10.4+).
- For the slide deck (artifact 03), these are rendered as static captures from Mermaid Live and inlined into reveal.js as needed.