Pretraining Basics
Pretraining is the most computationally expensive phase of building an LLM, where a model learns general knowledge and language patterns from massive datasets.
Fine-Tuning Strategies
Fine-tuning adapts a pretrained model to specific tasks, domains, or styles. Today, fine-tuning is less about "teaching facts" and more about "teaching format and behavior."
LoRA, QLoRA, and PEFT
Parameter-Efficient Fine-Tuning (PEFT) is the industry standard for adapting LLMs. This chapter covers the mechanics and advanced variants of LoRA and other PEFT methods.
RLHF and DPO (Alignment)
Alignment is the process of ensuring an LLM's behavior matches human values and instructions. The field has moved from traditional RLHF to more efficient and scalable methods like DPO and Online RL.
Knowledge Distillation
Knowledge distillation is the process of transferring the intelligence from a large, complex model ("Teacher") to a smaller, more efficient one ("Student"). This is the secret to the high performance of today's small open-weight models that punch well above their parameter count.
Synthetic Data Generation
The industry has hit the "Data Wall", the exhaustion of high-quality human text on the web. Synthetic data is now the primary engine for model improvement, sitting at the core of every modern frontier-model recipe.
Quantization Deep Dive
Quantization is the process of reducing the precision of model weights (e.g., from 16-bit to 4-bit) to save memory and increase inference speed. This is the primary tool for deploying large models on consumer and single-GPU hardware.
Training Reasoning Models: RLVR and GRPO
The frontier reasoning models (OpenAI's o-series, DeepSeek-R1, Qwen's thinking models) are trained with a recipe distinct from the RLHF and DPO covered in RLHF and DPO: reinforcement learning with verifiable rewards (RLVR). Take a capable base model, have it generate long chains of thought on problems with checkable answers (math, code, logic), and reward correctness directly with a programmatic verifier instead of a learned reward model. This chapter covers RLVR, the GRPO algorithm DeepSeek made popular, the reward-design failure modes, the live debate over what RL actually adds, and the cheaper distillation path most teams should take first.