PART 1
0/4Log-Probability Building Blocks
Numerically stable softmax/log-softmax, token log-prob gathering, and masked sequence log-probabilities used everywhere downstream.
DPO from Scratch
DPO from Scratch
Implement log-prob utilities, Bradley–Terry preferences, DPO gradients, IPO variants, and a full evaluation pipeline.
Every source step is its own lesson with intuition, concepts, correctly rendered MathJax mathematics, implementation, tests, mistakes, and a checkpoint.
PART 2
0/4Policy Model & Sequence Log-Probs
Initialize policy parameters, compute token logits, sequence log-probs, and their gradients w.r.t. the policy.
PART 3
0/4Preference Data & Bradley–Terry
Bradley–Terry pairwise loss, reward accuracy, and utilities to build and sample preference pairs.
PART 4
0/5DPO Loss Formulation
Frozen reference log-probs, policy–reference log-ratios, pair margins, the DPO objective, and its gradient.
PART 5
0/2DPO Training Loop
A single DPO train step and the outer training loop over preference batches.
PART 6
0/6Variants, Implicit Rewards & Diagnostics
Length-normalized log-probs, IPO loss, implicit rewards, preference accuracy, KL to reference, and reward-margin statistics.
PART 7
0/2Evaluation & Full Pipeline
Aggregate DPO evaluation metrics and wire everything into a complete train-and-evaluate pipeline.