RLHF from Scratch

Build decoding, SFT, LoRA, reward modeling, PPO, preference optimization, evaluation, and a model comparison interface.

8 parts65 individual lessons5.4 estimated hours325 XP available

0%0 of 65 complete

No compressed chapters.

Every source step is its own lesson with intuition, concepts, correctly rendered MathJax mathematics, implementation, tests, mistakes, and a checkpoint.

PART 1

Model Setup and Decoding Strategies

Load distilgpt2 and its tokenizer, sanity-check generation, and implement the core decoding primitives (greedy, temperature, top-k, top-p) used throughout the rest of the project.

0/8

001define model setup and decoding strategies+5 XP 002derive model setup and decoding strategies+5 XP 003prepare model setup and decoding strategies+5 XP 004implement model setup and decoding strategies+5 XP 005connect model setup and decoding strategies+5 XP 006validate model setup and decoding strategies+5 XP 007debug model setup and decoding strategies+5 XP 008optimize model setup and decoding strategies+5 XP

PART 2

SFT Data Pipeline

Construct a synthetic instruction dataset and build the formatting, tokenization, label masking, padding, batching, and train/val splitting utilities needed for supervised fine-tuning.

0/11

009define sft data pipeline+5 XP 010derive sft data pipeline+5 XP 011prepare sft data pipeline+5 XP 012implement sft data pipeline+5 XP 013connect sft data pipeline+5 XP 014validate sft data pipeline+5 XP 015debug sft data pipeline+5 XP 016optimize sft data pipeline+5 XP 017benchmark sft data pipeline+5 XP 018integrate sft data pipeline+5 XP 019define sft data pipeline+5 XP

PART 3

SFT Training Loop

Implement the optimization machinery for supervised fine-tuning: shifted next-token loss, AdamW, warmup, gradient clipping, accumulation, training and evaluation steps.

0/8

020define sft training loop+5 XP 021derive sft training loop+5 XP 022prepare sft training loop+5 XP 023implement sft training loop+5 XP 024connect sft training loop+5 XP 025validate sft training loop+5 XP 026debug sft training loop+5 XP 027optimize sft training loop+5 XP

PART 4

LoRA Adapters

Add low-rank adapters on top of distilgpt2 by implementing the LoRA delta, forward pass, initialization, parameter freezing, counting, and merge-back routines.

0/6

028define lora adapters+5 XP 029derive lora adapters+5 XP 030prepare lora adapters+5 XP 031implement lora adapters+5 XP 032connect lora adapters+5 XP 033validate lora adapters+5 XP

PART 5

Reward Modeling

Build a synthetic preference dataset and train a reward head with pairwise losses, BCE alternatives, and pairwise accuracy diagnostics.

0/7

034define reward modeling+5 XP 035derive reward modeling+5 XP 036prepare reward modeling+5 XP 037implement reward modeling+5 XP 038connect reward modeling+5 XP 039validate reward modeling+5 XP 040debug reward modeling+5 XP

PART 6

PPO-Based RLHF

Implement the PPO ingredients required to optimize the policy against the learned reward: log-probs, KL penalties, returns, GAE, clipped surrogates, value loss, and entropy bonus.

0/11

041define ppo based rlhf+5 XP 042derive ppo based rlhf+5 XP 043prepare ppo based rlhf+5 XP 044implement ppo based rlhf+5 XP 045connect ppo based rlhf+5 XP 046validate ppo based rlhf+5 XP 047debug ppo based rlhf+5 XP 048optimize ppo based rlhf+5 XP 049benchmark ppo based rlhf+5 XP 050integrate ppo based rlhf+5 XP 051define ppo based rlhf+5 XP

PART 7

Preference Optimization Alternatives

Implement modern reference-based and reference-free preference losses including DPO, IPO, KTO, ORPO, and SimPO as drop-in replacements for the PPO pipeline.

0/7

052define preference optimization alternatives+5 XP 053derive preference optimization alternatives+5 XP 054prepare preference optimization alternatives+5 XP 055implement preference optimization alternatives+5 XP 056connect preference optimization alternatives+5 XP 057validate preference optimization alternatives+5 XP 058debug preference optimization alternatives+5 XP

PART 8

Evaluation and Chat Interface

Generate completions on a held-out prompt set, score and compare models via win rate, and wrap everything in a streaming chat function with stop-token handling.

0/7

059define evaluation and chat interface+5 XP 060derive evaluation and chat interface+5 XP 061prepare evaluation and chat interface+5 XP 062implement evaluation and chat interface+5 XP 063connect evaluation and chat interface+5 XP 064validate evaluation and chat interface+5 XP 065debug evaluation and chat interface+5 XP