PART 1
0/6Tokenization and Batching
Build the vocabulary, encode and decode token ids, and pack padded sequences into batched tensors.
Transformer from Scratch
Transformer from Scratch
Reimplement the original encoder-decoder Transformer with multi-head attention, scheduling, and beam search.
Every source step is its own lesson with intuition, concepts, correctly rendered MathJax mathematics, implementation, tests, mistakes, and a checkpoint.
PART 2
0/7Embeddings and Positional Encoding
Scale embeddings and construct the sinusoidal positional encoding matrix added to input embeddings.
PART 3
0/9Masks and Scaled Dot-Product Attention
Build padding and causal masks and assemble scaled dot-product attention step by step.
014build padding mask+5 XP015build causal mask+5 XP016combine padding and causal masks+5 XP017compute raw attention scores+5 XP018scale attention scores+5 XP019mask attention scores with neg inf+5 XP020softmax attention weights+5 XP021apply attention weights to values+5 XP022scaled dot product attention+5 XP
PART 4
0/9Multi-Head Attention
Split, permute, and merge heads, project Q/K/V, and assemble the full multi-head attention module.
023split last dim into heads+5 XP024transpose heads before sequence+5 XP025merge heads back to model dim+5 XP026apply linear projection+5 XP027project to query key value+5 XP028split qkv into heads+5 XP029multi head scaled dot product attention+5 XP030merge heads and project output+5 XP031assemble multi head attention forward+5 XP
PART 5
0/7Feed-Forward, LayerNorm, and Dropout
Implement the position-wise feed-forward network, layer normalization, residual add-and-norm, and dropout primitives.
PART 6
0/13Encoder, Decoder, and Full Model
Stack encoder and decoder layers, tie output projections to embeddings, and run the complete forward pass.
039encoder layer self attention sublayer+5 XP040encoder layer feed forward sublayer+5 XP041assemble encoder layer+5 XP042stack encoder layers+5 XP043decoder layer masked self attention sublayer+5 XP044decoder layer cross attention sublayer+5 XP045decoder layer feed forward sublayer+5 XP046assemble decoder layer+5 XP047stack decoder layers+5 XP048apply final output projection+5 XP049tie output projection to token embeddings+5 XP050apply log softmax over vocab+5 XP051run transformer forward+5 XP
PART 7
0/4Parameter Initialization
Allocate the raw weight tensors (with requires_grad) for encoder/decoder layers and embeddings, and gather them for the optimizer.
PART 8
0/8Training Objective and Schedule
Implement teacher forcing, Noam warmup, label-smoothed KL loss, and token-level accuracy.
056shift targets right with start token+5 XP057compute noam learning rate+5 XP058build uniform smoothing distribution+5 XP059set confidence on gold tokens+5 XP060zero pad column and pad token rows+5 XP061compute label smoothed kl loss+5 XP062average loss over non pad tokens+5 XP063compute token accuracy ignoring pad+5 XP
PART 9
0/6Adam Optimizer From Scratch
Build the Adam optimizer step by step: moment buffers, exponential-moving-average updates, bias correction, the parameter update rule, and gradient zeroing.
PART 10
0/3Training Step and Loop
Tie everything together: run a forward pass, compute the label-smoothed loss, backpropagate, and step Adam across many iterations.
PART 11
0/7Decoding and Beam Search
Generate sequences with greedy argmax and a length-penalized beam search over candidate hypotheses.