PART 1
0/7Tokenizer
Build a tiny character-level tokenizer with vocab, stoi/itos, and encode/decode helpers.
Tiny GPT From Scratch
Tiny GPT From Scratch
Grow a character-level language model from a bigram baseline into a working GPT with multi-head attention and sampling.
Every source step is its own lesson with intuition, concepts, correctly rendered MathJax mathematics, implementation, tests, mistakes, and a checkpoint.
PART 2
0/26NumPy and Softmax Foundations
Get fluent with NumPy arrays, indexing, broadcasting, reductions, and numerically stable softmax.
008make 1d array+5 XP009get array shape+5 XP010get array dtype+5 XP011make 2d zeros+5 XP012make 2d random+5 XP013index element+5 XP014slice row+5 XP015slice column+5 XP016slice subblock+5 XP017elementwise add+5 XP018elementwise multiply+5 XP019scalar broadcast add+5 XP020vector matrix broadcast add+5 XP021array exp+5 XP022array log+5 XP023sum all+5 XP024sum axis0+5 XP025sum axis1+5 XP026max along axis+5 XP027matmul+5 XP028transpose matrix+5 XP029sum keepdims+5 XP030naive softmax 1d+5 XP031softmax overflow demo+5 XP032stable softmax 1d+5 XP033stable softmax 2d rowwise+5 XP
PART 3
0/23Data Pipeline and Bigram Baseline
Load the corpus, build batched (X, Y) sequences, and train a counting-based bigram model as a baseline.
034read text file+5 XP035encode corpus to int array+5 XP036pick split point+5 XP037slice train and val+5 XP038pick block size+5 XP039slice x at offset+5 XP040slice y at offset+5 XP041sample random batch offsets+5 XP042stack x batch+5 XP043stack y batch+5 XP044get batch+5 XP045allocate count matrix+5 XP046loop fill counts+5 XP047vectorize counts add at+5 XP048add one smoothing+5 XP049row sums of counts+5 XP050normalize counts to probs+5 XP051sample next token+5 XP052generate sequence+5 XP053decode generated sequence+5 XP054log prob of pair+5 XP055sum negative log probs+5 XP056average nll+5 XP
PART 4
0/17Single-Layer Neural Bigram
Replace the count table with a learned weight matrix and derive cross-entropy, gradients, and SGD updates.
057initialize w random+5 XP058scale w small+5 XP059one hot encode batch+5 XP060forward logits onehot+5 XP061observe lookup equivalence+5 XP062forward logits lookup+5 XP063logits to probs rowwise+5 XP064gather correct token probs+5 XP065cross entropy loss+5 XP066derive dlogits on paper+5 XP067compute dlogits+5 XP068derive dw on paper+5 XP069compute dw scatter add+5 XP070sgd update w+5 XP071run one training step+5 XP072train neural bigram loop+5 XP073sample from neural bigram+5 XP
PART 5
0/18Layer Primitives and Backprop
Implement forward and backward passes for linear, bias, ReLU, softmax+CE, and LayerNorm building blocks.
074linear forward+5 XP075derive dx on paper+5 XP076derive linear dw on paper+5 XP077linear backward dx+5 XP078linear backward dw+5 XP079bias add forward+5 XP080bias add backward db+5 XP081relu forward+5 XP082relu backward+5 XP083softmax cross entropy backward+5 XP084layernorm forward mean+5 XP085layernorm forward variance+5 XP086layernorm forward normalize+5 XP087layernorm forward affine+5 XP088layernorm backward subtract mean+5 XP089layernorm backward divide std+5 XP090layernorm backward full+5 XP091layernorm backward implementation+5 XP
PART 6
0/39Embeddings and Self-Attention
Add token and positional embeddings, then build masked single-head and multi-head self-attention with full backward passes.
092create token embedding+5 XP093token embedding forward+5 XP094token embedding backward+5 XP095create positional embedding+5 XP096slice positional embedding+5 XP097add token and positional embeddings+5 XP098embedding sum backward+5 XP099create qkv projections+5 XP100compute query+5 XP101compute key+5 XP102compute value+5 XP103compute attention scores+5 XP104scale attention scores+5 XP105build causal mask+5 XP106apply causal mask+5 XP107softmax attention weights+5 XP108attention weighted values+5 XP109apply output projection+5 XP110output projection backward+5 XP111attention value backward+5 XP112masked softmax backward+5 XP113scale scores backward+5 XP114qk scores backward+5 XP115qkv projection backward+5 XP116choose attention head config+5 XP117create multihead qkv projections+5 XP118create multihead output projection+5 XP119reshape to heads+5 XP120transpose heads to front+5 XP121get multihead n heads+5 XP122get multihead sequence length+5 XP123compute d head+5 XP124multihead masked softmax scores+5 XP125multihead weighted sum+5 XP126transpose heads to back+5 XP127get multihead output sequence length+5 XP128merge heads to d model+5 XP129multihead output projection forward+5 XP130multihead reshape transpose backward+5 XP
PART 7
0/16FFN, Blocks, and Full Model
Compose feed-forward networks, residual connections, and pre-LN Transformer blocks into the complete GPT forward/backward.
131ffn linear one forward+5 XP132ffn activation forward+5 XP133ffn linear two forward+5 XP134ffn backward+5 XP135residual forward+5 XP136residual backward+5 XP137pre layernorm sublayer forward+5 XP138transformer block forward+5 XP139transformer block backward+5 XP140stack transformer blocks+5 XP141forward through all blocks+5 XP142backward through all blocks+5 XP143final layernorm forward+5 XP144lm head linear forward+5 XP145full model forward+5 XP146full model backward+5 XP
PART 8
0/20Adam, Training Loop, and Generation
Implement Adam, wire up the full training and validation loop, then sample text with temperature and top-k decoding.
147initialize adam moments+5 XP148initialize adam step counter+5 XP149adam increment step+5 XP150adam update first moment+5 XP151adam update second moment+5 XP152adam bias correction+5 XP153adam parameter update+5 XP154wire full training loop+5 XP155logging and validation loss+5 XP156encode prompt+5 XP157crop context to block size+5 XP158forward to get logits+5 XP159take last position logits+5 XP160apply temperature+5 XP161top k filter+5 XP162softmax to probs+5 XP163sample one token+5 XP164append token to sequence+5 XP165generation loop for n steps+5 XP166decode final sequence+5 XP