PART 1
0/6Patch Embedding and ViT Input
Turn a raw image tensor into a sequence of patch embeddings with a class token and learnable position embeddings.
Vision-Language Model
Vision-Language Model
Build a ViT encoder, multimodal projector, causal decoder, training loop, and caption generation from raw tensor operations.
Every source step is its own lesson with intuition, concepts, correctly rendered MathJax mathematics, implementation, tests, mistakes, and a checkpoint.
PART 2
0/13Multi-Head Self-Attention
Implement scaled dot-product attention from primitives and assemble it into a full multi-head self-attention module.
007define multi head self attention+5 XP008derive multi head self attention+5 XP009prepare multi head self attention+5 XP010implement multi head self attention+5 XP011connect multi head self attention+5 XP012validate multi head self attention+5 XP013debug multi head self attention+5 XP014optimize multi head self attention+5 XP015benchmark multi head self attention+5 XP016integrate multi head self attention+5 XP017define multi head self attention+5 XP018derive multi head self attention+5 XP019prepare multi head self attention+5 XP
PART 3
0/10MLP, Normalization, and Encoder Blocks
Build the position-wise MLP, layer normalization, pre-norm residual sublayers, and stack them into a vision encoder.
020define mlp normalization and encoder blocks+5 XP021derive mlp normalization and encoder blocks+5 XP022prepare mlp normalization and encoder blocks+5 XP023implement mlp normalization and encoder blocks+5 XP024connect mlp normalization and encoder blocks+5 XP025validate mlp normalization and encoder blocks+5 XP026debug mlp normalization and encoder blocks+5 XP027optimize mlp normalization and encoder blocks+5 XP028benchmark mlp normalization and encoder blocks+5 XP029integrate mlp normalization and encoder blocks+5 XP
PART 4
0/4Vision-to-Language Projector
Extract patch features and project them into the language model's embedding dimension with a two-layer MLP.
PART 5
0/8Text Embedding and Multimodal Fusion
Build the tokenizer, embed text tokens, locate the image placeholder, and splice projected image tokens into the multimodal embedding sequence.
034define text embedding and multimodal fusion+5 XP035derive text embedding and multimodal fusion+5 XP036prepare text embedding and multimodal fusion+5 XP037implement text embedding and multimodal fusion+5 XP038connect text embedding and multimodal fusion+5 XP039validate text embedding and multimodal fusion+5 XP040debug text embedding and multimodal fusion+5 XP041optimize text embedding and multimodal fusion+5 XP
PART 6
0/7Causal Decoder and Forward Pass
Build the masked transformer decoder, language modeling head, and wire up the full vision-language model forward pass.
042define causal decoder and forward pass+5 XP043derive causal decoder and forward pass+5 XP044prepare causal decoder and forward pass+5 XP045implement causal decoder and forward pass+5 XP046connect causal decoder and forward pass+5 XP047validate causal decoder and forward pass+5 XP048debug causal decoder and forward pass+5 XP
PART 7
0/8Loss and Caption Generation
Compute next-token cross-entropy loss with masking, then implement temperature, top-k, and sampling-based autoregressive generation.
049define loss and caption generation+5 XP050derive loss and caption generation+5 XP051prepare loss and caption generation+5 XP052implement loss and caption generation+5 XP053connect loss and caption generation+5 XP054validate loss and caption generation+5 XP055debug loss and caption generation+5 XP056optimize loss and caption generation+5 XP
PART 8
0/6Training Loop
Initialize parameters, run training steps with autograd, perform gradient descent updates, and execute a full training loop.