PART 1
0/2Toy Multimodal Dataset
Generate tiny grayscale images and pair each with a text label to form the multimodal training data.
Multimodal Image Generator
Multimodal Image Generator
Train a VQ-VAE image tokenizer and an autoregressive text-conditioned transformer with guided sampling.
Every source step is its own lesson with intuition, concepts, correctly rendered MathJax mathematics, implementation, tests, mistakes, and a checkpoint.
PART 2
0/8Patch Encoder and Decoder
Normalize images, split them into patches, and build the linear encoder/decoder that maps patches to latent vectors and back.
003define patch encoder and decoder+5 XP004derive patch encoder and decoder+5 XP005prepare patch encoder and decoder+5 XP006implement patch encoder and decoder+5 XP007connect patch encoder and decoder+5 XP008validate patch encoder and decoder+5 XP009debug patch encoder and decoder+5 XP010optimize patch encoder and decoder+5 XP
PART 3
0/15Vector Quantization and VQ-VAE Training
Create the codebook, quantize latents with nearest-neighbor lookup and the straight-through estimator, assemble the VQ-VAE losses, train it, and tokenize images.
011define vector quantization and vq vae training+5 XP012derive vector quantization and vq vae training+5 XP013prepare vector quantization and vq vae training+5 XP014implement vector quantization and vq vae training+5 XP015connect vector quantization and vq vae training+5 XP016validate vector quantization and vq vae training+5 XP017debug vector quantization and vq vae training+5 XP018optimize vector quantization and vq vae training+5 XP019benchmark vector quantization and vq vae training+5 XP020integrate vector quantization and vq vae training+5 XP021define vector quantization and vq vae training+5 XP022derive vector quantization and vq vae training+5 XP023prepare vector quantization and vq vae training+5 XP024implement vector quantization and vq vae training+5 XP025connect vector quantization and vq vae training+5 XP
PART 4
0/7Multimodal Sequences and Embeddings
Build a character vocabulary, encode labels, prepend text tokens to image tokens, and add token plus positional embeddings.
026define multimodal sequences and embeddings+5 XP027derive multimodal sequences and embeddings+5 XP028prepare multimodal sequences and embeddings+5 XP029implement multimodal sequences and embeddings+5 XP030connect multimodal sequences and embeddings+5 XP031validate multimodal sequences and embeddings+5 XP032debug multimodal sequences and embeddings+5 XP
PART 5
0/16Transformer Backbone
Implement causal masking, layer norm, multi-head self-attention, the feed-forward MLP, the stacked blocks, and the output logit projection.
033define transformer backbone+5 XP034derive transformer backbone+5 XP035prepare transformer backbone+5 XP036implement transformer backbone+5 XP037connect transformer backbone+5 XP038validate transformer backbone+5 XP039debug transformer backbone+5 XP040optimize transformer backbone+5 XP041benchmark transformer backbone+5 XP042integrate transformer backbone+5 XP043define transformer backbone+5 XP044derive transformer backbone+5 XP045prepare transformer backbone+5 XP046implement transformer backbone+5 XP047connect transformer backbone+5 XP048validate transformer backbone+5 XP
PART 6
0/3Transformer Training
Compute the next-token cross entropy over image positions and apply optax updates to the transformer parameters.
PART 7
0/7Guided Sampling and Generation
Add classifier-free guidance dropout, combine conditional and unconditional logits, apply temperature and top-k sampling, and autoregressively decode tokens into an image.
052define guided sampling and generation+5 XP053derive guided sampling and generation+5 XP054prepare guided sampling and generation+5 XP055implement guided sampling and generation+5 XP056connect guided sampling and generation+5 XP057validate guided sampling and generation+5 XP058debug guided sampling and generation+5 XP
PART 8
0/6Evaluation and End-to-End Pipeline
Measure accuracy, reconstruction error, and generation quality, then run the full VQ-VAE and transformer training and end-to-end conditional generation.
059define evaluation and end to end pipeline+5 XP060derive evaluation and end to end pipeline+5 XP061prepare evaluation and end to end pipeline+5 XP062implement evaluation and end to end pipeline+5 XP063connect evaluation and end to end pipeline+5 XP064validate evaluation and end to end pipeline+5 XP