Fused LLM Inference Kernels

Implement reductions, activations, fused RMSNorm, Softmax, RoPE, and SwiGLU kernels for efficient inference.

7 parts20 individual lessons1.7 estimated hours100 XP available

Fused LLM Inference Kernels project artwork

0%0 of 20 complete

No compressed chapters.

Every source step is its own lesson with intuition, concepts, correctly rendered MathJax mathematics, implementation, tests, mistakes, and a checkpoint.

PART 1

Warp and Block Reductions

Build fundamental warp-level and block-level sum/max reduction primitives used by later kernels.

0/4

001warp reduce sum+5 XP 002warp reduce max+5 XP 003block reduce sum+5 XP 004block reduce max+5 XP

PART 2

Residual and Activation Kernels

Implement residual addition and common transformer activations: GELU, SiLU, and SwiGLU.

0/4

005add residual kernel+5 XP 006gelu kernel+5 XP 007silu kernel+5 XP 008swiglu kernel+5 XP

PART 3

Normalization Kernels

Write RMSNorm, LayerNorm, and a fused residual-plus-RMSNorm kernel for pre-norm blocks.

0/3

009rmsnorm kernel+5 XP 010layernorm kernel+5 XP 011fused add rmsnorm kernel+5 XP

PART 4

Softmax Kernels

Implement row-wise and causal softmax for attention score normalization.

0/2

012softmax row kernel+5 XP 013causal softmax kernel+5 XP

PART 5

Embeddings and RoPE

Build token embedding lookup and rotary positional embedding (RoPE) kernels.

0/2

014embedding lookup kernel+5 XP 015rope kernel+5 XP

PART 6

Linear Layers and Fused MLP Ops

Implement dense linear layers, fused linear+bias+GELU, and a full SwiGLU MLP forward.

0/3

016linear kernel+5 XP 017fused linear bias gelu kernel+5 XP 018mlp swiglu forward+5 XP

PART 7

Composite Transformer Blocks

Compose RMSNorm-residual blocks and an end-to-end transformer FFN path for fused inference.

0/2

019rmsnorm residual block+5 XP 020run transformer ffn+5 XP