LVL 01SK
Back to roadmap
Fused LLM Inference Kernels project artworkCUDA
PROJECT 13 · GPU / CUDA

Fused LLM Inference Kernels

Implement reductions, activations, fused RMSNorm, Softmax, RoPE, and SwiGLU kernels for efficient inference.

Hard1.7 hours20 steps

Overview

Implement reductions, activations, fused RMSNorm, Softmax, RoPE, and SwiGLU kernels for efficient inference.

What you'll learn

CUDA Kernel fusion Profiling
FULL WALKTHROUGH

Learn every step on its own page

This project is no longer compressed into a few chapters. Open the dedicated learning workspace for a lesson-by-lesson explanation with concepts, MathJax mathematics, code, tests, mistakes, checkpoints, and persistent navigation.

Open 20-step walkthrough →

Build progress

0 / 20 steps0%

Move the tracker as you finish the original Deep-ML steps. Reaching 100% unlocks the completion action and certificate.

Architecture

Work through the system one dependable layer at a time. Each stage feeds the next and remains independently testable.

01Warp reductions→
02RoPE→
03SwiGLU
∑

Mathematics & visual explanation

Translate the core equations into code, validate intermediate tensors, and compare the implementation with a small numerical reference.

objective(θ) = data_term(θ) + λ · regularization(θ)

The exact objective evolves with each milestone. Keep a notebook of shapes, invariants, and numerical checks.