A/B Testing & Causal Inference

Build proportion tests, sample sizing, multiple-testing corrections, difference-in-differences, and synthetic control.

8 parts22 individual lessons1.8 estimated hours110 XP available

A/B Testing & Causal Inference project artwork

0%0 of 22 complete

No compressed chapters.

Every source step is its own lesson with intuition, concepts, correctly rendered MathJax mathematics, implementation, tests, mistakes, and a checkpoint.

PART 1

Normal Distribution Primitives

Implement the standard normal CDF and its inverse used throughout hypothesis testing and confidence intervals.

0/2

001define normal distribution primitives+5 XP 002derive normal distribution primitives+5 XP

PART 2

Two-Proportion Hypothesis Testing

Compute pooled and unpooled standard errors, the z-statistic, p-values, and confidence intervals for comparing two proportions.

0/6

003define two proportion hypothesis testing+5 XP 004derive two proportion hypothesis testing+5 XP 005prepare two proportion hypothesis testing+5 XP 006implement two proportion hypothesis testing+5 XP 007connect two proportion hypothesis testing+5 XP 008validate two proportion hypothesis testing+5 XP

PART 3

Experiment Design: Power & Sample Size

Determine required per-variant sample size and evaluate statistical power for planned experiments.

0/2

009define experiment design power sample size+5 XP 010derive experiment design power sample size+5 XP

PART 4

Sample Ratio Mismatch Diagnostics

Use a chi-square check to detect broken randomization or traffic allocation bugs.

0/2

011define sample ratio mismatch diagnostics+5 XP 012derive sample ratio mismatch diagnostics+5 XP

PART 5

Multiple Testing Corrections

Apply Bonferroni and Benjamini-Hochberg procedures to control error rates across many hypotheses.

0/2

013define multiple testing corrections+5 XP 014derive multiple testing corrections+5 XP

PART 6

Difference-in-Differences Estimation

Estimate treatment effects from pre/post outcomes via simple differences and an OLS regression with an interaction term.

0/5

015define difference in differences estimation+5 XP 016derive difference in differences estimation+5 XP 017prepare difference in differences estimation+5 XP 018implement difference in differences estimation+5 XP 019connect difference in differences estimation+5 XP

PART 7

Synthetic Control Method

Fit non-negative donor weights to build a synthetic counterfactual and read off the post-period treatment effect.

0/2

020define synthetic control method+5 XP 021derive synthetic control method+5 XP

PART 8

Ship Decision Logic

Combine the primary metric outcome with guardrail metrics into a final ship/no-ship decision.

0/1

022define ship decision logic+5 XP