AlphaZero on Connect-4
Follow the data, decisions, feedback, and validation boundaries before writing the full system.
How to read this diagram
Read left to right for the forward path: raw information becomes a representation, passes through the project’s main computational ideas, and produces an output that can be measured. Then follow the feedback path back toward the trainable or decision-making components.
PUCT
PUCT balances exploitation of actions with high mean value and exploration of actions favored by the policy prior but visited less often. Repeated selection, expansion, evaluation, and backup turns neural estimates into a stronger search policy.
Boundary check: document its accepted input, output shape, mutable state, failure modes, and the metric that proves this stage is correct before connecting it downstream.
Value networks
A value network compresses the expected outcome of a state into one scalar from the current player’s perspective. Perspective must flip during tree backup; otherwise an action good for the opponent is accidentally reinforced.
Boundary check: document its accepted input, output shape, mutable state, failure modes, and the metric that proves this stage is correct before connecting it downstream.
Replay buffer
A replay store breaks temporal correlation and lets expensive experience support multiple gradient updates. Its schema, sampling policy, age distribution, and target semantics are part of the algorithm—not mere storage details.
Boundary check: document its accepted input, output shape, mutable state, failure modes, and the metric that proves this stage is correct before connecting it downstream.
Architecture review checklist
- Every arrow has a documented shape, dtype, unit, or schema.
- Training and evaluation paths cannot leak information into each other.
- Randomness is seeded and captured in experiment metadata.
- Expensive stages expose timing, memory, throughput, and error metrics.
- Each feedback loop has a stop condition and a rollback strategy.
- Small reference implementations exist for numerical comparisons.