Mini LLM Inference Server
Follow the data, decisions, feedback, and validation boundaries before writing the full system.
How to read this diagram
Read left to right for the forward path: raw information becomes a representation, passes through the project’s main computational ideas, and produces an output that can be measured. Then follow the feedback path back toward the trainable or decision-making components.
KV cache
Autoregressive decoding reuses keys and values from previous tokens. A KV cache changes attention from repeatedly recomputing history to appending one position, trading memory capacity and movement for lower token latency.
Boundary check: document its accepted input, output shape, mutable state, failure modes, and the metric that proves this stage is correct before connecting it downstream.
Paged attention
Attention builds a content-dependent weighted average. Queries describe what each position needs, keys describe what each position offers, and values carry the information. Scaling by the square root of key dimension prevents dot products from pushing softmax into saturation.
Boundary check: document its accepted input, output shape, mutable state, failure modes, and the metric that proves this stage is correct before connecting it downstream.
Continuous batching
Continuous batching admits and retires sequences at iteration boundaries instead of waiting for a fixed batch to finish. Throughput improves, while fairness, memory admission, cancellation, and tail latency become scheduling problems.
Boundary check: document its accepted input, output shape, mutable state, failure modes, and the metric that proves this stage is correct before connecting it downstream.
Architecture review checklist
- Every arrow has a documented shape, dtype, unit, or schema.
- Training and evaluation paths cannot leak information into each other.
- Randomness is seeded and captured in experiment metadata.
- Expensive stages expose timing, memory, throughput, and error metrics.
- Each feedback loop has a stop condition and a rollback strategy.
- Small reference implementations exist for numerical comparisons.