Stanford CS336 Lecture 10 traces LLM inference optimizations to memory-bound decoding

Meng Shao describes Stanford CS336 Lecture 10, taught by Percy Liang, which links LLM inference optimizations to one constraint: autoregressive decoding is memory-bound and cannot be parallelized along the sequence.

The lecture treats every optimization as a way to reduce the bytes moved per decoding step.

According to the post, the lecture groups these fixes into three areas: architecture changes that trade accuracy for speed (GQA, MLA, CLA, quantization, pruning), lossless speculative decoding, and systems techniques such as continuous batching and paged attention. The lecture's formulas are derived and checked symbolically with sympy in an executable-notes format.