Inside LLM Inference

How does a language model actually become served tokens on a GPU - and why is it so much slower than the hardware's headline TFLOPS promise?

Inside LLM Inference follows a single request through the entire stack: from a PyTorch call down through the CUDA runtime, the streaming multiprocessor, the warp, and the memory system, to the arithmetic cores that finally do the work - and back up to the throughput and cost you pay for.

Every claim is measured, not asserted. A production model runs on 2026's newest silicon - NVIDIA's Blackwell generation - across four inference engines (vLLM, SGLang, TensorRT-LLM, and Hugging Face Transformers), profiled to individual GPU kernels with hardware counters and cross-checked against the physics that predicts them.

You start from zero - Chapter 0 defines every term - and finish able to read a live server:

• classify a kernel as memory- or compute-bound, and prove it with a clock-sensitivity test
• size a KV-cache pool and the batch it admits from a model's own dimensions
• read sm_active and sm_occupancy off a running GPU and know what they mean
• choose an inference engine on evidence, not vibes