Section 9 of 9
How machine-learning algorithms actually run on hardware: GPUs, CUDA and C++, memory and parallelism, kernels, numerical computing, LLM inference and serving, distributed systems, and performance engineering.
9 items in this section.
Why the same GPU is nineteen times faster than the CPU at one operation, eleven times at the next, and slower at a third — worked out on a real Tesla T4, from two numbers per machine: how fast it computes and how fast it moves bytes. Along the way: warps and SIMT, latency hiding, the memory hierarchy, arithmetic intensity, the roofline, and a power limit that moves the ceiling.
Most of the arithmetic in a neural network is matrix multiplication. Written the obvious way it is correct, and fifty times slower than torch.mm on the same GPU. Where that difference comes from, one change at a time on a Tesla T4: coalescing, tiles, shared memory, registers, tensor cores and precision — and why almost all of it is about where the numbers are kept rather than how they are multiplied.
What a GPU is actually doing while a language model answers: the generation loop, the prefill pass that reads the prompt and the decode steps that write the answer, the two waits a user feels — time to first token and time per output token — and why the second is limited by moving the model's weights through memory. Measured end to end on a Tesla T4, including why a token costs nine times what its bytes say, what a real kernel trace shows instead, and how latency and throughput pull apart when many people share one card.
One line of Python — index.search(query, k=10) — taken apart layer by layer, with a small nearest-neighbour library written in C++ and CUDA to take it apart with. Where numbers live and who frees them, what a copy of a gigabyte costs, why the order of additions changes both the speed and the answer, when an abstraction is free and when it is ruinous, what a template costs to compile, what happens between the call returning and the GPU finishing — and why the same search runs 25 times faster after eight rewrites that change nothing about what it computes. Measured end to end on a Kaggle T4.
A conversation's key/value cache grows one token at a time and nobody knows how long it will get. Give every request a contiguous buffer sized for the worst case and one Tesla T4 holds 573 of them, most of that memory reserved for answers that never arrive. Hand the same card out in blocks of 16 tokens and it holds several thousand. Measured on vLLM 0.29: what the block table costs the attention kernel, how the engine decides that two requests hold the same block, how much of the card the cache actually gets, and how many sequences all of it makes room for.
Follow one message into a 2026 AI server: through CPU memory, HBM4, compute dies, matrix units and the links between GPUs, then back out as an answer. Original interactive diagrams show where weights already live, what a request creates, and what the hardware waits for. Based on NVIDIA Rubin and AMD CDNA 5.
torchisotonic and the paper behind it, in plain words. A curve that may only go up is a ten-line walk on a CPU, and a walk is the one thing a GPU cannot share out: on ten million numbers a far bigger GPU can be even slower. The way out is to stop walking and draw instead, since the fit is a string pulled tight under the running totals, and then to stop copying the pieces of that string that never change.
A GPU's advertised TFLOPs describe one of two limits. Which one an operation actually hits is decided by a single ratio — FLOPs per byte moved — and most operations in a neural network sit far below the crossover.
Most GPU calls return before the GPU has done anything. That is the point — and it is also why a buffer can hold the previous batch's results, why a free can stall for 50 ms, and why an 'async' copy sometimes is not.