Labs
Where a long read argues the ideas, a lab proves them — a runnable, reproducible notebook that takes a claim apart on a real model, with code you can run, change, and break. Executed offline; every number reproduces on a re-render. Browse the long reads →.
torchisotonic and the paper behind it, in plain words. A curve that may only go up is a ten-line walk on a CPU, and a walk is the one thing a GPU cannot share out: on ten million numbers a far bigger GPU can be even slower. The way out is to stop walking and draw instead, since the fit is a string pulled tight under the running totals, and then to stop copying the pieces of that string that never change.
One line of Python — index.search(query, k=10) — taken apart layer by layer, with a small nearest-neighbour library written in C++ and CUDA to take it apart with. Where numbers live and who frees them, what a copy of a gigabyte costs, why the order of additions changes both the speed and the answer, when an abstraction is free and when it is ruinous, what a template costs to compile, what happens between the call returning and the GPU finishing — and why the same search runs 25 times faster after eight rewrites that change nothing about what it computes. Measured end to end on a Kaggle T4.
A model reads a support ticket, picks a department, and returns 0.94. RLCR — training that number with a reward only an honest probability can maximise — and a small typed-decision model built from an ordinary encoder: one forward pass, every option a parallel branch, and the same weights trained seven ways to find out what the reinforcement learning actually adds.
What a GPU is actually doing while a language model answers: the generation loop, the prefill pass that reads the prompt and the decode steps that write the answer, the two waits a user feels — time to first token and time per output token — and why the second is limited by moving the model's weights through memory. Measured end to end on a Tesla T4, including why a token costs nine times what its bytes say, what a real kernel trace shows instead, and how latency and throughput pull apart when many people share one card.
Training a small model on a large model's probabilities instead of the true labels often gives a better small model. Why that works, taken apart one ingredient at a time; what happens when the labels the teacher learned from were wrong; whether a stronger teacher makes a stronger student; what to copy, on which data and about which view of it; how long to train and how big the student should be before any of this matters; why copying the teacher and being right are different numbers; how a student can beat its teacher and which of its flaws it inherits; what changes for language models without logits; and when distillation pays for itself — measured on CIFAR-10 and on a small language model trained on this site's notes.
One number at the end of the network has to become an instruction for every number inside it. That trip is the whole of training: backpropagation as the chain rule run backwards through a graph, the product of many numbers that decides whether the instruction survives the journey, the initialization arithmetic that keeps that product near one and the normalization, clipping and precision choices that answer the other half of the problem, and regularization as the deliberate corruption that makes the result generalize — every claim carried by one worked example, from two weights done by hand to a network that overfits.
Most of the arithmetic in a neural network is matrix multiplication. Written the obvious way it is correct, and fifty times slower than torch.mm on the same GPU. Where that difference comes from, one change at a time on a Tesla T4: coalescing, tiles, shared memory, registers, tensor cores and precision — and why almost all of it is about where the numbers are kept rather than how they are multiplied.
Why the same GPU is nineteen times faster than the CPU at one operation, eleven times at the next, and slower at a third — worked out on a real Tesla T4, from two numbers per machine: how fast it computes and how fast it moves bytes. Along the way: warps and SIMT, latency hiding, the memory hierarchy, arithmetic intensity, the roofline, and a power limit that moves the ceiling.
Building a tiny AlphaGo from scratch on a 5×5 board: board representation, a small CNN policy network, a value network, and then Monte Carlo Tree Search — with each piece introduced only when the previous version visibly runs out of road. Ends at a self-play loop that is recognisably AlphaZero in miniature.
A tiny synthetic world where the rule is hidden and the evidence is controlled. Version spaces and what a dataset actually tells you, why a deterministic likelihood makes generalization entirely a matter of prior, shortcut features, expected information gain and active learning, compositional and length generalization, bottom-up program search with observational-equivalence pruning, LLM-guided synthesis, and libraries that grow their own primitives — each turned into a small experiment with a deterministic verifier.
An architecture is a claim about your data, and every claim here is measured: why a stack of linear layers is one straight line, what a CNN's first layer actually learns, how fast a receptive field grows, where an RNN's gradient dies — and PyTorch as three ideas: a tensor, a tape, and a module.
What it actually means to reverse-engineer a neural network into features and circuits: the residual stream as a linear object, the linear representation hypothesis, superposition and why features outnumber neurons, sparse autoencoders as dictionary learning, the QK/OV decomposition of attention, induction heads, and the causal methods — activation and path patching — that turn a plausible story into a verified mechanism.
Chunking turns out to be more architectural than it looks. A walk through what a chunk has to satisfy at six different stages simultaneously, what RecursiveCharacterTextSplitter actually does under the hood, and what the 2024–2026 toolbox — late chunking, contextual retrieval, contextualized chunk embeddings, BGE-M3 multi-functionality, ColBERT late interaction — is actually for.
One corpus of arXiv abstracts, clustered nine ways — and the finding that on real text the representation usually decides more than the algorithm.
Embeddings aren't an encoding of text — they're an attempt to make geometry behave as if it carried meaning. What it means to compress text into a fixed-length vector, how contrastive learning turns statistical structure into distances and directions, why cosine similarity works (and when it stops), how dimension, chunking, context window, and reranking change the physics of a retrieval pipeline, and where embeddings lie usefully.