Section 4 of 9
PyTorch, MLP/CNN/RNN, model compression, distillation, RL, generative models.
12 items in this section.
An architecture is a claim about your data, and every claim here is measured: why a stack of linear layers is one straight line, what a CNN's first layer actually learns, how fast a receptive field grows, where an RNN's gradient dies — and PyTorch as three ideas: a tensor, a tape, and a module.
Training a small model on a large model's probabilities instead of the true labels often gives a better small model. Why that works, taken apart one ingredient at a time; what happens when the labels the teacher learned from were wrong; whether a stronger teacher makes a stronger student; what to copy, on which data and about which view of it; how long to train and how big the student should be before any of this matters; why copying the teacher and being right are different numbers; how a student can beat its teacher and which of its flaws it inherits; what changes for language models without logits; and when distillation pays for itself — measured on CIFAR-10 and on a small language model trained on this site's notes.
What it actually means to reverse-engineer a neural network into features and circuits: the residual stream as a linear object, the linear representation hypothesis, superposition and why features outnumber neurons, sparse autoencoders as dictionary learning, the QK/OV decomposition of attention, induction heads, and the causal methods — activation and path patching — that turn a plausible story into a verified mechanism.
Building a tiny AlphaGo from scratch on a 5×5 board: board representation, a small CNN policy network, a value network, and then Monte Carlo Tree Search — with each piece introduced only when the previous version visibly runs out of road. Ends at a self-play loop that is recognisably AlphaZero in miniature.
One number at the end of the network has to become an instruction for every number inside it. That trip is the whole of training: backpropagation as the chain rule run backwards through a graph, the product of many numbers that decides whether the instruction survives the journey, the initialization arithmetic that keeps that product near one and the normalization, clipping and precision choices that answer the other half of the problem, and regularization as the deliberate corruption that makes the result generalize — every claim carried by one worked example, from two weights done by hand to a network that overfits.
The two layers differ in one axis — statistics across the batch, or across the features of one example — and every practical difference between them, including why transformers use one and not the other, follows from that.
Networks represent more concepts than they have dimensions by storing them as almost-orthogonal directions and tolerating the interference. That is why single neurons look like nonsense, and why sparse autoencoders exist.
Distillation temperature softens the teacher and the student at once — a different operation from the sampling temperature in text generation. The T² in front of the loss is not a tuning constant: it cancels a gradient that would otherwise shrink as the square of the temperature.
In distillation the student learns the teacher's answers at the points it is asked about, and nothing else. That set of points needs no labels, is a design decision rather than raw material, and moved the result more than the choice of teacher did.
A distilled model can be graded on how closely it reproduces its teacher, or on how often it is right. With a strong teacher the two look like the same number. They come apart exactly where it matters, and each answers a question the other cannot.
Distil a network into a copy of itself, same architecture, same data, no compression anywhere — and it can come out better than the model it learned from. Whether it does depends on how much the original memorized.
A trained classifier answers with one class and a list of numbers nobody reads. Those numbers say which wrong answers it considered nearly right, no label can carry them, and they are most of what a distilled student learns from.