Section 2 of 9
Linear algebra, probability, statistics, calculus, information theory, and optimization — the math the rest of the site leans on.
6 items in this section.
Everyone trains classifiers with cross-entropy, far fewer can say why that loss and not another. It isn't a design choice — it falls out of one principle: maximize the probability the model assigns to the data. Cross-entropy, log-loss, MSE, and KL are the same idea wearing four different hats.
KL divergence turns up in cross-entropy training, VAEs and variational inference, knowledge distillation, and the RLHF leash. That isn't a coincidence — a huge fraction of ML is secretly the sentence 'make distribution q close to distribution p,' and KL is what that sentence reduces to.
A matrix turns and stretches every arrow in space — except for a few special directions it only stretches. Those are eigenvectors, the natural grain of the transformation.
Once you know a matrix's un-turnable directions, you can rewrite the whole thing as three simple moves: rotate, stretch, rotate back. That's eigendecomposition.
Eigenvectors are beautiful on paper. Computing them for a 100,000×100,000 matrix is an engineering problem — and the tool that does it, iteratively, is an eigensolver.
The evidence lower bound looks like a formula handed down from on high. It isn't invented — it's forced. When the evidence integral is intractable but you still want maximum likelihood, there is essentially one move available, and the ELBO is what it produces.