lenatriestounderstand

Engineering Understanding

Lena Tries
To Understand

Interactive notes on AI, ML, data, systems, and the beautiful things behind them.

Read . Run . Understand .

Py Python-first
Run in-browser

Latest long reads

See all 47 notes

Matrix Multiplication: The Operation Behind Modern AI

Most of the arithmetic in a neural network is matrix multiplication. Written the obvious way it is correct, and fifty times slower than torch.mm on the same GPU. Where that difference comes from, one change at a time on a Tesla T4: coalescing, tiles, shared memory, registers, tensor cores and precision — and why almost all of it is about where the numbers are kept rather than how they are multiplied.

matrix-multiplication
cuda
tiling
shared-memory
+3

How GPUs Actually Run Machine Learning

Why the same GPU is nineteen times faster than the CPU at one operation, eleven times at the next, and slower at a third — worked out on a real Tesla T4, from two numbers per machine: how fast it computes and how fast it moves bytes. Along the way: warps and SIMT, latency hiding, the memory hierarchy, arithmetic intensity, the roofline, and a power limit that moves the ceiling.

gpu
cuda
roofline
memory-bandwidth
+2

Training Neural Networks: Backprop, Regularization, and Everything in Between

One number at the end of the network has to become an instruction for every number inside it. That trip is the whole of training: backpropagation as the chain rule run backwards through a graph, the product of many numbers that decides whether the instruction survives the journey, the initialization arithmetic that keeps that product near one and the normalization, clipping and precision choices that answer the other half of the problem, and regularization as the deliberate corruption that makes the result generalize — every claim carried by one worked example, from two weights done by hand to a network that overfits.

deep-learning
backpropagation
gradient-descent
initialization
+1

Teaching a Neural Network to Play Go

Building a tiny AlphaGo from scratch on a 5×5 board: board representation, a small CNN policy network, a value network, and then Monte Carlo Tree Search — with each piece introduced only when the previous version visibly runs out of road. Ends at a self-play loop that is recognisably AlphaZero in miniature.

reinforcement-learning
monte-carlo-tree-search
self-play
cnn
+2

Shorts

See all 31 shorts

Eigenvectors: The Directions a Matrix Won't Turn

A matrix turns and stretches every arrow in space — except for a few special directions it only stretches. Those are eigenvectors, the natural grain of the transformation.

linear-algebra
eigenvectors
pca
spectral-clustering

Eigendecomposition: Reading a Matrix in Its Own Language

Once you know a matrix's un-turnable directions, you can rewrite the whole thing as three simple moves: rotate, stretch, rotate back. That's eigendecomposition.

linear-algebra
eigenvectors
pca
spectral-clustering

Eigensolvers: Why Finding Eigenvectors Is a Whole Job

Eigenvectors are beautiful on paper. Computing them for a 100,000×100,000 matrix is an engineering problem — and the tool that does it, iteratively, is an eigensolver.

linear-algebra
eigenvectors
numerical-computing
spectral-clustering

SFT: Imitation, and the Ceiling It Hits

Supervised fine-tuning is the imitation step of post-training: show the model (prompt → ideal answer) pairs and minimize cross-entropy on the target tokens. It teaches the format of being an assistant — and hits a ceiling that preference learning exists to break.

sft
fine-tuning
post-training
instruction-tuning

Categories

Storage and Streaming

Storage formats, streaming, object stores, relational databases.

Math for ML

Linear algebra, probability, statistics, calculus, information theory, and optimization — the math the rest of the site leans on.

Classical ML

sklearn, boosting, clustering, feature engineering, model evaluation, A/B testing, interpretability, fairness.

Deep Learning

PyTorch, MLP/CNN/RNN, model compression, distillation, RL, generative models.

Time Series

Data preparation, forecasting models, deep learning, foundation models, evaluation, maintenance.

LLM

Transformer architecture, the generation loop, prompt engineering, structured outputs, agent patterns.

Embeddings and Retrieval

Embeddings, vector databases, sparse and hybrid retrieval, chunking, reranking, and the search lineage from BoW to E5.

Econometrics

Endogeneity, IV, RDD, panel data, causal ML.

ML Systems

How models actually run on hardware: GPUs, memory bandwidth and the roofline, kernels, LLM inference and serving, multi-GPU parallelism, capacity planning.

Labs

See all 10 labs

Deep Learning · Lab

Training Neural Networks: Backprop, Regularization, and Everything in Between

One number at the end of the network has to become an instruction for every number inside it. That trip is the whole of training: backpropagation as the chain rule run backwards through a graph, the product of many numbers that decides whether the instruction survives the journey, the initialization arithmetic that keeps that product near one and the normalization, clipping and precision choices that answer the other half of the problem, and regularization as the deliberate corruption that makes the result generalize — every claim carried by one worked example, from two weights done by hand to a network that overfits.

ML Systems · Lab

Matrix Multiplication: The Operation Behind Modern AI

Most of the arithmetic in a neural network is matrix multiplication. Written the obvious way it is correct, and fifty times slower than torch.mm on the same GPU. Where that difference comes from, one change at a time on a Tesla T4: coalescing, tiles, shared memory, registers, tensor cores and precision — and why almost all of it is about where the numbers are kept rather than how they are multiplied.

ML Systems · Lab

How GPUs Actually Run Machine Learning

Why the same GPU is nineteen times faster than the CPU at one operation, eleven times at the next, and slower at a third — worked out on a real Tesla T4, from two numbers per machine: how fast it computes and how fast it moves bytes. Along the way: warps and SIMT, latency hiding, the memory hierarchy, arithmetic intensity, the roofline, and a power limit that moves the ceiling.

Deep Learning · Lab

Teaching a Neural Network to Play Go

Building a tiny AlphaGo from scratch on a 5×5 board: board representation, a small CNN policy network, a value network, and then Monte Carlo Tree Search — with each piece introduced only when the previous version visibly runs out of road. Ends at a self-play loop that is recognisably AlphaZero in miniature.

Favorites

Matrix Multiplication: The Operation Behind Modern AI

Most of the arithmetic in a neural network is matrix multiplication. Written the obvious way it is correct, and fifty times slower than torch.mm on the same GPU. Where that difference comes from, one change at a time on a Tesla T4: coalescing, tiles, shared memory, registers, tensor cores and precision — and why almost all of it is about where the numbers are kept rather than how they are multiplied.

matrix-multiplication
cuda
tiling
shared-memory
+3

How GPUs Actually Run Machine Learning

Why the same GPU is nineteen times faster than the CPU at one operation, eleven times at the next, and slower at a third — worked out on a real Tesla T4, from two numbers per machine: how fast it computes and how fast it moves bytes. Along the way: warps and SIMT, latency hiding, the memory hierarchy, arithmetic intensity, the roofline, and a power limit that moves the ceiling.

gpu
cuda
roofline
memory-bandwidth
+2

Learning Rules from Examples: How Can a Model Discover a Hidden Rule?

A tiny synthetic world where the rule is hidden and the evidence is controlled. Version spaces and what a dataset actually tells you, why a deterministic likelihood makes generalization entirely a matter of prior, shortcut features, expected information gain and active learning, compositional and length generalization, bottom-up program search with observational-equivalence pruning, LLM-guided synthesis, and libraries that grow their own primitives — each turned into a small experiment with a deterministic verifier.

induction
program-synthesis
inductive-bias
hypothesis-space
+6

Mechanistic Interpretability: Looking Inside the Black Box

What it actually means to reverse-engineer a neural network into features and circuits: the residual stream as a linear object, the linear representation hypothesis, superposition and why features outnumber neurons, sparse autoencoders as dictionary learning, the QK/OV decomposition of attention, induction heads, and the causal methods — activation and path patching — that turn a plausible story into a verified mechanism.

interpretability
mechanistic-interpretability
superposition
sparse-autoencoders
+1

Text Clustering: Representation Over Algorithm

One corpus of arXiv abstracts, clustered nine ways — and the finding that on real text the representation usually decides more than the algorithm.

clustering
text
embeddings
tf-idf
+2

Why LLMs Sound Emotional — and Whether They Understand Emotion

Two halves of one question. Why an LLM's emotional language is generated, not felt, and where it comes from — preference data, reward models, safety tuning, system prompts; and whether it can actually understand emotion in others — theory of mind, the recognition benchmarks, and where the fluent performance turns brittle.

emotion
empathy
safety-tuning
theory-of-mind

Why Different Models Feel Like Different Personalities

Same engine, different knobs: why one model reads as warm and another as businesslike. Traces 'personality' to concrete training choices — data mix, preference guidelines, reward model, safety tuning, character training — and folds in sycophancy as a personality artifact of RLHF.

personality
rlhf

Embeddings and Retrieval

Embeddings: How Geometry Pretends to Be Meaning

Embeddings aren't an encoding of text — they're an attempt to make geometry behave as if it carried meaning. What it means to compress text into a fixed-length vector, how contrastive learning turns statistical structure into distances and directions, why cosine similarity works (and when it stops), how dimension, chunking, context window, and reranking change the physics of a retrieval pipeline, and where embeddings lie usefully.

embeddings

Pricing and Elasticity

Pricing as the worked-example for the Econometrics track — why price is endogenous, the identification strategies (IV, FE, RD, DML, CATE), cross-price elasticity and cannibalization, and what-if analysis with its constant-elasticity caveats.

pricing
elasticity
demand
causal-inference

Causal ML Beyond Econometrics

Causal ML at the meeting point of econometrics and ML — ATE vs CATE, uplift modelling, DML and orthogonal-moments inference, causal forests, counterfactual prediction, off-policy evaluation, and the standard mistakes from treating predictive models as causal.

causal-ml
uplift
dml
cate

Fine-Tuning LLMs: When the Weight Delta Is Worth It

Fine-tuning is not prompt repair. It is a decision to write a reusable parameter delta into an existing checkpoint. That delta changes future logits, defaults, and trade-offs. This note is about when that is worth doing: what fine-tuning actually buys, how to tell whether a gap belongs in the weights, and why data, evals, forgetting, and probability shape matter more than the slogan 'just fine-tune it'.

fine-tuning
residual-stream

Embeddings and Retrieval

Chunking Strategies

Chunking turns out to be more architectural than it looks. A walk through what a chunk has to satisfy at six different stages simultaneously, what RecursiveCharacterTextSplitter actually does under the hood, and what the 2024–2026 toolbox — late chunking, contextual retrieval, contextualized chunk embeddings, BGE-M3 multi-functionality, ColBERT late interaction — is actually for.

chunking
rag

Embeddings and Retrieval

How Text Became Geometry

Sixty years of incremental work behind the modern embedding vector. From bag-of-words and BM25, through Word2Vec's distributional hypothesis, to contextual embeddings and contrastive retrieval models. The point isn't trivia — it's that each older idea is still in production today, and the picture only makes sense once you've seen the road that led here.

embeddings
history
retrieval

The Hindsight Corpus: Time in LLM Pretraining Data

Saying a model was 'trained on text written before T' invites a picture of human knowledge as of T. The actual corpus is volumetrically skewed toward recent years, dominated by retroactively-edited sources like Wikipedia, missing reliable per-document timestamps, and survivor-biased for older periods. The mechanisms, the failure modes that fall out, what's silently absent from datasheets, and what time-aware pretraining would have to do differently.

pretraining
training-data
temporal

The Missing Now: Temporal Grounding in LLM Agents

A chat transcript preserves order but not elapsed time, world state, or whether earlier hypotheses have expired. For long-running agents, temporal grounding is a runtime problem, not a model problem — what 'now' actually is, the failure modes that fall out when context gets treated as state, the primitives (clocks, event logs, state reducers, expectations, monitors) that close the gap, and how to measure whether it works.

agents
temporal-grounding
state-management

The Physics of Hallucination

What hallucination looks like at the level of the transformer's internal computation — distributed representations, signal competition in the residual stream, the softmax bottleneck, the activation-output gap, and the architectural reasons there is no first-class epistemic channel.

hallucinations