lenatriestounderstand

Latest

Every note, newest first

The full feed, sorted by the most recent update. Use the filter to narrow by topic, tag, or track — or browse by category →.

47 notes

Matrix Multiplication: The Operation Behind Modern AI

Most of the arithmetic in a neural network is matrix multiplication. Written the obvious way it is correct, and fifty times slower than torch.mm on the same GPU. Where that difference comes from, one change at a time on a Tesla T4: coalescing, tiles, shared memory, registers, tensor cores and precision — and why almost all of it is about where the numbers are kept rather than how they are multiplied.

matrix-multiplication
cuda
tiling
shared-memory
+3

How GPUs Actually Run Machine Learning

Why the same GPU is nineteen times faster than the CPU at one operation, eleven times at the next, and slower at a third — worked out on a real Tesla T4, from two numbers per machine: how fast it computes and how fast it moves bytes. Along the way: warps and SIMT, latency hiding, the memory hierarchy, arithmetic intensity, the roofline, and a power limit that moves the ceiling.

gpu
cuda
roofline
memory-bandwidth
+2

Training Neural Networks: Backprop, Regularization, and Everything in Between

One number at the end of the network has to become an instruction for every number inside it. That trip is the whole of training: backpropagation as the chain rule run backwards through a graph, the product of many numbers that decides whether the instruction survives the journey, the initialization arithmetic that keeps that product near one and the normalization, clipping and precision choices that answer the other half of the problem, and regularization as the deliberate corruption that makes the result generalize — every claim carried by one worked example, from two weights done by hand to a network that overfits.

deep-learning
backpropagation
gradient-descent
initialization
+1

Teaching a Neural Network to Play Go

Building a tiny AlphaGo from scratch on a 5×5 board: board representation, a small CNN policy network, a value network, and then Monte Carlo Tree Search — with each piece introduced only when the previous version visibly runs out of road. Ends at a self-play loop that is recognisably AlphaZero in miniature.

reinforcement-learning
monte-carlo-tree-search
self-play
cnn
+2

Learning Rules from Examples: How Can a Model Discover a Hidden Rule?

A tiny synthetic world where the rule is hidden and the evidence is controlled. Version spaces and what a dataset actually tells you, why a deterministic likelihood makes generalization entirely a matter of prior, shortcut features, expected information gain and active learning, compositional and length generalization, bottom-up program search with observational-equivalence pruning, LLM-guided synthesis, and libraries that grow their own primitives — each turned into a small experiment with a deterministic verifier.

induction
program-synthesis
inductive-bias
hypothesis-space
+6

Neural Network Architectures and PyTorch (MLP, CNN, RNN/LSTM)

An architecture is a claim about your data, and every claim here is measured: why a stack of linear layers is one straight line, what a CNN's first layer actually learns, how fast a receptive field grows, where an RNN's gradient dies — and PyTorch as three ideas: a tensor, a tape, and a module.

deep-learning
pytorch
cnn
lstm

Mechanistic Interpretability: Looking Inside the Black Box

What it actually means to reverse-engineer a neural network into features and circuits: the residual stream as a linear object, the linear representation hypothesis, superposition and why features outnumber neurons, sparse autoencoders as dictionary learning, the QK/OV decomposition of attention, induction heads, and the causal methods — activation and path patching — that turn a plausible story into a verified mechanism.

interpretability
mechanistic-interpretability
superposition
sparse-autoencoders
+1

Clustering Methods

The main clustering algorithms organised by the shape each assumes a cluster has — centroid, covariance, density, graph cut, hierarchy — where each fits, and a comparative table of the family.

clustering
k-means
gmm
hdbscan
+2

Text Clustering: Representation Over Algorithm

One corpus of arXiv abstracts, clustered nine ways — and the finding that on real text the representation usually decides more than the algorithm.

clustering
text
embeddings
tf-idf
+2

Clustering in Practice

Practical wrap-up for clustering work — cluster labelling, two-corpus comparison, validation, choosing K when it isn't known, hyperparameter tuning order, common mistakes, and reproducibility.

clustering
topic-modelling

How LLMs Learn Human Preferences: RLHF, RLAIF and Beyond

How a next-token predictor is bent toward what humans prefer: reward modeling, PPO, RLAIF/Constitutional AI, and the offline-preference family — and why so much of a model's 'personality' and 'emotional' behavior is decided in this stage.

rlhf
rlaif
reward-model
ppo
+1

Why Different Models Feel Like Different Personalities

Same engine, different knobs: why one model reads as warm and another as businesslike. Traces 'personality' to concrete training choices — data mix, preference guidelines, reward model, safety tuning, character training — and folds in sycophancy as a personality artifact of RLHF.

personality
rlhf

Why LLMs Sound Emotional — and Whether They Understand Emotion

Two halves of one question. Why an LLM's emotional language is generated, not felt, and where it comes from — preference data, reward models, safety tuning, system prompts; and whether it can actually understand emotion in others — theory of mind, the recognition benchmarks, and where the fluent performance turns brittle.

emotion
empathy
safety-tuning
theory-of-mind

Embeddings and Retrieval

Embeddings: How Geometry Pretends to Be Meaning

Embeddings aren't an encoding of text — they're an attempt to make geometry behave as if it carried meaning. What it means to compress text into a fixed-length vector, how contrastive learning turns statistical structure into distances and directions, why cosine similarity works (and when it stops), how dimension, chunking, context window, and reranking change the physics of a retrieval pipeline, and where embeddings lie usefully.

embeddings

Endogeneity

Endogeneity — what it is formally, why OLS becomes biased and inconsistent under it, the three main sources (omitted variables, simultaneity, measurement error) plus self-selection, how to detect it in practice, and why this is the central problem the rest of the Econometrics track is built around.

endogeneity
causal-inference
ols
ovb

Instrumental Variables (2SLS)

Instrumental Variables / 2SLS as the foundational tool for causal inference in observational data when the treatment is endogenous — the IV idea in plain terms, the four conditions a valid instrument must satisfy (relevance, exclusion, independence, monotonicity), the two-stage estimator, weak-instrument diagnostics beyond the F>10 rule, what IV actually identifies (LATE, not ATE), and worked instrument choices for pricing problems.

iv
2sls
causal-inference
late

Regression Discontinuity

Regression discontinuity as a quasi-experimental method built on a threshold-based assignment rule — the continuity assumption that does the identification work, sharp and fuzzy variants, the modern bandwidth and diagnostic toolkit, and why the resulting effect is local to the cutoff rather than population-wide.

rdd
causal-inference
quasi-experiment
late

Control Function Approach

The control function approach as the residual-based alternative to 2SLS — when the two agree, what CF's structural assumption actually is, where an ML-flexible first stage helps and where it does not, and how the idea extends through cross-fitting into double / debiased machine learning.

causal-inference
iv
ml-econometrics
double-ml

Panel Data (Fixed Effects)

Fixed effects for panel data — the within-transformation, what it controls for and what it misses, cluster-robust standard errors, the LSDV equivalence, the staggered-DiD problem, and when FE is the right tool.

fixed-effects
did
panel
causal-inference

Causal ML Beyond Econometrics

Causal ML at the meeting point of econometrics and ML — ATE vs CATE, uplift modelling, DML and orthogonal-moments inference, causal forests, counterfactual prediction, off-policy evaluation, and the standard mistakes from treating predictive models as causal.

causal-ml
uplift
dml
cate

Pricing and Elasticity

Pricing as the worked-example for the Econometrics track — why price is endogenous, the identification strategies (IV, FE, RD, DML, CATE), cross-price elasticity and cannibalization, and what-if analysis with its constant-elasticity caveats.

pricing
elasticity
demand
causal-inference

Fine-Tuning LLMs: When the Weight Delta Is Worth It

Fine-tuning is not prompt repair. It is a decision to write a reusable parameter delta into an existing checkpoint. That delta changes future logits, defaults, and trade-offs. This note is about when that is worth doing: what fine-tuning actually buys, how to tell whether a gap belongs in the weights, and why data, evals, forgetting, and probability shape matter more than the slogan 'just fine-tune it'.

fine-tuning
residual-stream

Fine-Tuning LLMs: Post-Training Is a Pipeline, Not a Step

Post-training is not one fine-tuning method. It is a sequence of objective signals. Continued pretraining teaches substrate, SFT teaches examples and defaults, preference optimization teaches comparisons, RLVR teaches verifiable trajectories, and distillation transfers the resulting behavior. The important design question is not which acronym is fashionable, but which stage matches the behavior you are trying to install.

fine-tuning
post-training
sft
dpo
+3

Fine-Tuning LLMs: Modern Post-Training Deep Dive

A reference-style deep dive into the modern knobs around post-training: preference optimization variants, LoRA and PEFT methods, memory-efficient full fine-tuning, model merging, distillation, tooling, and serving. Read this after the pipeline note, once you know which stage you actually need.

fine-tuning
dpo
lora
qlora
+4

How LLM Generation Works: Transformer, Sampling, Tokens, Batching, and Validation

What happens inside a transformer when you send a prompt, and how the practical knobs — temperature, max_tokens, structured outputs, batching strategy, retry-with-catch-up — fall out of that picture.

llm
transformers
attention
tokenization

LLM Agent Architectures

Agent architecture is where LLM engineering stops being mostly about prompts and starts looking like distributed systems. Covers workflows vs agents, the classical loop, five paradigms (ReAct, Function Calling, Plan-and-Execute, Reflection, CodeAct), MCP as the protocol layer above per-vendor function calling, multi-agent patterns, computer use, memory and resumability, production failure modes including indirect prompt injection, tool security, cost levers, and observability.

agent-architectures
tool-use
mcp
multi-agent

Embeddings and Retrieval

Chunking Strategies

Chunking turns out to be more architectural than it looks. A walk through what a chunk has to satisfy at six different stages simultaneously, what RecursiveCharacterTextSplitter actually does under the hood, and what the 2024–2026 toolbox — late chunking, contextual retrieval, contextualized chunk embeddings, BGE-M3 multi-functionality, ColBERT late interaction — is actually for.

chunking
rag

Embeddings and Retrieval

How Text Became Geometry

Sixty years of incremental work behind the modern embedding vector. From bag-of-words and BM25, through Word2Vec's distributional hypothesis, to contextual embeddings and contrastive retrieval models. The point isn't trivia — it's that each older idea is still in production today, and the picture only makes sense once you've seen the road that led here.

embeddings
history
retrieval

Interpretability and Production Maintenance for Deep Learning Time Series

A practical guide to the post-training lifecycle of deep learning forecasters, covering SHAP-based prediction explanations, lightweight recalibration for drift, and the broader production-maintenance roadmap.

interpretability
shap
model-maintenance

The Hindsight Corpus: Time in LLM Pretraining Data

Saying a model was 'trained on text written before T' invites a picture of human knowledge as of T. The actual corpus is volumetrically skewed toward recent years, dominated by retroactively-edited sources like Wikipedia, missing reliable per-document timestamps, and survivor-biased for older periods. The mechanisms, the failure modes that fall out, what's silently absent from datasheets, and what time-aware pretraining would have to do differently.

pretraining
training-data
temporal

The Missing Now: Temporal Grounding in LLM Agents

A chat transcript preserves order but not elapsed time, world state, or whether earlier hypotheses have expired. For long-running agents, temporal grounding is a runtime problem, not a model problem — what 'now' actually is, the failure modes that fall out when context gets treated as state, the primitives (clocks, event logs, state reducers, expectations, monitors) that close the gap, and how to measure whether it works.

agents
temporal-grounding
state-management

Attention Is All You Need — But Not All Attention Is the Same

Why modern LLMs are no longer just decoder-only transformers with standard multi-head attention. Attention has become a design space — MHA, MQA, GQA, MLA, sliding-window, sparse, linear, recurrent, hybrid — plus position encoding, attention sinks, and KV-cache compression. Each variant solves a different bottleneck.

attention
kv-cache
long-context

Prompt Engineering

What separates a working LLM prompt from a flaky one in 2026 — instruction hierarchy, in-context learning, chain-of-thought, structured outputs, reasoning-model specifics, and the prompt-injection trust boundary.

prompt-engineering

The Physics of Hallucination

What hallucination looks like at the level of the transformer's internal computation — distributed representations, signal competition in the residual stream, the softmax bottleneck, the activation-output gap, and the architectural reasons there is no first-class epistemic channel.

hallucinations

Practical Training Recipes for Deep Learning Time Series

A practical checklist for training deep learning time-series models, covering windowing, initialization, embeddings, regularization, optimizers, learning-rate schedules, loss functions, and the Keras training workflow.

training
regularization
optimization

Classical Statistical Forecasting: ARIMA, SARIMA, and SARIMAX

A practical guide to ARIMA, SARIMA, and SARIMAX: how classical statistical forecasting models handle autocorrelation, differencing, seasonality, external regressors, diagnostics, and where they still fit in modern forecasting.

arima
sarima
sarimax
statistical-models

Deep Learning Architectures for Time Series

An overview of deep learning architectures for time-series forecasting: LSTM, TCN, DeepAR, N-BEATS, and TFT, with a focus on input shapes, local vs global training, covariates, probabilistic outputs, and practical model selection.

lstm
n-beats
tft
deepar

RNN and LSTM for Time-Series Forecasting

How recurrent networks model sequences: vanilla RNN, the vanishing-gradient problem, the LSTM gating mechanism, cell state, and the practical use of return_sequences in Keras.

lstm
rnn
vanishing-gradients

TCN: Causal and Dilated Convolutions for Time Series Forecasting

A practical explanation of Temporal Convolutional Networks for time-series forecasting: causal convolutions, dilations, receptive field, residual blocks, optional GLU gating, output heads, and when TCN is useful as a standalone or hybrid forecasting component.

tcn

Playing with a Hybrid Architecture for Forecasting

Playing with a hybrid forecasting architecture, assembled out of common neural building blocks: TCN front-end, LSTM recurrence, multi-head attention, TFT-style VSN feature routing with GRN, N-BEATS-style decomposition heads, seasonal and event branches.

lstm
deep-learning

Storage and Streaming

Data Storage Formats

Why CSV, JSON, and Parquet all coexist in real pipelines: file size, read speed, and tool compatibility rarely peak in the same format.

csv
json
parquet

Storage and Streaming

Data Streams: Kafka and Protobuf

Kafka as a distributed log for streaming data transport, and Protobuf as a compact binary serialization format — how each works, when to use them, and why they pair well together.

kafka
protobuf

Storage and Streaming

Object Storage

Covers the object storage model — buckets, keys, versioning, lifecycle policies — and Minio as a self-hosted S3-compatible implementation.

object-storage
s3
minio

Storage and Streaming

Relational Databases

A practitioner's overview of PostgreSQL: how it handles concurrency (MVCC), its type system, index types, and the Python tooling stack for working with it.

relational-databases
postgresql
sqlalchemy
alembic

Time Series Data: Univariate, Multivariate, Panel, and Exogenous Variables

A practical guide to the main types of time series data and the common confusion between multivariate, panel, and exogenous-variable forecasting.

forecasting
multivariate

Time Series Preprocessing: Interpolation, Bucketing, Time Zones, and Missing Intervals

A hands-on explanation of how raw time series data is cleaned, regularized, interpolated, bucketed, and prepared for reliable modeling or visualization.

preprocessing
interpolation
bucketing

Anomaly Detection: Statistical Baselines

An introduction to statistical baseline methods for time-series anomaly detection: Z-Score, IQR, moving averages, simple ensembles, and evaluation metrics.

anomaly-detection
z-score"
iqr
ensemble-methods