Batch Norm vs Layer Norm: What Each One Normalizes Over

Both layers do the same arithmetic: subtract a mean, divide by a standard deviation, then rescale with learned parameters. They differ in which numbers the mean and variance are computed over.

batch norm   per feature, across the examples in the batch
layer norm   per example, across the features of that example

That single axis choice decides whether a layer's behaviour depends on the other examples that happened to be in the batch — which is where most of the practical differences come from.

Batch norm creates a dependency between examples. A given input's output depends on its batch-mates. That is a regularizer of sorts, and it is also a liability: statistics estimated from a small batch are noise, the layer behaves differently at training and inference time (running averages have to be kept and used), and distributed training needs a decision about whether to sync statistics across devices.

Measured, the batch-size cliff is exactly as advertised. The same batch-norm architecture — a smaller run than the comparison below: 6 layers, 2,000 examples, 6 epochs, two seeds — trained at batch 32 reached 85.5%; trained at batch 2, 41.2%. And not for want of updates: at a fixed number of epochs the batch-2 run takes sixteen times as many optimizer steps and still collapses.

The arithmetic says why. Normalize two numbers by their own mean and standard deviation and you get +1 and −1, whatever they were — all that survives is which of the pair was larger. At batch 2 the layer is not estimating statistics badly; it is discarding nearly everything the two activations said.

Layer norm removes the batch from the equation. Each example — in a transformer, each token — is normalized over its own features. The same statistics are used in training and at inference, so there are no running averages to keep and nothing to sync across devices, and one input's output is independent of what else is in the batch, at any batch size.

Worth being precise about what this is not. Batch norm runs perfectly well at inference batch 1: in eval mode it uses the running statistics it accumulated during training. Its costs sit everywhere else — the train/inference discrepancy, those running averages, the coupling between examples while training, the sync decision when training is distributed. Batch independence is one reason layer norm suits transformers. Another is what normalization does for activation and gradient scale through a deep residual stack, where even its placement relative to the residual add is a design decision.

What normalization buys is less certain than the folklore. The original story, that it removes "internal covariate shift", has not held up; the more durable claim is that it smooths optimization and keeps gradients better behaved, so larger learning rates become usable. On a well-initialized network the benefit can vanish: on a 12-layer ReLU MLP with He initialization, three seeds, no normalization at all reached 91.0% ± 0.8, layer norm 90.0% ± 0.3 and batch norm 89.1% ± 0.3.

Read those spreads before reading the ranking. No-norm against layer norm is a one-point gap inside the seed noise — on this task those two are not distinguishable. Batch norm trailing both is the part that holds.

Batch norm's own numbers in that run say something sharper than "it did not help": it reached the lowest training loss of the three (0.025, against 0.048 with no norm and 0.053 with layer norm) and the lowest test accuracy. In this setup, at this budget, it fit the training set best and generalized worst — the reminder that a normalization layer is an optimization tool, and optimizing better is not the same as learning better.

Choosing, in practice:

• Convolutional vision models with healthy batch sizes: batch norm is still the default, and it also averages over spatial positions, which makes small batches less dire than in an MLP.
• Anything with variable-length sequences, batch-1 inference, or tiny batches: layer norm, or RMSNorm, which scales by the root mean square without centring at all.
• Very small batches by necessity: group norm normalizes over channel groups within one example and keeps batch independence.
• Before adding any of them to a small, well-initialized network: measure without. Initialization may already have done the job.

Measured — gradient flow, training curves, the batch-2 collapse, and a pre-norm/post-norm probe that came out inconclusive, both placements leaving the gradient at layer 2 about 4.8× the gradient at layer 24 — in Training Neural Networks.