In-Batch Negatives: Where a Retrieval Model Gets Its Contrast

Retrieval embeddings are trained on a deceptively simple task. The model sees pairs — a question and its answer, a title and its body, a query and the document someone clicked — and is asked to put the two close together. Closeness alone is trivially satisfiable: map everything to the same point. The training needs an opposing force, and the cheapest source of one is already in the batch.

In a batch of 256 pairs, each query has one positive — its own partner — and 255 negatives it gets for free: everybody else's positives. The negatives cost nothing: no mining, no annotation. The positives still have to come from somewhere, which is what the pairing — clicked documents, title and body, question and accepted answer — is for.

q, d    L2-normalized, so a dot product is a cosine
scores  = queries · documentsᵀ        (256 × 256)
loss    = cross-entropy( scores / τ, correct index = the diagonal )

This is why batch size is a quality parameter, not just a throughput one. More rows mean more negatives per query and finer distinctions to make, which is why serious embedding training reaches for large batches, gradient caching and cross-device negative sharing. Measured here — a head trained with exactly this loss, every run scored on the same held-out pool of 2,400 documents:

batch      4      8     16     32     64    128    256
recall@1  .151   .195   .257   .284   .321   .343   .348

Two things in that row. At batch 4 the model comes out worse than the untrained head (.174): three negatives per query is not enough contrast to learn from. And the gains flatten — 64 → 128 buys 2.2 points, 128 → 256 buys 0.5.

The negatives you get for free are mostly easy. A random other document in the batch is usually obviously wrong, and once the model separates the obvious cases those negatives contribute almost nothing to the gradient. That is the motivation for hard negatives: documents retrieved by an earlier model that look right and are not. They are also where the method turns delicate — a "hard negative" that is actually a correct answer nobody labelled teaches the model to push apart two things that belong together.

And the temperature is not a formality. Dividing the similarities by a small τ sharpens the softmax, so the loss concentrates on the closest negatives instead of averaging over all of them. It also has a geometric effect worth knowing: many pretrained transformer encoders are anisotropic — their embeddings occupy a narrow cone, so almost everything looks similar to everything else. Contrastive training with a small temperature does not only pull positives together; it pushes the space toward uniformity, so that being close becomes informative in the first place.

The cone is real and large. On the same sentences, a plain pretrained encoder puts the mean cosine between unrelated pairs at +0.79, with the 5th-to-95th range running 0.69 to 0.87 — nothing is far from anything. A contrastively trained encoder of the same size: +0.06, from −0.11 to +0.28.

But turning τ down does not simply turn the cone off. Held-out recall peaks at τ = 0.1 and falls away on both sides, and at τ = 0.01 the space partly re-collapses — positives at cosine 0.92, negatives at 0.79. Too small a temperature spends the whole loss on the few closest negatives, which are by construction the pairs most likely to be relevant and unlabelled, and stops doing the global spreading that made closeness informative. Nor is the positive-negative gap the thing to maximize: it grows all the way to τ = 1, where retrieval is at its worst.

What this means when using such a model. Its notion of "similar" was shaped by what counted as a negative during training, which is why an embedding model tuned on question–answer pairs can behave oddly on title–title similarity: nothing in its training ever contrasted those. And it is why the retrieval metric has to match the training setup: with L2-normalized embeddings cosine and dot product are the same number, but a model trained with an unnormalized dot product can carry information in vector length, and normalizing it at query time silently changes the ranking.

The full picture — what the geometry can and cannot carry, why cosine is the right metric, and where the cone comes from — is in Embeddings: How Geometry Pretends to Be Meaning. Related short: cosine, dot product, Euclidean.