PPO vs DPO: Two Ways to Pay for the Same Preference

Both procedures answer the same request: make the model prefer the answers people prefer, without drifting far from the model you started with. The objective they share is

maximize  E[ reward(x, y) ]  −  β · KL( policy ‖ reference )

The reward pulls; the KL term is a leash to the frozen starting model. PPO-based RLHF and DPO are both tied to that objective, but they are not the same kind of thing: one is a way of optimizing a policy against a reward you have trained, the other is a loss you can write down directly over preference pairs. The link between them is a derivation, and it carries assumptions.

PPO evaluates it online, with machinery. Human comparisons train a reward model that assigns a scalar to a single prompt–completion pair, fitted so that preferred answers score above rejected ones. Then a loop: the current policy samples completions, the reward model scores them, and the policy is updated toward higher reward with the KL penalty holding it near the reference. To make the update stable there is usually a fourth network, a value model, predicting the reward the policy expects so the update is driven by the advantage rather than raw reward. Four model roles in play — policy, reference, reward, value — though they need not be four separate networks resident at once; the value model is often a head on a shared trunk. And generation sits inside the training loop.

DPO evaluates it offline, on the pairs themselves. The key algebraic move: for that KL-regularized objective the optimal policy has a closed form in terms of the reference and the reward, which can be inverted to express the reward in terms of the policy. Substitute, and the reward model disappears from the loss — you train directly on (prompt, preferred, rejected) triples with a simple classification-style objective. Two models, no sampling loop, no value network.

"No reward model" is half the story. DPO does not abolish the reward; it makes it implicit, read off the policy and the reference. That is why the two families behave like relatives rather than rivals. But the derivation assumes a particular model of how preferences arise from rewards, and exact optimization of it — so the two do not have to land on the same policy, and in practice they often do not.

Where the practical difference actually lives:

• Data. PPO samples its own training distribution as the policy moves, which is the mechanism exploration would need — though sampling alone does not guarantee it finds anything useful. Standard DPO trains on the pairs you already have and collects no new feedback as the policy changes; online variants exist precisely to put that back.
• Cost and stability. PPO's loop is operationally heavy: generation inside training, several model roles to keep fed, a reward model that can be gamed. DPO is a fine-tuning run.
• Different failure modes. PPO can find answers its reward model scores highly and humans do not; the KL leash exists to bound how far it may wander looking for them. DPO has no explicit reward model to exploit, which removes that particular failure and not the general one: it can still overfit the preference data, inherit whatever artefacts the annotation has, and raise the probability of answers nobody wanted on prompts the pairs never covered.
• The β knob has a shared origin, not the same behaviour. It is the same coefficient in the underlying trade-off between preference strength and distance from the reference. But DPO never enforces that KL explicitly: its β sits inside the loss, scaling the gradients, so tuning it moves optimization dynamics as well as the trade-off, and the policy's actual drift from the reference need not follow it monotonically.

How to choose in practice: for offline alignment against a fixed set of comparisons, DPO-family methods are the strong and simple starting point. If you need the model to go beyond the collected pairs, or you have a reward signal worth optimizing against while the policy moves, the loop earns its cost.

The whole pipeline — reward modeling, PPO, the offline-preference family and what they have in common — is in How LLMs Learn Human Preferences. Related shorts: RLHF, RLAIF, SFT.