Reinforcement Learning from Human Feedback

Reinforcement learning from human feedback (RLHF) is a training method that uses human preferences to align a language model's behavior with what people actually want. After pretraining, the model produces multiple responses to the same prompt, human annotators rank them, and those rankings train a separate reward model that predicts human preference. The language model is then optimized against that reward signal, commonly with the PPO algorithm, so it favors responses people rate highly. The result is the shift from a raw text predictor into an assistant that follows instructions, declines harmful requests, and formats answers usefully. RLHF is a large part of why ChatGPT felt usable compared with the base models that preceded it, and every major assistant uses some version of it. Anthropic's Constitutional AI substitutes a written set of principles for much of the human labeling, having the model critique and revise its own outputs. Direct preference optimization (DPO) skips the separate reward model and trains directly on preference pairs, which is simpler and now widely used. The known weaknesses are reward hacking, where the model learns to produce answers that score well rather than answers that are correct, and sycophancy, where it agrees with the user because agreement was rated favorably.