Reinforcement Learning from Human Feedback (RLHF)
The post-training technique that turns a raw next-token predictor into a helpful assistant by optimizing against human preference judgments.
What is RLHF?
RLHF is why chatbots chat. A pretrained LLM only continues text; RLHF post-training makes it helpful: humans rank pairs of model responses, a reward model learns those preferences, and reinforcement learning optimizes the LLM to score highly. The result answers questions, follows instructions, and declines harmful requests — behaviors nowhere in the pretraining objective.
The family tree
- RLHF (PPO) — the original InstructGPT/ChatGPT recipe: reward model + policy optimization.
- RLAIF / Constitutional AI — AI feedback graded against written principles replaces most human labeling; scales far cheaper.
- DPO (Direct Preference Optimization) — skips the reward model, optimizing directly on preference pairs; simpler and popular for open-weight models.
- RLVR (verifiable rewards) — RL on checkable outcomes (tests pass, math verifies) rather than taste; the engine behind reasoning models.
Why practitioners should care
Post-training explains model personality — why models hedge, over-apologize, or refuse oddly: those are learned preferences, not knowledge limits. It also explains sycophancy (agreeing with users because agreement rated well) and historically contributed to hallucination — confident answers out-rated honest “I don’t know”s. When two models with similar benchmarks feel different to use, you’re feeling their post-training.
What people get wrong
- “RLHF makes models truthful.” It makes them preferred — aligned to what raters rewarded, which correlates imperfectly with truth.
- Confusing it with fine-tuning you can buy. API fine-tuning products are supervised tuning; preference-based post-training at quality requires data and infrastructure few teams have — though DPO on open models has narrowed the gap.
- Fighting post-training with prompts. If a model’s trained-in behavior (refusal style, verbosity) conflicts with your product, switching models usually beats prompt-wrestling.
Historical figures and technical concepts for informational purposes only. Not technical, professional, legal, or financial advice. Sources: Official Documentation.