RLHF (Reinforcement Learning from Human Feedback)

Reinforcement Learning from Human Feedback is the second-stage training process that turned raw pretrained language models into the helpful, polite, useful assistants you talk to today. Pretraining on internet text produces a model that knows things but doesn't necessarily say them helpfully. RLHF shapes the behavior on top.

The standard pipeline:

1. Pretrain — train a base model on a huge text corpus to predict next tokens.
2. Supervised fine-tune (SFT) — fine-tune on high-quality human-written examples of "good" responses.
3. Reward model training — humans rank model outputs. Train a separate "reward model" to predict those rankings.
4. RL fine-tune — use the reward model as a signal in reinforcement learning to push the LLM toward higher-ranked outputs.

What RLHF actually delivers:

• Helpfulness — answers questions instead of repeating the question.
• Harmlessness — refuses dangerous requests, avoids slurs, etc.
• Style — concise, structured responses instead of meandering completions.
• Format compliance — responds in JSON/Markdown when asked.

The successors to RLHF:

• DPO (Direct Preference Optimization) — simpler math, comparable results, now the default in many labs.
• Constitutional AI (Anthropic) — uses a written constitution + model-generated feedback to scale beyond what human raters can produce.
• RLAIF — RL from AI feedback, where another model plays the rater role.

The whole class of techniques is what people mean by "alignment." It's also the reason the same base model can produce wildly different products depending on who post-trained it — Claude, GPT, Gemini, Llama-Instruct all started from comparable pretraining and diverged through their alignment recipes.