Opening Aligning Open Language Models
Aligning Open Language Models
- Length
- 1 h 16 min
- Level
- Graduate
- Recorded
- Stanford University2024
Chapter 1 of 11: Before the lecture
Traces open alignment from Alpaca's synthetic instructions and QLoRA to DPO, which skips the reward model.
Aligning open language models is the subject of session 14 of Stanford's CS25: Transformers United, given by Nathan Lambert of the Allen Institute for AI. Lambert starts from autoregressive loss, transformers and scaling laws, then argues that RLHF is necessary but not sufficient for alignment. The open response follows: Alpaca's instruction tuning on synthetic data, the OpenAssistant dataset, QLoRA's pairing of quantization with low-rank adaptation, and the Llama 2 safety debates. Direct preference optimization, which skips the reward model, stands against reinforcement learning from human feedback, while Chatbot Arena is weighed against AlpacaEval and MT-Bench. The talk assumes familiarity with transformers and supervised fine-tuning, and closes on data scarcity, synthetic data and personalized small models.
Aligning open language models is the subject of session 14 of Stanford's CS25: Transformers United, given by Nathan Lambert of the Allen Institute for AI. Lambert starts from autoregressive loss, transformers and scaling laws, then argues that RLHF is necessary but not sufficient for alignment. The open response follows: Alpaca's instruction tuning on synthetic data, the OpenAssistant dataset, QLoRA's pairing of quantization with low-rank adaptation, and the Llama 2 safety debates. Direct preference optimization, which skips the reward model, stands against reinforcement learning from human feedback, while Chatbot Arena is weighed against AlpacaEval and MT-Bench. The talk assumes familiarity with transformers and supervised fine-tuning, and closes on data scarcity, synthetic data and personalized small models.
A Mathematical Theory of Communication
Attention Is All You Need
Scaling Laws for Neural Language Models
This lecture
Talk 14 of 21 of Transformers United.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.
Your comment is kept while you sign up.Sign in
Release Strategies and the Social Impacts of Language Models
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?
Constitutional AI: Harmlessness from AI Feedback
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron et al., 2023
Direct Preference Optimization