Opening Disincentivizing Hallucination in Language Models
Disincentivizing Hallucination in Language Models
- Length
- 34 min
- Level
- Research
- Recorded
- Simons Institute (UC Berkeley)2026
Chapter 1 of 7: Before the lecture
Language model hallucination, Kalai argues, follows from evaluation: benchmarks that score a blank answer like a wrong one never reward abstaining over guessing. Human experts, by contrast, decline questions they cannot answer. Kalai analyzes a consistency reduction that queries a model several times and outputs "I don't know" when its answers disagree. That reduction makes the model more humble but lowers raw accuracy, a trade-off on standard leaderboards that open rubrics remove by putting the grading rule in the prompt. Drawing on the Nature paper with Nachum, Vempala and Zhang, the seminar assumes familiarity with next-word prediction and benchmark metrics.

Your comment is kept while you sign up.Sign in
Kalai shows how accuracy-only scoring penalizes "I don't know" and proposes open rubrics that state the grading rule.
Language model hallucination, Kalai argues, follows from evaluation: benchmarks that score a blank answer like a wrong one never reward abstaining over guessing. Human experts, by contrast, decline questions they cannot answer. Kalai analyzes a consistency reduction that queries a model several times and outputs "I don't know" when its answers disagree. That reduction makes the model more humble but lowers raw accuracy, a trade-off on standard leaderboards that open rubrics remove by putting the grading rule in the prompt. Drawing on the Nature paper with Nachum, Vempala and Zhang, the seminar assumes familiarity with next-word prediction and benchmark metrics.
Evaluating large language models for accuracy incentivizes hallucination
Remarks on the Disproof of the Unit Distance Conjecture
First Proof
This lecture
Talk 20 of 21 of Theoretical Foundations of Modern Machine Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above.
Press a chosen node again to open it.