Opening The Challenge of Valid Evaluations
The Challenge of Valid Evaluations
- Length
- 33 min
- Level
- Research
- Recorded
- Simons Institute (UC Berkeley)2026
Chapter 1 of 6: Before the lecture
Coston bounds performance metrics when past decisions, proxy labels and hospital covariate shift taint evaluation data.
Evaluating decision-making algorithms in high-stakes settings is harder than standard machine learning evaluation, because historical data reflect past interventions. Coston, speaking in the Simons Institute series on Prediction, Incentives & Causal Intervention, frames the problem through construct validity and external validity. She shows how selective labels and proxy outcomes with one-sided measurement error make performance and fairness claims fragile. Combining causal assumptions with measurement error models yields partial identification bounds on performance metrics. A second part estimates target risk under covariate shift between hospitals, using doubly robust estimators that pair propensity and outcome models. The talk assumes potential outcomes and doubly robust estimation, and closes with questions on transfer learning and covariate shift in clinical data.
Evaluating decision-making algorithms in high-stakes settings is harder than standard machine learning evaluation, because historical data reflect past interventions. Coston, speaking in the Simons Institute series on Prediction, Incentives & Causal Intervention, frames the problem through construct validity and external validity. She shows how selective labels and proxy outcomes with one-sided measurement error make performance and fairness claims fragile. Combining causal assumptions with measurement error models yields partial identification bounds on performance metrics. A second part estimates target risk under covariate shift between hospitals, using doubly robust estimators that pair propensity and outcome models. The talk assumes potential outcomes and doubly robust estimation, and closes with questions on transfer learning and covariate shift in clinical data.
Predicting Pneumonia Risk
Ground(less) Truth: A Causal Framework for Proxy Labels in Human Algorithm Decision-Making
A Validity Perspective on Evaluating the Justified Use of Data-driven Decision-making Algorithms
Validity and reliability in social science research
Counterfactual Risk Assessments, Evaluation, and Fairness
Characterizing Fairness Over the Set of Good Models Under Selective Labels

This lecture
Talk 20 of 22 of Prediction, Incentives & Causal Intervention.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above.
Press a chosen node again to open it.
Your comment is kept while you sign up.Sign in
Fairness under Unobserved Feature Variability via Measurement Error Models
The Fragility of Fairness: A Measurement-Theoretic Critique of Algorithmic Fairness
Dissecting racial bias in an algorithm used to manage the health of populations
Clinical decision instruments developed using a cohort that's overly white and male
Algorithmic Risk Assessment in the Presence of Alternatives
Auditing for Discrimination in Algorithms
Analysis of Covariate Shift by Importance Weighting
Direct Importance Estimation with Model Selection and Its Application to Covariate Shift Adaptation
Correcting Sample Selection Bias by Unlabeled Data
AISTATS 2020 paper by Fogliato, Chouldechova, and G'sell