Triage Score: A Counterfactual Risk Assessment Instrument (SIMONS-ECON)
39:56LLM-Guided Reinforcement Learning for Mastery Learning: Large-Scale Field Evidence (SIMONS-ECON)
35:06Non-parametric Causal Inference in Dynamic Thresholding Designs (SIMONS-ECON)
30:21Agreement and Alignment for Human-AI Collaborative Decision Making (SIMONS-ECON)
45:21Human Values in Clinical Algorithms, Old and New (SIMONS-ECON)
38:46When Prediction Becomes Policy: What Do Risk Models Actually Learn? (SIMONS-ECON)
33:31Evaluating Probabilistic Decision Making by LLMs: A Decision-Theoretic Framework and Application (SIMONS-ECON)
32:16Algorithmic UDAP (SIMONS-ECON)
32:13Operational Dosage: The Impact of Capacity Constraints on RCT Evaluation (SIMONS-ECON)
35:01The Role of Standardized Tests in Admissions: Predictions vs. Decisions (SIMONS-ECON)
33:37Individual Gaps in Prediction Generalization (SIMONS-ECON)
30:41Beyond Prediction Performance: How Modeling Decisions Shape Fairness Outcomes in Statistical Profiling (SIMONS-ECON)
27:56The Feedback Loop of Statistical Discrimination (SIMONS-ECON)
32:03Decision Theoretic Foundations for Human-AI Collaboration (SIMONS-ECON)
31:01Evaluation Design (SIMONS-ECON)
29:16Recommendation in Equilibrium for the NYC High School Match (SIMONS-ECON)
32:51Automating the US Welfare State: Historical Experiments in Technological Federalism (SIMONS-ECON)
35:41On Machine Learning for Prediction and Prioritization in the Allocation of Scarce Societal Resources (SIMONS-ECON)
35:26Evaluating Probabilistic Decision Making by LLMs: A Decision-Theoretic Framework and Application (SIMONS-ECON)
This session proposes a revealed-preference approach to evaluate probabilistic decision-making in large language models, applying a decision-theoretic framework to high-stakes applications like medical triage.
Bryan Wilder (Carnegie Mellon University)
Bridging Prediction and Intervention Problems in Social Systems
Many recent efforts use language models to assist with high-stakes decisions, for example via triaging medical symptoms, or by helping applicants navigate social services. Such applications require models to engage in decision making under uncertainty, weighing the costs and benefits of different courses of action under uncertainty about the user's true state (e.g., a patient's true diagnosis). How can system-builders evaluate, diagnose, and improve such capabilities? We propose a revealed-preference approach which elicits decisions alongside probabilistic beliefs from a model and infers the utility function which would rationalize those decisions. We then apply this framework to medical diagnosis tasks and trace how failures are jointly attributable to the beliefs and utilities that explain the model's actions.
When Prediction Becomes Policy: What Do Risk Models Actually Learn? (SIMONS-ECON)
Next · Lecture 8Algorithmic UDAP (SIMONS-ECON)