Triage Score: A Counterfactual Risk Assessment Instrument (SIMONS-ECON)
39:56LLM-Guided Reinforcement Learning for Mastery Learning: Large-Scale Field Evidence (SIMONS-ECON)
35:06Non-parametric Causal Inference in Dynamic Thresholding Designs (SIMONS-ECON)
30:21Agreement and Alignment for Human-AI Collaborative Decision Making (SIMONS-ECON)
45:21Human Values in Clinical Algorithms, Old and New (SIMONS-ECON)
38:46When Prediction Becomes Policy: What Do Risk Models Actually Learn? (SIMONS-ECON)
33:31Evaluating Probabilistic Decision Making by LLMs: A Decision-Theoretic Framework and Application (SIMONS-ECON)
32:16Algorithmic UDAP (SIMONS-ECON)
32:13Operational Dosage: The Impact of Capacity Constraints on RCT Evaluation (SIMONS-ECON)
35:01The Role of Standardized Tests in Admissions: Predictions vs. Decisions (SIMONS-ECON)
33:37Individual Gaps in Prediction Generalization (SIMONS-ECON)
30:41Beyond Prediction Performance: How Modeling Decisions Shape Fairness Outcomes in Statistical Profiling (SIMONS-ECON)
27:56The Feedback Loop of Statistical Discrimination (SIMONS-ECON)
32:03Decision Theoretic Foundations for Human-AI Collaboration (SIMONS-ECON)
31:01Evaluation Design (SIMONS-ECON)
29:16Recommendation in Equilibrium for the NYC High School Match (SIMONS-ECON)
32:51Automating the US Welfare State: Historical Experiments in Technological Federalism (SIMONS-ECON)
35:41On Machine Learning for Prediction and Prioritization in the Allocation of Scarce Societal Resources (SIMONS-ECON)
35:26LLM-Guided Reinforcement Learning for Mastery Learning: Large-Scale Field Evidence (SIMONS-ECON)
This session presents a large-scale RCT on LLM-powered AI tutors, exploring personalized practice sequences using a POMDP framework for student mastery inference in Python certification.
Hamsa Bastani (Upenn)
Bridging Prediction and Intervention Problems in Social Systems
We describe a large-scale RCT in collaboration with the Taipei Department of Education for ~1000 high school students obtaining Python certification. All students had access to a LLM-powered AI tutor, and had to solve a required number of weekly practice problems. Half the students were randomized to a fixed practice sequence, while half were randomized to a personalized practice sequence using a POMDP framework to infer the student's mastery before moving on to more difficult problems. Existing POMDP formulations have limited visibility into student progress, hindering their ability to effectively provide personalized support. We show that student interactions on the practice platform provide a powerful view into their mastery, so we can significantly improve performance by using LLM-extracted features from platform interactions (e.g., meaningful code edits) as the POMDP observations. At the end of the semester, all students took a written exam with no AI assistance to receive certification. Students in the personalized arm performed 0.15 SD better, equivalent to several months of additional schooling.
Triage Score: A Counterfactual Risk Assessment Instrument (SIMONS-ECON)
Next · Lecture 3Non-parametric Causal Inference in Dynamic Thresholding Designs (SIMONS-ECON)