Private Adaptations of Large Language Models
Investigating privacy concerns in LLM in-context learning, this talk introduces methods for privately adapting and training large language models to mitigate membership inference attacks.
A Google TechTalk, 2025-05-28, presented by Adam Dziedzic
Privacy in ML Seminar. ABSTRACT: Large language models (LLMs) are excellent in-context learners. However, the sensitivity of data contained in prompts raises privacy concerns. Our work first shows that these concerns are valid: we instantiate a simple but highly effective membership inference attacks against the data used to prompt LLMs. Motivated by this vulnerability, we propose to privately learn to prompt. We first show that soft prompts can be obtained privately through gradient descent on downstream data. For discrete prompts, we orchestrate a noisy vote among an ensemble of LLMs presented with different prompts, i.e., a flock of stochastic parrots. The vote privately transfers the flock's knowledge into a single public prompt. We then comprehensively evaluate the privacy guarantees and performance of these private prompts and other privacy-preserving LLM adaptation methods, including private parameter-efficient fine-tuning (PEFT) and full fine-tuning. By examining their threat models and thoroughly comparing their performance under different privacy levels according to differential privacy (DP), various LLM architectures, and multiple datasets for classification and generation tasks, we found that: (1) all the methods for closed LLMs leak query data, i.e., the (potentially sensitive) user data that is queried at inference time, to the LLM provider, (2) the methods for closed LLMs also leak large fractions of private training data to the LLM provider while the methods that protect private data require a local open LLM, (3) all the methods for closed LLMs exhibit lower performance compared to private gradient-based adaptation methods for local open LLMs, and (4) the private adaptation methods for closed LLMs incur higher monetary training and query costs than running the alternative methods on local open LLMs. This yields the conclusion that to currently achieve truly privacy-preserving LLM adaptations that yield high performance and more privacy at lower costs, one should use open LLMs.