Opening Behind the Scenes of LLM Pre-training: StarCoder Use Case
Behind the Scenes of LLM Pre-training: StarCoder Use Case
- Length
- 1 h 2 min
- Level
- Undergraduate
- Recorded
- Stanford University2024
Chapter 1 of 8: Before the lecture
Ben Allal follows StarCoder's data through MinHash deduplication, PII removal and benchmark decontamination.
Pre-training a large language model from scratch depends as much on data curation as on architecture, and Ben Allal takes the StarCoder code models as her case. She opens with scaling laws, contrasting Kaplan et al. with Chinchilla, and explains why inference cost favours smaller models trained on far more tokens. Data sources follow: Common Crawl, FineWeb, The Stack and synthetic corpora such as Cosmopedia. The filtering pipeline covers MinHash near-deduplication, PII removal, opt-out mechanisms and benchmark decontamination. HumanEval evaluation and a survey of open code models close the session, which assumes basic familiarity with transformers and tokenization.
Pre-training a large language model from scratch depends as much on data curation as on architecture, and Ben Allal takes the StarCoder code models as her case. She opens with scaling laws, contrasting Kaplan et al. with Chinchilla, and explains why inference cost favours smaller models trained on far more tokens. Data sources follow: Common Crawl, FineWeb, The Stack and synthetic corpora such as Cosmopedia. The filtering pipeline covers MinHash near-deduplication, PII removal, opt-out mechanisms and benchmark decontamination. HumanEval evaluation and a survey of open code models close the session, which assumes basic familiarity with transformers and tokenization.
Google "We Have No Moat, And Neither Does OpenAI"
Kaplan et al. (2020)
Chinchilla (2022)

This lecture
Talk 18 of 21 of Transformers United.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.
Your comment is kept while you sign up.Sign in
Scaling Data-Constrained Language Models
DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
The Stack
Textbooks Are All You Need
Yi: Open Foundation Models
Personal Copilot: Train Your Own Coding Assistant
DeepSeek Paper
Lima Paper