Cross-Entropy, KL Divergence & Arithmetic Coding in Machine Learning
Explains why every LLM minimizes cross-entropy loss: measuring the exact bit-rate penalty D_KL(P || Q) incurred when compressing true distribution P using predicted distribution Q.
Where the loss function for training LLMs comes from.
Job opportunities aligned to this audience: https://3b1b.co/talent
Early views and other perks for supporters: https://3b1b.co/support
Home page: https://www.3blue1brown.com
Comments
Add a comment...Sign in