ConceptAI & ML
Distillation
A technique where a student model is trained to mimic the predictions or output distributions of a teacher model.
Know first
Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.
Neural networks2Connection weights1Loss function1Model compression1Pre-trained models1Teacher-student models1
Opens up
Concepts that name this one as a prerequisite.
Lecturers
3Each in their own terms, from the easiest level.
Geoffrey Hinton
A process to transfer knowledge from a teacher to a student by showing the student correct responses to various inputs.Ashish Vaswani, Lucas Beyer & William Fedus
A technique to transfer knowledge from a large sparse model to a smaller dense model to retain quality improvements while compressing parameters.
Lectures that assume it
3Connections
13The lectures that define it or assume it, and the concepts next to it.
Concepts7
Show 1 moreShow fewer
Lectures6
- Transfer Learning: Models
- Digital vs. Biological Intelligence (Vienna Gödel Lecture)
- Mixture of Experts (MoE) paradigm and the Switch Transformer
- Let's Build the GPT Tokenizer: Byte Pair Encoding & Unicode Internals
- Deep Dive into Large Language Models: Pretraining to RLHF
- AI Seminar: Actionable Interpretability for Language Model Reasoning