ConceptAI & ML
Pretraining
Pretraining teaches a neural network to mimic its training data from the internet by predicting the next word given a block of text.
Know first
Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.
Opens up
Concepts that name this one as a prerequisite.
Lecturers
3Each in their own terms, from the easiest level.
Andrej Karpathy
The first stage of training a large language model where it learns from a large corpus of internet text.Sara Beery
Training a neural network on prior tasks to give it access to prior knowledge.
Lectures that assume it
5Connections
17The lectures that define it or assume it, and the concepts next to it.
Concepts9
- Large language models
- Neural networks
- Exposition
- Language model
- Neural network training
- Canary forgetting