ConceptAI & ML
Masked language modeling
A pre-training task where a fraction of input sequence tokens are masked, and the model is trained to predict the original masked characters from their positions.
Know first
Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.
Opens up
Concepts that name this one as a prerequisite.
Lecturers
3Each in their own terms, from the easiest level.
Ashish Vaswani, Lucas Beyer & William Fedus
A pre-training task where specific words in a sequence are replaced with placeholders, and the model is trained to predict the original words based on the surrounding context.Ming Ding
Masked language modeling is a pretraining task good at understanding text by predicting masked tokens.
Lectures that assume it
2Connections
10The lectures that define it or assume it, and the concepts next to it.