/
lectern.tv
Play15s30s1x0:00/--:--
DoneWatched
Expand
lectern.tv
10:11/1:02:50
1:42
5:10
2:53
1:55
8:40
12:25
3:20
2:20
2:10
5:36
3:46
0:48
A core lecture from MIT's Introduction to Deep Learning, dissecting the architecture and functional principles of Recurrent Neural Networks, Transformer models, and the critical attention mechanism.
Explores sequence modeling using neural networks, beginning with the foundations of recurrent neural networks (RNNs), their mathematical updates, training via backpropagation through time, and applications like music generation. It then addresses RNN limitations such as vanishing gradients and the encoding bottleneck, before introducing self-attention and transformer architectures.
Referenced inHacker News · 2 discussions
Lecture Guide
- 1Sequential data requires architectures that account for temporal order and past memory, unlike static feed-forward networks.
- 2Recurrent Neural Networks (RNNs) use a shared weight recurrence relation and hidden states to process inputs of variable lengths.
- 3Backpropagation through time (BPTT) enables training on sequential data, but suffers from exploding and vanishing gradient problems.
- 4Gated recurrent units like LSTMs use selective gates to mitigate the vanishing gradient problem and track long-term dependencies.
Proofs, Examples & Demos