Opening Transformers that Transform Well Enough to Support Near-Shallow Architectures
Transformers that Transform Well Enough to Support Near-Shallow Architectures
- Length
- 1 h 20 min
- Level
- Graduate
- Recorded
- Stanford University2024
Chapter 1 of 10: Before the lecture
Initializes language model layers from log-co-occurrence matrices, with no gradient descent at all.
Precision language models derive non-random parameter initializations for specialized self-attention architectures to enable efficient training with limited resources. The discussion covers the bit-cipher algorithm for low-dimensional vector assignment, log-co-occurrence matrix layer initialization without gradient descent, and alternative context models like radial and document formulations. These techniques reduce computational costs and allow small language models to be trained on microprocessors for on-edge hardware control applications.
Precision language models derive non-random parameter initializations for specialized self-attention architectures to enable efficient training with limited resources. The discussion covers the bit-cipher algorithm for low-dimensional vector assignment, log-co-occurrence matrix layer initialization without gradient descent, and alternative context models like radial and document formulations. These techniques reduce computational costs and allow small language models to be trained on microprocessors for on-edge hardware control applications.
arxiv.07510
arxiv.03635
arxiv.11012
arxiv.07498

This lecture
Talk 16 of 21 of Transformers United.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.
Your comment is kept while you sign up.Sign in
arxiv.3781