Opening Demystifying Mixtral of Experts
Demystifying Mixtral of Experts
- Length
- 1 h 5 min
- Level
- Graduate
- Recorded
- University of Cambridgeat Stanford University2024
Chapter 1 of 8: Before the lecture
Jiang counts Mixtral's 256 experts and 46.7 billion parameters, then tests domain specialization with routing maps.
Mixtral 8x7B is a sparse mixture-of-experts language model whose router sends each token to two of eight feedforward experts at every layer. Jiang, of Mistral AI and Cambridge, starts from the dense Mistral 7B, with grouped-query and sliding-window attention. He then traces the line from Shazeer's sparsely-gated layer to Switch Transformers and explains why 47 billion total parameters need only 13 billion active ones. Expert layers lift knowledge-heavy benchmarks more than reasoning, and routing visualizations across shallow, middle and deep layers test claims of specialization. Open problems include inference load balancing and extreme sparsification, and the CS25 session assumes working knowledge of transformers.
Mixtral 8x7B is a sparse mixture-of-experts language model whose router sends each token to two of eight feedforward experts at every layer. Jiang, of Mistral AI and Cambridge, starts from the dense Mistral 7B, with grouped-query and sliding-window attention. He then traces the line from Shazeer's sparsely-gated layer to Switch Transformers and explains why 47 billion total parameters need only 13 billion active ones. Expert layers lift knowledge-heavy benchmarks more than reasoning, and routing visualizations across shallow, middle and deep layers test claims of specialization. Open problems include inference load balancing and extreme sparsification, and the CS25 session assumes working knowledge of transformers.
Mixtral of Experts
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

This lecture
Talk 15 of 21 of Transformers United.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.
Your comment is kept while you sign up.Sign in
Switch Transformers
Mixture-of-Depths: Dynamically allocating compute in transformer-based language models