ConceptEngineering
Sparsity
A structural property of a matrix where a large number of its entries are zero.
Know first
Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.
Matrices3Deep learning1Factored encoding1Linear algebra1Matrix-vector multiplication1Neural network weights1Neural scaling laws1Single-cell RNA sequencing1
Opens up
Concepts that name this one as a prerequisite.
Lecturers
5Each in their own terms, from the easiest level.
Onur Mutlu
The property of matrices in machine learning and other applications where a large number of values are zeros.Stephen Boyd
A matrix property where a large portion of the entries are zero, allowing algorithms to skip unnecessary computations.Charles Leiserson & Julian Shun
A property of data structures where most elements are zero, allowing for optimizations by ignoring these zero elements.Leslie Kaelbling
A property where most factors do not change most of the time.Ashish Vaswani, Lucas Beyer & William Fedus
A new dimension for scaling model architectures where each input is routed to sparsely used parameters with a fixed computation per example.
Lectures that assume it
14The first 12 are shown.
- Geometric Problems Continued
- Convex Optimization II: Truncated Newton Method and L1-Norm Methods
- Proof: All Norms Are Convex
- Equality Constrained Minimization
- Learning Representations: A Challenge for Learning Theory, COLT 2013
- Convex Optimization II: Sequential Convex Programming and Conjugate Gradient Methods
- Convex Optimization II: L1 Methods for Convex-Cardinality Problems
- Demystifying Mixtral of Experts
- Approximation and Fitting
- Equality Constrained Minimization Continued
- Efficient Methods and Hardware for Deep Learning
- AI in Science and Engineering: The Role of Academia in Understanding AI
Connections
30The lectures that define it or assume it, and the concepts next to it.
Concepts9
- Deep learning
- Factored encoding
- Linear algebra
- Matrix-vector multiplication
- $l_1$-norm methods
- Compressed sensing
Show 3 moreShow fewer
Lectures21
- Bentley Rules for Optimizing Work
- Arch4Health: 4th Workshop on Architecture for Health
- GPU Architectures II & Memory Overview and Technology (ETH Zürich, Digital Design and Computer Architecture, Lecture 20)
- The Role of Rationality in Modern Robotics
- Mixture of Experts (MoE) paradigm and the Switch Transformer
- Observability and State Estimation
Show 15 moreShow fewer
- Numerical Linear Algebra Background
- Geometric Problems Continued
- Convex Optimization II: Truncated Newton Method and L1-Norm Methods
- Proof: All Norms Are Convex
- Equality Constrained Minimization
- Learning Representations: A Challenge for Learning Theory, COLT 2013
- Convex Optimization II: Sequential Convex Programming and Conjugate Gradient Methods
- Convex Optimization II: L1 Methods for Convex-Cardinality Problems
- Demystifying Mixtral of Experts
- Approximation and Fitting
- Equality Constrained Minimization Continued
- Efficient Methods and Hardware for Deep Learning
- AI in Science and Engineering: The Role of Academia in Understanding AI
- Dual Averaging Method for Regularized Stochastic Learning and Online Optimization
- Efficient Algorithms for Reliable Machine Learning