Matrix Methods in Data Analysis, Signal Processing, and Machine Learning
Prof. Gilbert StrangGilbert Strang bridges numerical linear algebra and modern machine learning: singular value decomposition, low-rank approximations, Eckart-Young theorem, randomized linear algebra, and stochastic gradient descent.
Syllabus
20Lecture 1: The Column Space of A Contains All Vectors Ax
MIT 18.065 Matrix Methods in Data Analysis, Signal Processing, and Machine Learning, Spring 2018.
Lecture 2: Multiplying and Factoring Matrices
MIT 18.065 Matrix Methods in Data Analysis, Signal Processing, and Machine Learning, Spring 2018.
3. Orthonormal Columns in Q Give Q'Q = I
MIT 18.065 Matrix Methods in Data Analysis, Signal Processing, and Machine Learning, Spring 2018.
4. Eigenvalues and Eigenvectors
MIT 18.065 Matrix Methods in Data Analysis, Signal Processing, and Machine Learning, Spring 2018.
5. Positive Definite and Semidefinite Matrices
MIT 18.065 Matrix Methods in Data Analysis, Signal Processing, and Machine Learning, Spring 2018.
6. Singular Value Decomposition (SVD)
MIT 18.065 Matrix Methods in Data Analysis, Signal Processing, and Machine Learning, Spring 2018.
7. Eckart-Young: The Closest Rank k Matrix to A
MIT 18.065 Matrix Methods in Data Analysis, Signal Processing, and Machine Learning, Spring 2018.
Lecture 8: Norms of Vectors and Matrices (18.065)
This lecture introduces various vector and matrix norms, including S-norms, the nuclear norm, and the Frobenius norm, to measure the 'size' of mathematical objects.
Lecture 9: Four Ways to Solve Least Squares Problems (18.065)
Professor Strang explores four fundamental methods for solving least squares problems, essential for data fitting and numerous practical applications.
Lecture 10: Survey of Difficulties with Ax = b (18.065)
This session addresses the computational challenges encountered when solving the matrix equation Ax=b, particularly with large-scale matrices.
Lecture 11: Minimizing "x" Subject to Ax = b (18.065)
This lecture revisits least squares problems, emphasizing the Gram-Schmidt process for constructing orthogonal vectors and its role in minimization under constraints.
Lecture 12: Computing Eigenvalues and Singular Values (18.065)
This lecture delves into numerical linear algebra, detailing computational methods for eigenvalues and singular values, including properties of the Hessenberg matrix.
Lecture 13: Randomized Matrix Multiplication (18.065)
This session introduces randomized linear algebra techniques, specifically randomized matrix multiplication, for efficient computation with very large matrices.
Lecture 14: Low Rank Changes in A and Its Inverse (18.065)
Professor Strang explores low rank matrices and the Sherman-Morrison-Woodbury formula for efficiently computing how small matrix changes impact their inverses.
Lecture 15: Matrices A(t) Depending on t, Derivative = dA/dt (18.065)
This lecture examines how eigenvalues and singular values change when matrices evolve over time, deriving essential formulas for their derivatives.
Lecture 16: Derivatives of Inverse and Singular Values (18.065)
This lecture covers derivatives of inverse and singular values, then pivots to discuss LASSO optimization, the nuclear norm, matrix completion, and compressed sensing.
Lecture 17: Rapidly Decreasing Singular Values (18.065)
Guest lecturer Alex Townsend explains why low rank matrices are prevalent in computational mathematics and their critical role in applications like image compression.
Lecture 18: Counting Parameters in SVD, LU, QR, Saddle Points (18.065)
Professor Strang analyzes the number of free parameters in SVD, LU, and QR decompositions, then introduces saddle points and Lagrange multipliers for constrained optimization.
Lecture 19: Saddle Points Continued, Maxmin Principle (18.065)
This lecture extends the discussion of saddle points, crucial for deep learning, and introduces the Maxmin Principle for optimizing outcomes in probability and statistics.
Lecture 20: Definitions and Inequalities (18.065)
This lecture focuses on probability concepts vital for data analysis, covering sample and expected means, variance, covariance matrices, and Chebyshev's and Markov's inequalities.