Learning From Data

This inaugural lecture for Caltech's CS 156 introduces the core components of the learning problem, distinguishing between supervised, unsupervised, and reinforcement learning paradigms. It establishes foundational concepts for algorithmic inference.

This lecture explores the fundamental question of generalization in machine learning: how to infer properties of an entire space from a limited sample. It introduces the relationship between in-sample and out-of-sample error, laying the groundwork for statistical learning theory.

This lecture introduces the foundational concepts of the linear model, covering both linear classification and linear regression. It also explores methods for extending the applicability of linear models through nonlinear transformations of input features.

This lecture, part of Caltech's CS 156, formally introduces principled choices for error measures in machine learning contexts. It examines the implications of noisy target functions on learning algorithms and model performance.

This lecture formally distinguishes training and testing phases in machine learning, exploring the mathematical conditions for model generalization. It addresses the core challenge of ensuring learned models perform effectively on unseen data.

This lecture formalizes the core concept of generalization in machine learning, exploring how models can learn from finite samples. It introduces foundational theoretical results, including the VC dimension and its role in establishing generalization bounds.

This lecture introduces the Vapnik-Chervonenkis (VC) dimension as a fundamental measure of the capacity of a hypothesis space, explaining its relationship to model complexity and generalization error in statistical learning theory.

Professor Abu-Mostafa elucidates the fundamental bias-variance tradeoff in machine learning, decomposing prediction error into irreducible noise, bias from model assumptions, and variance from finite training data. The lecture details the impact on generalization performance.

This lecture delves into advanced linear models, covering logistic regression, the principles of maximum likelihood estimation, and the iterative optimization process of gradient descent.

This session introduces biologically inspired neural networks, detailing their architecture, the role of hidden layers, and the efficient backpropagation algorithm for training.

This lecture explores the phenomenon of overfitting, differentiating between deterministic and stochastic noise as factors contributing to models that fit data too well.

This session examines regularization techniques to mitigate overfitting, discussing hard and soft constraints, augmented error formulations, and the concept of weight decay.

This lecture covers validation methods for robust model selection, addressing data contamination and introducing cross-validation techniques for out-of-sample evaluation.

This session introduces Support Vector Machines (SVMs), detailing their mechanism for achieving complex decision boundaries while maintaining a simple underlying model structure.

This lecture explores kernel methods, extending SVM capabilities to infinite-dimensional spaces using the kernel trick and handling non-separable data through soft margins.

This session introduces Radial Basis Functions (RBFs) as a key learning model, highlighting their connections to various machine learning techniques and architectures.

This lecture outlines critical pitfalls in machine learning practice, discussing Occam's razor for model simplicity, the impact of sampling bias, and the dangers of data snooping.

The concluding lecture provides an overview of the machine learning landscape, briefly touching upon Bayesian learning paradigms and aggregation methods for improved model performance.