Deep Learning

This lecture provides an overview of deep neural networks, detailing their fundamental concepts and essential building blocks. It sets the foundation for understanding deep learning models.

This session explains neural network training via stochastic gradient descent (SGD), backpropagation, and automatic differentiation. It highlights their role in differentiable programming.

This lecture investigates the approximation capabilities of deep neural networks, covering universal approximation theorems and Barron's theorem. It also discusses the impact of network depth on expressivity.

This lecture focuses on convolutional neural networks (CNNs), presenting them as an optimal architectural choice for data structured on a grid, such as images.

This session covers graph neural networks (GNNs), demonstrating their connections to MLPs and CNNs, and detailing message passing algorithms. Theoretical limitations of GNN expressive power are also discussed.

This lecture introduces fundamental generalization theory, discussing overparameterization, double descent phenomena, and limitations of VC dimension. It also examines the role of inductive biases in deep learning.

This video analyzes neural computation spectrally, explores feature learning and hyperparameter transfer. It presents critical scaling rules for adapting hyperparameters across varying network widths and depths.

This lecture introduces transformer architectures, explaining the core concepts of tokens, attention mechanisms, and positional codes. It relates transformers to MLPs, GNNs, and CNNs.

This lecture provides practical insights and anecdotal advice for effectively training deep neural networks to achieve desired performance. It covers pragmatic considerations for model deployment and debugging.

This session examines memory-focused architectures for sequence modeling, including Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTMs). It highlights their mechanisms for retaining information over time.

This lecture introduces reconstruction-based representation learning, covering autoencoders, clustering, vector quantization (VQ), and self-supervised methods. It draws connections between network representations and neural processing.

This session focuses on similarity-based representation learning, including metric learning, contrastive learning, and the InfoNCE loss. It details alignment, uniformity principles, and handling hard negatives.

This lecture delves into the theoretical underpinnings of representation learning, emphasizing architectural inductive biases and the intriguing relationship between neural networks and Gaussian processes.

This lecture introduces foundational generative models, including density and energy models, sampling techniques, GANs, autoregressive models, and diffusion models. It provides a broad overview of their mechanisms.

This session explores the synergy between representation learning and generative modeling, focusing on Variational Autoencoders (VAEs) and the effective utilization of latent variables in model design.

This lecture covers conditional generative models such as cGANs, cVAEs, and conditional diffusion models. It showcases diverse applications like image-to-image and text-to-image generation.

This video investigates out-of-distribution (OOD) generalization challenges, including adversarial robustness and managing distribution shifts. It provides strategies for improving model performance in novel environments.

This lecture covers advanced transfer learning techniques, specifically fine-tuning, linear probes, and knowledge distillation. It also introduces the emerging concept of foundation models.

This session explores data-centric aspects of transfer learning, encompassing generative models for data augmentation, domain adaptation techniques, and effective prompting strategies.

This lecture examines scaling laws in neural architectures, detailing power laws, their limitations, and theoretical foundations. It introduces the concept of critical batch size in deep learning.