Mathematics of Deep Learning

MA5001 (Autumn 2025) Course Details

Study Material

Chapter 1: Introduction to Neurons (Included in Quiz 1 and Midsem Exam)
Introduction to Neurons
Motivating Example
Water Source-Sink Control Problem
Electric Circuit
Linear Regression
Artificial Neurons
Mathematical Formulation
Comparison with Biological Neurons
Supervised Learning: An Overview
Chapter 2: Perceptrons (Included in Quiz 1 and Midsem Exam)
Perceptrons
Neuron Model
Boolean Functions
Illustrative Datasets
Separability and Algorithm
Linear Separability
Training Algorithm
Perceptron Convergence Theorem
Beyond Linear Separability
Feature Space Mapping
Chapter 3: Support Vector Machine (Included in Quiz 2 and Midsem Exam)
Support Vector Machine
Linearly Separable Dataset
Hard and Soft Margins
Loss-function form
Subgradient Descent Algorithm
Dual Optimazation Problem
Nonlinearly Separable Datasets
Kernel Method: Implicit Feature Mapping
Kernel Trick
Chapter 4: Multilayer Perceptrons (Included in Midsem Exam (Up to Batch Propagation), Quiz 3(FULL), Endsem Exam(FULL))
Multilayer Perceptrons
Fully Connected Feedforward Networks
Definition and Network Dimensions
Shallow Networks
Batch Forward Propagation
Batch Propagation
Activation Functions
Step Functions
Heaviside function
Bipolar Function
Sigmoid Functions: Logistic Regression
Hyperbolic Tangent Function
Bumped-type Functions
Hockey-stick Functions
Leaky Rectified Linear Unit
Sigmoid Linear Units
Softplus Function
Multi-class Classification
Softmax Function
Chapter 5: Training Neural Network (Included in Quiz 3 (Up to Adam), Quiz 4 (From Regularization and Generalization), Endsem Exam)
Training Neural Networks
Cost Functions
Mean Square Error
Cross-Entropy
Kullback-Leibler divergence
Backpropagation Algorithm
Chain Rule and Gradient Computation
Gradient for a Neuron
Shallow Network
Deep Network
Weight Updates Using Gradient Descent
Batch Gradient Descent Method
Stochastic Gradient Descent (SGD)
Mini-batch Gradient Descent
Momentum Method
Adaptive Step Size Methods
AdaGrad.
Root Mean Square Propagation (RMSProp)
Adaptive Moments (Adam).
Regularization and Generalization
Underfitting and Overfitting
Parameter-based Regularization
Process-based Regularization
Early Stopping
Dropout
Generalization
Bias-Variance Tradeoff
Chapter 6: Expressivity Over Function Spaces (Included in Quiz 4, Endsem Exam)
Expressivity over Function Spaces
Function Approximations
Continuous Function Approximations
Feature Mapping Perspective
Connection to kernel machines
Learning dynamics and the Neural Tangent Kernel
Physics Informed Neural Networks for ODEs
Trial Solution
Residual and Training Loss
Training Algorithm
Research Perspectives
Chapter 7: Special Architectures (Included in Quiz 4, Endsem Exam)
Special Architectures
Convolution Neural Network
Networks of One-Dimensional Signals
One-Dimensional Convolutional Layer
Two-Dimensional Convolutional Layer
Convolution Operator
2D Convolution Layer
Recurrent Neural Networks
Transformers

References

General Instructions