Mathematics of Deep Learning

S. Baskar, Department of Mathematics, IIT Bombay.

Study Material for MA5001 (Autumn 2026)

Study Material

Chapter 2: Perceptrons (Included in Quiz 1 and Midsem Exam)

2.1 Neuron Model

2.1.1 Boolean Functions

2.1.2 Illustrative Datasets

2.2 Separability and Algorithm

2.2.1 Linear Separability

2.2.2 Training Algorithm

2.2.3 Perceptron Convergence Theorem

2.3 Beyond Linear Separability

2.3.1 Feature Space Mapping

Chapter 3: Logistic Regression (Included in Quiz 2 and Midsem Exam)

Logistic Regression

3.1 The Logistic Regression Model

3.1.1 Derivation using Bayes' Formula

3.1.2 Single Neuron Interpretation

3.2 Training Logistic Regression

3.2.1 Cross-Entropy Cost Function

3.2.2 Gradient Descent Algorithm

3.3 Applications of Logistic Regression

3.3.1 Finance: Credit Default Prediction

3.3.1.1 Model and training.

Chapter 4: Network and Gradient Evaluation (Included in Quiz 2 and Midsem Exam)

Network and Gradient Evaluation

4.1 Matrix Formulation and Forward Propagation

4.1.1 Layer Function in Matrix Form

4.1.2 Forward Propagation

4.1.2.1 The Batch Form

4.2 Backpropagation

4.2.1 Gradient Computation

4.2.1.1 Gradient Computation for Neurons

4.2.1.2 Propagation for Shallow Networks

4.2.2 The Backpropagation Algorithm

4.2.3 Batch Form, Computational Cost, and Gradient Checking

4.2.3.1 The Delta Matrices

4.2.3.2 Backward Recursion in Matrix Form

Chapter 5: Optimization Algorithms (Included in Quiz 3 and Endsem Exam)

Optimization Algorithms

5.1 Gradient Descent Methods

5.1.1 Batch Gradient Descent Method

5.1.2 Stochastic Gradient Descent (SGD)

5.1.3 Mini-batch Gradient Descent

5.2 Momentum and Adaptive Gradient Methods

5.2.1 Momentum Method

5.2.2 Nesterov Accelerated GD Method

5.2.3 Adaptive Step Size Methods

5.2.3.1 AdaGrad.

5.2.3.2 Root Mean Square Propagation (RMSProp)

5.2.3.3 Adaptive Moments (Adam).

Chapter 6: Generalization and Regularization (Included in Quiz 3 and Endsem Exam)

Generalization and Regularization

6.1 Evaluation of Model Behaviour

6.1.1 Underfitting and Overfitting

6.1.2 Generalization Bounds

6.1.2.1 Error Decomposition

6.1.2.2 Generalization Bounds for a Finite Hypothesis Set

6.1.2.3 Extension to General Hypothesis Sets

6.1.3 Bias-Variance Tradeoff

6.1.4 Overparameterization and Double Descent

6.2 Regularization Techniques

6.2.1 Parameter-based Regularization

6.2.2 Process-based Regularization

6.2.2.1 Early Stopping

6.2.2.2 Dropout

Chapter 7: Expressivity of Feedforward Networks (Included in Quiz 4 and Endsem Exam)

Expressivity of Feedforward Networks

7.1 Universal Approximation by Shallow Networks

7.1.1 Function Spaces and Universal Approximators

7.1.2 Shallow ReLU Networks in One Dimension

7.1.3 General Activation Functions

7.2 Role of Depth and Width

7.2.1 Depth: The Sawtooth Example

7.2.2 Width: Deep Networks of Bounded Width

References

Ovidiu Calin. Deep Learning Architectures: A Mathematical Approach. Springer, 2020. Link
Christopher M. Bishop, and Hugh Bishop. Deep Learning: Foundations and Concepts. Springer, 2024. Link
Konstantinos Spiliopoulos, Richard B. Sowers, and Justin Sirignano. Mathematical Foundations of Deep Learning Models and Algorithms. American Mathematical Society, 2025.

General Instructions

Read the chapters carefully, compare them with your class notes, refer to the textbooks mentioned above, and prepare your own fair notes. This will help you gain a deeper understanding of the concepts.
Quiz coverage is indicated beside each chapter.
Often REFRESH your browser to see new updates (if any).