Overview

Instructor: Prof. P.S. Sastry, Department of Electronics and Communication Engineering, IISc Bangalore.

This course provides a fairly comprehensive view of the fundamentals of pattern classification and regression. Topics covered in the lectures include an overview of pattern classification and regression; Bayesian decision making and Bayes classifier; parametric estimation of densities; mixture densities and EM algorithm; Nonparametric Density Estimation; Linear Models for Classification and Regression; overview of statistical learning theory; empirical risk minimization and VC-dimension; artificial neural networks for classification and regression; support vector machines and kernel-based methods; feature selection, model assessment and cross-validation; boosting and classifier ensembles.

Syllabus

Mod-01 Lec-01 Introduction to Statistical Pattern Recognition.
Mod-01 Lec-02 Overview of Pattern Classifiers.
Mod-02 Lec-03 The Bayes Classifier for minimizing Risk.
Mod-02 Lec-04 Estimating Bayes Error; Minimax and Neymann-Pearson classifiers.
Mod-03 Lec-05 Implementing Bayes Classifier; Estimation of Class Conditional Densities.
Mod-03 Lec-06 Maximum Likelihood estimation of different densities.
Mod-03 Lec-07 Bayesian estimation of parameters of density functions, MAP estimates.
Mod-03 Lec-08 Bayesian Estimation examples; the exponential family of densities and ML estimates.
Mod-03 Lec-09 Sufficient Statistics; Recursive formulation of ML and Bayesian estimates.
Mod-04 Lec-10 Mixture Densities, ML estimation and EM algorithm.
Mod-04 & 05 Lec-11 Convergence of EM algorithm; overview of Nonparametric density estimation.
Mod-05 Lec-12 Nonparametric estimation, Parzen Windows, nearest neighbour methods.
Mod-06 Lec-13 Linear Discriminant Functions; Perceptron -- Learning Algorithm and convergence proof.
Mod-06 Lec-14 Linear Least Squares Regression; LMS algorithm.
Mod-06 Lec-15 AdaLinE and LMS algorithm; General nonliner least-squares regression.
Mod-06 Lec-16 Logistic Regression; Statistics of least squares method; Regularized Least Squares.
Mod-06 Lec-17 Fisher Linear Discriminant.
Mod-06 Lec-18 Linear Discriminant functions for multi-class case; multi-class logistic regression.
Mod-07 Lec-19 Learning and Generalization; PAC learning framework.
Mod-07 Lec-20 Overview of Statistical Learning Theory; Empirical Risk Minimization.
Mod-07 Lec-21 Consistency of Empirical Risk Minimization.
Mod-07 Lec-22 Consistency of Empirical Risk Minimization; VC-Dimension.
Mod-07 Lec-23 Complexity of Learning problems and VC-Dimension.
Mod-07 Lec-24 VC-Dimension Examples; VC-Dimension of hyperplanes.
Mod-08 Lec-25 Overview of Artificial Neural Networks.
Mod-08 Lec-26 Multilayer Feedforward Neural networks with Sigmoidal activation functions;.
Mod-08 Lec-27 Backpropagation Algorithm; Representational abilities of feedforward networks.
Mod-08 Lec-28 Feedforward networks for Classification and Regression; Backpropagation in Practice.
Mod-08 Lec-29 Radial Basis Function Networks; Gaussian RBF networks.
Mod-08 Lec-30 Learning Weights in RBF networks; K-means clustering algorithm.
Mod-09 Lec-31 Support Vector Machines -- Introduction, obtaining the optimal hyperplane.
Mod-09 Lec-32 SVM formulation with slack variables; nonlinear SVM classifiers.
Mod-09 Lec-33 Kernel Functions for nonlinear SVMs; Mercer and positive definite Kernels.
Mod-09 Lec-34 Support Vector Regression and ?-insensitive Loss function, examples of SVM learning.
Mod-09 Lec-35 Overview of SMO and other algorithms for SVM; ?-SVM and ?-SVR; SVM as a risk minimizer.
Mod-09 Lec-36 Positive Definite Kernels; RKHS; Representer Theorem.
Mod-10 Lec-37 Feature Selection and Dimensionality Reduction; Principal Component Analysis.
Mod-10 Lec-38 No Free Lunch Theorem; Model selection and model estimation; Bias-variance trade-off.
Mod-10 Lec-39 Assessing Learnt classifiers; Cross Validation;.
Mod-11 Lec-40 Bootstrap, Bagging and Boosting; Classifier Ensembles; AdaBoost.
Mod-11 Lec-41 Risk minimization view of AdaBoost.