Statistical learning: Algorithms and theory

Class Times: Tuesdays and Thursdays 11:40-12:55
Location: First week: Old Chem 025 after that 2240b CIEMAS
Instructors: Sayan Mukherjee
Office Hours: By appointment
Email Contact : sayan at stat.duke
Similar class: Statistical learning theory

Course description

The problem of supervised learning will be developed in the framework of statistical learning theory. Two classes of machine learning algorithms that have been used successfully in a variety of applications will be studied in depth: regularization algorithms and voting algorithms. Support vector machines (SVMs) are an example of a popular regularization algorithm and AdaBoost is an example of a popular voting algorithm. The course will
1) introduce these two classes of algorithms
2) illustrate practical uses of the algorithms
via problems in computational biology and computer graphics
3) state theoretical results on the generalization and consistency of these algorithms.

Prerequisites

Familiarity with probability, functional analysis, and linear algebra will be very helpful. We try to keep the mathematical prerequisites to a minimum, but we will introduce complicated material at a fast pace.

Grading

Three problem sets for 50% of the grade. A final project for 50% of the grade.

Syllabus

Follow the link for each class to find a detailed description, suggested readings, and class slides. Some of the later classes may be subject to reordering or rescheduling.



Date Title
Class 01 Thur 13 Jan Course at a glance
Class 02 Tues 18 Jan Learning problem in perspective (Second lecture in notes).
Class 03 Thur 20 Jan Regularization and Reproducing Kernel Hilbert Spaces
Class 04 Tues 25 Jan Kernel ridge-regression
Class 05 Thur 27 Jan Support Vector Machines for classification
Class 06 Tues 1 Feb Spline models and regularization networks
Class 07 Thur 3 Feb Voting algorithms and weak learners
Class 08 Tues 8 Feb Probably approximately correct framework and the boosting hypothesis
Class 09 Thur 10 Feb Adaptive boosting (Adaboost)
Class 10 Tues 15 Feb Applications in computational biology
Class 11 Thur 17 Feb Applications in computer graphics
Class 12 Tues 22 Feb Multiclass classification and text classification
Class 13 Thur 24 Feb Generalization and consistency
Class 14 Tues 1 Mar One-dimensional concentration inequalities
Class 15 Thur 3 Mar Vapnik-Chervonenkis classes, shattering dimensions, and covering numbers
Class 16 Tues 8 Mar Uniform Glivenko-Cantelli classes
Class 17 Thurs 10 Mar Discussion about final projects
SPRING BREAK
Class 18 Tues 22 Mar Symmetrization and Rademacher averages
Class 19 Thur 24 Mar Kolmogorov chaining and Dudley's entropy integral
Class 20 Tues 29 Mar Stability and generalization
Class 21 Thurs 31 Mar Stability of Tikhonov regularization
Class 22 Tues 5 Apr Necessary and sufficient conditions for the consistency or error minimization
Class 23 Thurs 7 Apr Bounds for error minimization
Class 24 Tues 12 Apr Bounds for boosting algorithms I
Class 25 Thur 14 Apr Bounds for boosting algorithms II
Class 26 Tues 19 Apr Project presentations
Class 27 Thur 21 Apr Project presentations

Math Camp 1 TBD Analysis and basic probability theory
Math Camp 2 TBD More analysis and probability theory

Reading List

There is no textbook for this course.

I am preparing a series of lectures that will grow in length and hopefully clarity as the course continues. These lecture notes will contain in much greater depth all topics I will cover. I will update the notes such that contents relevant for each lecture will appear at least a day before the lecture. Note, I have not yet had a chance to add references. The books and papers listed below are useful general reference reading, especially from the theoretical viewpoint.