Skip to main content

EE606A: Optimization For Big Data

Course Description

This course covers complexity results for first order and related optimization algorithms. Such algorithms are widely applied to numerous problems in machine learning, signal processing, communications, etc. and have witnessed a surge in popularity since the advent of Big Data around 2005. Wherever possible, we will attempt to follow a generic template for the proofs, thereby unifying a large number of results in the literature.

Course Content

  1. Introduction: convex functions, black box model
  2. Basics: dual norm, smoothness and strong convexity
  3. Gradient descent for smooth functions
  4. Projected gradient descent
  5. Subgradient descent
  6. Frank-Wolfe or conditional gradient descent
  7. Mirror descent
  8. Proximal gradient descent
  9. Accelerated gradient descent for smooth functions
  10. Dual descent, saddle point method
  11. Augmented Lagrangian method of multipliers
  12. Stochastic gradient descent, mini batch SGD
  13. Variance-reduced SGD
  14. Other topics as time permits