Instructor: Dr. Mingrui Liu
Time/Location
Office Hours:
Contact Information:
Graduate Teaching Assistant (GTA):
Recommended Textbooks:
Optimization plays a central role in modern machine learning, serving as the engine that enables models to learn from data. This course studies optimization as the foundation of machine learning, beginning with the basics of convex and stochastic optimization and then moving to the training of large-scale neural networks and foundation models. We develop both classical theory and recent insights into training dynamics, scaling behavior, and optimization in overparameterized models.
The course is intended for students who want to pursue research or advanced practice in machine learning, particularly those interested in optimization theory, large-scale model training, and the design of efficient and reliable learning algorithms.
Prerequisites: Students should have a solid background in linear algebra, probability, and calculus. Prior exposure to machine learning and basic optimization (e.g., gradient descent) is expected. Familiarity with Python and PyTorch is recommended for assignments and projects.
Foundations of Optimization
Aug 24 (week 1): Introduction, Course Overview: Optimization for Modern Machine Learning
Aug 31 (week 2): Basic Concepts in Machine Learning and Optimization
Sep 7: Labor Day, No Class
Sep 14 (week 3): Gradient Descent, Nesterov’s Accelerated Gradient Descent
Sep 21 (week 4): Lower Bounds and Complexity of First-Order Methods
Sep 28 (week 5): Proximal Methods and Composite Optimization
Stochastic and First-Order Methods
Oct 5 (week 6): Mirror Descent, Stochastic Gradient Descent (SGD)
Oct 12: Fall Break, No Class
Oct 19 (week 7): Beyond SGD: Adaptive Gradient Methods, Variance Reduction
Nonconvex and Minimax Optimization
Oct 26 (week 8): Large-Scale Training Dynamics: Instability and Large Stepsizes
Nov 2 (week 9): Nonconvex Optimization: Stationary Points and Escaping Saddle Points
Nov 9 (week 10): Minimax Optimization and Applications: Generative Adversarial Networks and Robust Learning
Optimization for Large-Scale Neural Networks
Nov 16 (week 11): Deep Network Optimization: Transformers and Non-Euclidean Geometry-Aware Methods
Nov 23 (week 12): Tensor Programs and Scaling Laws for Neural Networks
Nov 30 (week 13): Final Project Presentation
Dec 7 (week 14): Final Project Presentation
Please see the Office for Academic Integrity (https://oai.gmu.edu/) for a full description of the code and the honor committee process, and the Honor Code Policies of the Department of Computer Science (https://cs.gmu.edu/resources/honor-code/) regarding the course project. GMU is an Honor Code university. The principle of academic integrity is taken seriously and violations are treated gravely. If you rely on someone else's work in an aspect of the course project, you should give full credit in the proper, accepted form. Another aspect of academic integrity is the free play of ideas. Vigorous discussion and debate are encouraged in this course, with the firm expectation that all aspects of the class will be conducted with civility and respect for differing ideas, perspectives, and traditions. When in doubt (of any kind) please ask for guidance and clarification.
By default, assessments are [closed book, closed notes, and there are no authorized devices]. Policies regarding authorized material and devices will be specified per assessment. The possession of an unauthorized device or unauthorized material during any assessment will result in an automatic 0 on that assessment, even if the device or material is not used. For example, if you are asked to turn off your phone and put it away in your backpack, and then you are seen with a phone in your pocket during a quiz or exam (even if it is off), then that quiz or exam will be an automatic 0. The same rule also applies for any limited access devices or material. For example, if you are allowed some form of notes and a calculator during a quiz or exam but there are restrictions given (such as the quantity of notes, or capabilities of the calculator), and you are seen with something outside the parameters specified, then that quiz or exam will be an automatic 0. This policy is non-negotiable and it is your responsibility to confirm before the assessment if something does or does not adhere to the assessment policies.