ML Optimization
Posts and notes about ml optimization.
Series & Posts
1
What is optimization and why ML needs it
2 Convex sets and convex functions
3 Optimality conditions: first order
4 Optimality conditions: second order
5 Line search method in optimization: Armijo, Wolfe, and backtracking
6 Least squares: the closed-form solution
7 Steepest descent (gradient descent)
8 Newton's method for optimization
9 Quasi-Newton methods: BFGS and L-BFGS
10 Conjugate gradient methods
11 Constrained optimization and Lagrangian duality
12 KKT conditions
13 Penalty and barrier methods
14 Interior point methods
15 The simplex method
16 Frank-Wolfe method
17 Optimization in dynamic programming and optimal control
18 Stochastic gradient descent and variants