Gradient Descent
Definition
1.[in optimization] an iterative procedure that reduces a differentiable objective by repeatedly stepping in the direction opposite to its slope at the current point.
Training halted once gradient descent stopped lowering the validation objective.
2.[in machine learning] the family of training algorithms that update model parameters from the slope of a loss surface, including the stochastic and mini-batch variants used in practice.
Nearly every deep network in production was fitted by some form of gradient descent.
Formal statement
theta(t+1) = theta(t) - eta * grad J(theta(t))eta is the step size, or learning rate; grad J is the slope of the objective at the current parameters.
Etymology
From gradient, Latin gradiens 'stepping', plus descent, Old French descente 'a going down'. Named for the geometric picture of walking downhill on a surface.
Synonyms
- steepest descentsense 1 · Near
Antonyms
- gradient ascentsense 1 · Relational
See also
References
- Cauchy, A. (1847). Méthode générale pour la résolution des systèmes d'équations simultanées.Comptes Rendus de l'Académie des Sciences, 25, 536–538.
- Robbins, H. & Monro, S. (1951). A Stochastic Approximation Method.The Annals of Mathematical Statistics, 22(3), 400–407.