Learning Rate

/ˈlɜːrnɪŋ reɪt/learn·ing ratenoun

Optimization CoreFoundational

Definition

1.[in optimization] a hyperparameter that controls the step size taken at each iteration of an optimization algorithm.

If the learning rate is too high, the optimization process may overshoot the minimum loss.

Formal statement

w_t+1 = w_t - (learning rate / gradient) * grad L(w_t)

The learning rate determines the magnitude of the step taken in the direction opposite to the gradient.

Etymology

From learning, Old English leornian 'to acquire knowledge', plus rate, Medieval Latin rata (pars) 'a fixed amount'. Compounded in the cybernetics literature of the late 1950s, where it named the constant scaling the correction applied at each training step.

Synonyms

step sizesense 1 · Absolute

References

Goodfellow et al. (2016). Deep Learning, ch. 8.MIT Press.