Backpropagation

/ˌbækprɒpəˈɡeɪʃən/back·prop·a·ga·tionnoun

Neural Networks CoreFoundational

Definition

1.[in neural networks] an application of the chain rule that computes the sensitivity of a loss to every parameter in a layered network by passing partial derivatives from the output back toward the input.

One backpropagation pass yields every parameter's derivative in roughly the cost of one forward pass.

Formal statement

dL/dW(l) = delta(l) * a(l-1)^T, delta(l) = W(l+1)^T * delta(l+1) .* sigma'(z(l))

delta is the error signal at a layer; the circled dot is elementwise multiplication.

Etymology

A 1980s contraction of 'backward propagation of errors'. Popularised by Rumelhart, Hinton and Williams in 1986, though the underlying reverse-mode differentiation was described by Linnainmaa in 1970.

Synonyms

reverse-mode automatic differentiationsense 1 · Near

Antonyms

forward passsense 1 · Relational

References

Rumelhart, D., Hinton, G. & Williams, R. (1986). Learning representations by back-propagating errors.Nature, 323, 533–536.
Linnainmaa, S. (1970). The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors.Master's thesis, University of Helsinki.