Softmax

/ˈsɒftmæks/soft·maxnoun

Deep Learning Core

Definition

1.[in machine learning] a function that takes a vector of real numbers and maps it to a probability distribution, ensuring that all elements are positive and sum to one.

We applied the Softmax function to the final layer's logits to get class probabilities.

Formal statement

softmax(z)_i = e^z_i / sum(e^z_j)

z is the vector of raw scores (logits), and K is the number of classes.

Etymology

A compound of soft and max, naming a smooth relaxation of the maximum: where max returns a single winner, this returns a graded distribution over all candidates. Named by John Bridle in 1990, who introduced it as a differentiable stand-in for winner-take-all selection.

Synonyms

probability-distribution-functionsense 1 · Near

References

Goodfellow et al. (2016). Deep Learning, ch. 7.MIT Press.