Activation Function
/ˌæktɪˈveɪʃən ˈfʌŋkʃən/ac·ti·va·tion func·tionnoun
Definition
1.[in machine learning] a mathematical function that introduces non-linearity into the output of a layer, allowing the network to learn complex patterns.
The choice of activation function, such as ReLU, significantly impacts the training stability of the model.
Formal statement
f(x) = max(0, x)This is the formula for the Rectified Linear Unit (ReLU).
Etymology
From activate, Latin activus 'in action', by way of the biological metaphor of a neuron firing once its input crosses a threshold. The idea dates to the McCulloch-Pitts threshold unit of 1943; the smooth differentiable forms arrived with the backpropagation work of the 1980s.
Synonyms
- non-linearitysense 1 · Near
See also
References
- Goodfellow et al. (2016). Deep Learning, ch. 6.MIT Press.