Attention Mechanism

/əˈtɛnʃən ˈmɛkəˌnɪzəm/at·ten·tion mech·a·nismnoun

Deep Learning Core

Definition

1.[in deep learning] a layer that produces each output as a weighted blend of all available inputs, where the weights are computed from the similarity between a query and a set of keys.

The attention mechanism let the decoder consult any source token instead of a single fixed summary vector.

Formal statement

Attention(Q,K,V) = softmax(Q K^T / sqrt(d_k)) V

Division by sqrt(d_k) keeps the dot products in a range where softmax gradients do not vanish.

Etymology

Borrowed from cognitive psychology, where attention names the selective allocation of processing. Introduced to neural machine translation by Bahdanau et al. in 2014.

Synonyms

soft alignmentsense 1 · Near

References

Bahdanau, D., Cho, K. & Bengio, Y. (2015). Neural Machine Translation by Jointly Learning to Align and Translate.ICLR 2015.
Vaswani, A. et al. (2017). Attention Is All You Need.NeurIPS 2017.