Regularization

/rɛɡjʊləˌraɪˈzeɪʃən/re·gu·la·ri·za·tionnoun

Statistical Learning CoreFoundational

Definition

1.[in machine learning] a technique used to prevent overfitting by adding a penalty term to the loss function, which discourages overly complex models.

We applied L2 regularization to the model to prevent overfitting on the training data.

Formal statement

Loss(w) + lambda ||w - w0||2^2

The penalty term controls the complexity of the model weights (w).

Etymology

From Latin regula 'rule', via regularis 'according to rule'. The mathematical sense — constraining an ill-posed problem until it becomes well-posed — is due to Andrey Tikhonov, whose work gives the method its other name, Tikhonov regularization.

Synonyms

weight-decaysense 1 · Absolute

References

Tikhonov, A. (1963). Regularization in ill-posed problems.SIAM Journal on Numerical Analysis.