Weight Decay

Weight decay is a regularization technique that discourages a model’s learned weights from becoming unnecessarily large. During optimization, weights are slightly reduced at each update or penalized through the training objective, depending on the algorithm. This can limit overly complex solutions, reduce sensitivity to noise, and improve performance on unseen data. The decay strength is a hyperparameter: too little may have no useful effect, while too much can prevent the model from learning important patterns. Weight decay interacts with the optimizer, learning rate, architecture, normalization, and training duration, so it should be selected through validation rather than copied unchanged from another model.