Dropout

Dropout is a neural network regularization technique that randomly sets a proportion of activations to zero during each training step. Because different units disappear on different passes, the network cannot depend too heavily on a small set of connections and is encouraged to learn representations that remain useful in varied combinations. At inference time, all units are normally active and the implementation scales activations appropriately. The dropout rate is a hyperparameter: too little may provide no benefit, while too much can make learning difficult. Modern architectures may use dropout alongside weight decay, data augmentation, normalization, or early stopping, and its value must be verified on validation data.