Gradient Clipping

Gradient clipping stabilizes neural network training by restricting gradients before the optimizer updates model parameters. A common method rescales the full gradient vector when its norm exceeds a threshold; another clips individual values to a fixed range. The technique is particularly useful for recurrent networks, transformers, and other deep models that can suffer from exploding gradients. Clipping prevents a single step from causing extreme weight changes, numerical overflow, or loss divergence. It does not correct the underlying cause of every instability, and a threshold that is too low can slow learning. Teams should monitor gradient norms and combine clipping with suitable initialization, learning rates, precision settings, and normalization.