Gradient Descent

Gradient descent is an optimization method that improves a model by repeatedly moving its parameters in a direction that reduces error. The gradient indicates how the loss would change if each parameter changed slightly, and the learning rate controls the size of the update. Full-batch gradient descent uses the entire dataset for each step, while stochastic and mini-batch variants estimate the direction from fewer examples. Updates that are too large may overshoot a useful solution, whereas tiny updates can make training slow or stall. Modern optimizers extend the basic idea with momentum, adaptive step sizes, and other techniques designed for complex neural networks and uneven optimization landscapes.