Home Glossary Activation Function

Activation Function

An activation function determines how the combined input to an artificial neuron is transformed before being passed to the next layer. Without nonlinear activations, stacking many neural network layers would still behave like a single linear transformation and could not represent complex patterns. ReLU, sigmoid, tanh, GELU, and softmax are common choices, each with different behavior, computational cost, and gradient properties. The selection affects training stability, sparsity, output range, and how easily gradients move through the network. Hidden layers and output layers often use different functions because tasks such as regression, binary classification, and multiclass prediction require different forms of output.

Related News