Home Glossary Softmax

Softmax

Softmax transforms a vector of arbitrary scores, often called logits, into nonnegative values that sum to one. In multiclass classification, those values can represent probabilities over classes; in language generation, they form a distribution over possible next tokens; in attention, they normalize compatibility scores into weights. Temperature can make the distribution sharper or flatter before sampling. Implementations subtract the largest logit or use fused operations to prevent numerical overflow. Softmax forces alternatives to compete, but the resulting numbers are not automatically well calibrated and can appear confident on unfamiliar inputs. Calibration tests and appropriate thresholds are still required when probabilities guide real decisions.

Related News