Home Glossary Cross-Entropy

Cross-Entropy

Cross-entropy is a loss commonly used to train classification and language models. It penalizes a model when the probability assigned to the correct outcome is low, with confident wrong predictions receiving especially large penalties. In binary tasks it compares a predicted probability with a yes-or-no label; in multiclass and token prediction it operates over a distribution created by softmax. Minimizing cross-entropy encourages probability mass to move toward correct targets, but a lower training loss does not guarantee calibrated probabilities or good performance on new data. Numerical stability, label quality, class weighting, masking, and reduction across examples all affect how the loss behaves in practice.

Related News