Perplexity - Page 8

Perplexity is a language-model evaluation metric derived from average cross-entropy loss. Intuitively, it reflects how many plausible choices the model appears to consider at each step; lower perplexity means the model assigns higher probability to the observed sequence. It is useful for comparing similar models on the same tokenized dataset, but it is not directly comparable across different vocabularies, tokenizers, or evaluation procedures. A lower score does not necessarily mean responses are more factual, helpful, safe, or preferred by people. Data contamination can also produce deceptively strong results. Perplexity is best treated as a training and predictive-fit measure alongside task-specific and human evaluations.