Causal Language Modeling (CLM) - Page 33

Causal language modeling, abbreviated CLM, is a training objective in which a model predicts the next token from tokens that came before it. A causal attention mask prevents information from later positions from leaking into each prediction. Repeating this task across large text collections teaches an autoregressive model statistical patterns of language, knowledge associations, and ways to continue a sequence. At inference time, the model generates text token by token using the same left-to-right dependency. CLM differs from masked language modeling, which hides selected tokens inside a sequence and uses surrounding context on both sides. The objective supports open-ended generation but does not guarantee factual or intentional understanding.