Masked Language Modeling (MLM)

Masked language modeling, abbreviated MLM, is a pretraining objective that hides selected tokens in a text sequence and asks a model to predict the originals. Because the model can use context before and after each hidden position, MLM produces bidirectional representations that are effective for understanding tasks such as classification, search, and entity recognition. BERT popularized this approach. The masking strategy matters: the rate, replacement method, vocabulary, and choice of spans influence what the model learns. MLM differs from causal language modeling, where every token is predicted only from earlier context. It is not naturally optimized for open-ended left-to-right generation without an added decoding design.