Pretraining - Page 15

Pretraining teaches an AI model broad patterns from a large dataset before it is adapted to a particular application. A language model may predict missing or next tokens across diverse text, while a vision model may learn from labeled images, self-supervised objectives, or image-text pairs. The resulting model provides reusable representations and capabilities that reduce the data and computing needed for downstream tasks. Pretraining is expensive and shapes nearly everything that follows, including strengths, blind spots, bias, language coverage, and memorized information. Developers may adapt the base through prompting, retrieval, instruction tuning, or fine-tuning, but those later steps do not automatically remove limitations introduced by the original data and objective.