Home Glossary Pretraining

Pretraining - Page 11

Pretraining teaches an AI model broad patterns from a large dataset before it is adapted to a particular application. A language model may predict missing or next tokens across diverse text, while a vision model may learn from labeled images, self-supervised objectives, or image-text pairs. The resulting model provides reusable representations and capabilities that reduce the data and computing needed for downstream tasks. Pretraining is expensive and shapes nearly everything that follows, including strengths, blind spots, bias, language coverage, and memorized information. Developers may adapt the base through prompting, retrieval, instruction tuning, or fine-tuning, but those later steps do not automatically remove limitations introduced by the original data and objective.

Andrew Tulloch Leaves $12B AI Startup to Join Meta After Turning Down $1.5B Offer
By • 3 mins read
AI & Machine Learning, Immersive Reality (AR, VR, MR, and XR), News, Startups & Investment

Andrew Tulloch Leaves $12B AI Startup to Join Meta After Turning Down $1.5B Offer

By • 3 mins read

Andrew Tulloch, co-founder of the $12 billion AI startup Thinking Machines Lab, has joined Meta after previously rejecting what reports described as a $1.5 billion offer — a figure Meta has since called ‘inaccurate and ridiculous.’