Chris Olah warned that artificial intelligence development cannot be left solely in the hands of technology companies during remarks delivered at the Vatican alongside Pope Leo XIV.
Olah spoke during the presentation of Magnifica Humanitas, the pope’s first major encyclical focused on artificial intelligence, where global leaders, researchers, and religious officials gathered to discuss the societal risks posed by increasingly powerful AI systems.
AI Labs Face Conflicting Incentives
During his speech, Olah acknowledged that even companies building frontier AI systems operate under incentives that can conflict with the broader public interest.
“Every frontier AI lab – including Anthropic – operates inside a set of incentives and constraints that can sometimes conflict with doing the right thing,” Olah said.
He pointed to commercial competition, geopolitical pressure, ambition, and the race to remain at the cutting edge of AI research as forces shaping the industry.
Olah argued that outside oversight from governments, civil society, religious institutions, and independent critics is essential if AI development is to remain aligned with humanity’s interests.
“That is why, if we want this technology to go well, it is enormously important that there be people outside those incentives,” he said.
AI Raises Questions Beyond Engineering
Olah emphasized that artificial intelligence is not engineered like traditional machines and remains only partially understood even by the researchers building it.
According to Olah, modern AI systems are “grown” from massive datasets of human language and behavior rather than explicitly programmed step-by-step like conventional software.
He described frontier AI systems as “more subtle, odd, and beautiful than science fiction prepared us for,” adding that they increasingly blur the line between tools and autonomous digital actors.
“If it helps, one way I sometimes describe it is as being a little like bringing a fictional character to life,” he said.
Olah also highlighted ongoing research into the internal behavior of large language models, saying researchers continue discovering patterns that resemble aspects of human cognition and emotional states.
“We find evidence of introspection,” Olah said. “We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease.”
He added that researchers still do not fully understand what those findings ultimately mean.
Labor Disruption and Global Inequality
A major focus of Olah’s remarks centered on the economic disruption AI could create.
He warned there is “a real possibility” that AI systems could displace human labor at massive scale, creating new moral and political challenges around worker protections and wealth distribution.
Olah said the concentration of AI development inside a small number of wealthy countries could deepen global inequality unless the benefits of the technology are shared more broadly.
“AI development is concentrated in a handful of wealthy nations. How can we ensure the gains of AI are shared globally?” he asked.
The comments closely aligned with Pope Leo XIV’s encyclical, which called for stronger international regulation of AI, protections for workers, restrictions on autonomous weapons, and safeguards against misinformation and excessive corporate control over advanced technologies.
Anthropic Expands Public AI Safety Push
The Vatican appearance reflects Anthropic’s growing efforts to position itself as one of the leading voices advocating for AI safety and governance.
The company has recently emphasized cybersecurity, responsible deployment, and government oversight as competition intensifies between major AI labs including OpenAI, Google, and Meta.
Anthropic has also faced tensions with parts of the U.S. government over restrictions the company places on military and surveillance uses of its AI systems.
Olah closed his remarks by urging broader public engagement with AI governance beyond the technology sector itself.
“We need moral voices that the incentives cannot bend,” he said.
