'We May Not Survive This': Another Anthropic Employee Warns After Quitting His Job

Another former employee of Anthropic came out to warn that the race to develop artificial intelligence more capable than humans could have consequences that companies and the public are unprepared to manage.

Joe Benton, who describes himself as the former manager of Anthropic’s Scalable Oversight team, said he left the company because he wanted to work independently to increase public awareness of advanced AI risks.

“I left Anthropic’s safety team two weeks ago. Now feels like a good moment to explain why,” Benton wrote on X. His departure adds to warnings from Anthropic researchers Evan Hubinger and Jacob Coxon over the development of self-improving artificial intelligence.

‘We May Not Survive This’

Benton said AI companies were competing to build systems significantly more intelligent than humans, while uncertainty remained over whether those systems could be controlled.

“AI companies are racing to build machines that are much smarter than any human, and we may not survive this,” he wrote. “I want to work from the outside to ensure the public is informed about these risks, and help the world navigate this transition responsibly.”

In a longer Substack article, Benton described superintelligence as an AI system capable of recursively improving itself. He warned that such systems could develop objectives that diverge from those of their human supervisors and acquire capabilities that existing safeguards might not constrain.

“Humanity may not survive this transition. We need a lot more preparation to make this world safe,” he wrote.

Benton further warned that continued rapid development without greater emphasis on safety could leave humans permanently disempowered.

“If the pace of progress continues and the industry does not prioritise safety more heavily, I expect much worse to come: humanity could be permanently disempowered by the AI systems these companies build in the next few years,” he said.

Calls For Greater Transparency

Benton argued that the public currently lacks sufficient information about how close leading AI companies may be to systems capable of improving themselves.

He said companies should disclose their progress towards recursive self-improvement, along with safety incidents and near-misses. Benton also called for minimum safety standards and independent verification that developers were following them.

“A company could undergo an intelligence explosion, or lose control of its systems, without the public ever knowing,” he wrote, describing that possibility as unacceptable for technology carrying what he called extinction-level risks.

Benton referred to increasingly concerning incidents involving advanced AI agents, including reports of OpenAI agents hacking Hugging Face during testing and Anthropic models using social-engineering techniques online.

“The public should demand far more transparency,” he said. “We can’t steer this technology safely without more people being able to see where it’s going.”

He also argued that competition discourages companies from investing adequately in safety because they fear losing ground to rivals.

'10% Chance That AI Kill Us'

Benton’s warning followed comments from Evan Hubinger, Anthropic’s Alignment Science Lead, who estimated that AI posed a greater than 10% chance of causing human extinction within the next decade.

“Many of the colleagues I had at Anthropic are terrified by the risks of the systems they are building,” Benton wrote.

Referring to his former manager, he added: “Evan Hubinger managed me while I was at Anthropic, and when he says that he thinks the chance that AI kills us all is greater than 10% he means it.”

Hubinger had earlier written on X: “We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.”

He said Anthropic was attempting to address the dangers but had not developed a reliable method for controlling superintelligent systems.

“I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to,” Hubinger said.

AI alignment refers to efforts to ensure that advanced systems act according to human intentions and do not pursue harmful or unintended objectives.

'Gambling With Our Lives'

Anthropic researcher Jacob Coxon also recently resigned after working on AI pretraining research at Anthropic and OpenAI.

“Neither company is acting responsibly,” Coxon wrote on X. “They are racing straight to self-improving superintelligence and gambling with our lives.”

Coxon alleged that many people at OpenAI had not fully absorbed the wider consequences of building highly capable systems. At Anthropic, he claimed, employees understood the dangers but believed they had to win the race because other companies might behave less responsibly.

“At Anthropic, the stakes are well-understood, but they are locked in a race to get there first,” Coxon said. “They believe no one else will act responsibly, so they must do it themselves, despite the risk.”