Anthropic warns of 'risks of humans losing control over AI'

Anthropic has warned about the "risks of humans losing control over AI systems" and has called for a coordinated plan to slow down or temporarily halt AI development.

The maker of the popular Claude AI chatbot said in a statement that technical trends suggest AI systems will become much more capable in the coming years, with "huge implications".

"AI that can build itself would be a major development in the history of technology-one that could bring enormous good for the world in science, healthcare, and beyond," the tech giant said.

"But full recursive self-improvement also might increase the risks of humans losing control over AI systems. If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important."

The company said on Thursday that frontier AI developers should establish a coordinated, verifiable way to slow down or temporarily pause development if advanced systems start improving ​themselves quicker than society can manage the risks.

It would be "good ​for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology," ​the company said.

But it warned that unilateral or poorly coordinated slowdowns could run into problems ​if less cautious actors continue advancing, potentially impacting overall safety.

It noted that a meaningful pause ‌would ⁠require agreement among "multiple well-resourced labs" operating at the technological frontier, as well as rules on what conditions would spark or lift such a pause and who would oversee it.

A unilateral pause by a single company would be easier to put into practice, Anthropic ​said, but would ​have limited effect, ⁠primarily shifting leadership rather than promoting broader global deliberation.

The company said that as of May, more than ​80% of the code merged into its codebase was authored by Claude.

Its research arm, Anthropic Institute, plans to study and help construct systems that will be necessary to support a slowdown.

?Listen to This Is Why on your podcast app?

In the coming ​months, Anthropic ⁠plans to convene discussions examining key questions, involving policymakers, researchers, civil society groups and other AI firms.

These include how to manage AI-related risks, such as recursive ⁠self-improvement, ​and how to improve mechanisms for coordination.

Read more from Sky News:
Three charged over Henry Nowak murder protests
Water firm faces £44.7m 'enforcement package'

Last month, Anthropic, which was founded in 2021 by a group of former OpenAI employees, concluded a fundraising round that valued the company at $965bn (£717bn) and confidentially filed for a US initial ​public offering on Monday.

This makes it more valuable than OpenAI, which was valued at $852bn (£633bn) as of March.

(c) Sky News 2026: Anthropic warns of 'risks of humans losing control over AI'