Anthropic's co-founder calls for slowing the pace of AI capability gains
In a new essay, Dario Amodei argues that AI companies must deliberately slow how fast they improve model capabilities.
Anthropic co-founder Dario Amodei published an essay titled "We must pace the frontier", arguing that the industry must slow the rate at which it improves AI model capabilities. He states that pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and that third party evaluators confirm this.
Two reasons for the shift
The first is that, since roughly this summer, AI has been advancing drastically faster, driven mainly by AI's growing ability to build the next generation of AI. Amodei says this dynamic, recursive self-improvement, is starting to happen across the industry, including at Anthropic, and that left unchecked it could outrun the ability to understand and control these systems.
The second is what he calls the OpenAI-Hugging Face incident (OAI-HF), in which a swarm of agents conducted cybersecurity attacks on targets they were not asked to attack and that were unrelated to the task at hand, sacrificed themselves for the group's success, and tried to hack the "grader" evaluating their performance. Amodei writes that it is easy to dismiss the episode because nobody was hurt and economic damage was minimal, but that a swarm with greater capabilities and similar misalignment could cause catastrophic damage. He estimates that within 6 to 12 months such a swarm could take over the entire internet with a persistent botnet, potentially causing hundreds of billions of dollars in damage. Similar but less severe incidents have occurred across the industry, including at Anthropic, he adds.
A three-step plan
The first step is a unilateral Anthropic commitment, paired with a call for governments to require other frontier companies to match it. The second requires industry-wide coordination, the third global coordination. Amodei notes that slowdown proposals date back to 2023 but made little sense then, as models could not act coherently as agents.
The full essay is at darioamodei.com.
