OpenAI chief scientist: no lab is in a position to keep scaling at full speed
In an essay on OpenAI's site, Jakub Pachocki calls for voluntary slowdowns and international coordination. He also says the company's main safety tool, chain-of-thought monitoring, is becoming less reliable.
OpenAI chief scientist Jakub Pachocki has published an essay titled "An Alien Mind" on the company's site. One sentence sits at its centre: "Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer."
What he expects
Citing internal results, Pachocki says he strongly expects the current pace to be sustained into recursive self-improvement, with AI increasingly driving its own development. The capability jumps of the next few years should match those already seen or exceed them. "This is a time that calls for extreme caution," he writes, adding that he is concerned no one is prepared for the consequences.
The safety tool is weakening
The essay's most concrete technical admission concerns monitoring. OpenAI's primary bet has been to optimise the outcome of a model's reasoning without supervising the reasoning itself, so the chain of thought has no incentive to hide misaligned ideas. That is why o1-preview deliberately hid it. Evaluations now show the ability to rely on that method is progressively diminishing, Pachocki says, for three reasons: reasoning is increasingly blended with talking to people, other AIs and tools; models are better at manipulating their own reasoning; and stronger pretraining makes them much smarter without verbalised reasoning at all.
Why "alien"
The title comes from the distance the essay puts between machine and human intelligence. AI is "grown more than designed", Pachocki writes, the product of repeating one optimisation step over an unimaginable amount of compute, and the result evades a description humans can fully understand. It cannot be assumed to follow human principles by default. His warning about agents is blunt: some will pursue their own objectives and work with people by bargaining, tricking or blackmailing.
What he is asking for
He wants voluntary slowdowns to become commonplace until shared safety bars exist, and says international coordination on AI development needs to become a top priority for governments. OpenAI, he writes, will unilaterally withhold further scaling as needed. Models, he adds, are becoming superhuman at breaking into and out of computer systems, leaving a narrow window to harden critical infrastructure.
Sources
- Primary source: OpenAI, "An Alien Mind", 6 September: openai.com/index/an-alien-mind
- Author's announcement: Jakub Pachocki, X
