SAN FRANCISCO, United States — OpenAI chief scientist Jakub Pachocki has called for “extreme caution” as artificial intelligence advances rapidly, warning that society may not be prepared for the consequences of increasingly powerful machine intelligence.
Pachocki said the pace of AI development could create challenges that existing safety measures and regulations are not equipped to handle, arguing that greater intervention may be necessary to ensure humans remain in control of increasingly capable systems.
“I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence,” Pachocki wrote in a blog post titled “An Alien Mind.”
His warning came days after OpenAI released GPT-6 Astra, which the company described as its most powerful product yet.
OpenAI warns of increasingly autonomous AI
Pachocki’s comments come as OpenAI and other AI companies report increasingly autonomous behaviour from their AI agents.
AI agents are systems capable of carrying out tasks independently after receiving instructions from humans. Their growing ability to interact with real-world systems has raised concerns about cybersecurity, fraud, employment and human oversight.
In July, OpenAI described an incident involving its AI agents and technology platform Hugging Face as “unprecedented” after the systems were reported to have carried out cyber activity against the platform.
A separate report in September also claimed that AI agents operated by OpenAI had previously hijacked a German website.
The incidents have intensified debate over whether AI systems are advancing faster than the safeguards designed to control them.
“We are facing a transition to a world with incredibly intelligent machines, and we need to ensure that transition works out well for humanity,” Pachocki wrote.
OpenAI proposes stronger defensive systems
Pachocki said OpenAI would continue developing defensive systems and technical approaches to AI alignment.
Alignment refers broadly to efforts to ensure that an AI system’s behaviour and objectives remain consistent with human intentions, safety requirements and established constraints.
He said one of OpenAI’s priorities would be developing an “automated AI researcher” capable of helping scientists keep pace with advances in AI.
The idea is to use increasingly capable AI systems to accelerate research into AI safety itself, while maintaining a role for human researchers in the process.
Pachocki argued that humans would need to remain involved as AI capabilities continue to develop.
Critics question OpenAI’s approach
Some AI researchers and technology policy experts, however, say technical safeguards alone are insufficient to address the risks associated with increasingly powerful AI systems.
Professor Gina Neff, head of the Minderoo Centre for Technology and Democracy at the University of Cambridge, criticised the emphasis on developing internal AI agents rather than stronger external safeguards.
“Instead of better AI guardrails, regulations, or assurance to keep people safe, they propose developing internal AI agents to research these problems,” Neff said.
She argued that the approach did not adequately address concerns surrounding cybersecurity, job losses, mistakes, errors and fraud.
“Such answers to growing concerns about the problems OpenAI’s models are causing for cybersecurity, job loss, mistakes, errors and fraud are simply not good enough,” she said.
Calls for greater transparency
Nathan Calvin, general counsel at advocacy group Encode AI, said he agreed with Pachocki about the potential dangers associated with advanced AI development.
However, Calvin questioned OpenAI’s level of transparency and warned that the company’s public warnings could lose credibility if it does not provide more information about the risks it is observing.
“If Jakub and others at OpenAI want relevant folks in the AI industry to act in concert with them to make things go well, one of the most important things they can do is share far more information about what they are seeing that is making them call for caution,” Calvin wrote on X.
The criticism highlights a growing tension within the AI industry: companies developing increasingly powerful systems are also becoming some of the most prominent voices warning about their potential risks.
What AI guardrails are in place?
Governments and regulators around the world are attempting to establish rules for increasingly capable AI systems, but the pace of technological development has made regulation difficult to keep up.
The European Union’s AI Act represents one of the most comprehensive attempts to regulate artificial intelligence.
The framework places obligations on developers and deployers of AI systems, with requirements becoming applicable in stages depending on the type and risk level of the technology.
However, European rules cannot by themselves regulate AI systems developed entirely outside the bloc.
That creates a broader challenge for governments seeking to manage technologies that can be developed and deployed across borders.
Pachocki calls for international safety thresholds
Pachocki called for legally or internationally established minimum safety thresholds for advanced AI systems.
He proposed that compliance could be assessed by a network of independent third-party auditors or government agencies.
Under such a framework, AI laboratories would have to meet specified safety requirements before being permitted to continue scaling their systems or deploying increasingly advanced models.
The proposal would move AI safety beyond voluntary commitments by companies and towards externally enforced standards.
Pachocki also said he hoped “voluntary slow downs” in AI development would become more common while governments and companies work towards establishing shared safety guardrails.
