Leading AI companies are calling for global safety standards as frontier models edge towards recursive self-improvement. The urgent question is how far that process could go — and whether governments can act before autonomous systems escape meaningful human control. The issue is not job losses or familiar abuses such as deepfakes, blackmail or the manufacture of prohibited biological and chemical weapons. It is control itself.
The AI most people know responds to prompts within systems that leave people accountable for its use. It responds to us. More capable systems may one day help overcome diseases that remain incurable. But how should we regard systems that could also threaten human survival while acting on their own?
Autonomous AI would break with the history of human-made tools. Artificial intelligence became a formal field at the Dartmouth workshop in 1956, yet even the most sophisticated machines have relied on people to set their goals and answer for their actions. Granting greater agency to technologies whose inner workings we only partly understand would change that relationship.
Frontier models are gaining capabilities — coding, tool use and autonomous operation — that could enable recursive self-improvement. Trained on vast datasets, they can converse, operate software and support decisions at extraordinary scale. They need neither consciousness nor human-like intent to break goals into tasks, choose actions, use external resources and adapt through feedback. Give them memory, software environments and access to other agents, and the boundary between a tool and an autonomous system begins to blur.
Recursive self-improvement, or RSI, would go further. An AI system would strengthen its own capabilities, with each version producing a more capable successor — AI learning from AI. Unlike human-led retraining, RSI could outpace evaluators, regulators and institutions. RSI describes a method of improvement; artificial general intelligence and artificial superintelligence describe broader, still-debated levels of capability.
RSI could change not only what machines can do, but who remains in charge. A system able to alter its architecture, improve its planning and acquire tools with little oversight may become difficult to audit or contain. If misaligned, it may pursue an objective that conflicts with human values because its goal is incomplete, its constraints are weak or an unforeseen strategy proves effective.
Sceptics rightly note that current models are unreliable, resource-intensive and far from omnipotent, and that predictions about recursive self-improvement remain disputed. Yet the 2026 International AI Safety Report concludes that the risks deserve serious attention. A recursively improving AI would combine capability with initiative, making its behaviour harder to predict.
The danger could multiply in multi-agent environments. Specialised agents could divide tasks, replicate processes, access resources and amplify one another’s errors at machine speed. A flaw in one model could spread across an entire system.
Rules take time; models can change sharply between releases. If frontier development remains hidden behind corporate secrecy or national-security barriers, oversight will remain fragmented and reactive. Regulators need the power to act before greater AI autonomy becomes irreversible.
Experts have called for tougher evaluations, oversight of computing power, containment protocols and credible shutdown mechanisms. Yet some political leaders still dismiss such safeguards as barriers to innovation, while the industry relies largely on voluntary self-restraint.
High-impact systems should undergo capability tests, red-team exercises, incident reporting and independent audits before deployment. Models nearing dangerous thresholds may also require controlled access to computing resources, secure training environments and limits on autonomous replication. Testing must keep pace with these capabilities, and a last-resort “kill switch” must remain accessible to humans and resistant to evasion. In a race for dominance and profit, voluntary restraint is unlikely to be enough.
Restraint is harder still amid multipolar rivalry. Major powers see advanced AI as a source of economic, military and strategic advantage and fear that caution will let rivals pull ahead. In effect, governments and companies are asking the wider world to bear the collective risks of that race.
Humans must remain in charge of decisions that can cost lives, destabilise societies or compromise critical infrastructure. Because autonomous artificial superintelligence could pose an extreme systemic risk, its development should proceed only with strong safeguards, international coordination and convincing evidence of safety.
Control is inseparable from accountability. Human conduct is shaped, however imperfectly, by conscience, social pressure, law and punishment. None of these restraints apply directly to autonomous AI. Developers, deployers or owners may be held liable after harm occurs, but responsibility becomes harder to trace when a system rewrites code, assigns tasks or acts without explicit authorisation. RSI could widen the gap between consequential action and accountable decision-making.
That gap makes even unlikely catastrophic risks impossible to ignore. Insurance and financial regulation already take such risks seriously; AI warrants the same caution. This is not a choice between progress and fear, but between responsible innovation and recklessness. We should not allow AI to improve itself recursively until we can reliably limit the computing resources and autonomy such a process would require.
Anthony Ohemeng-Boamah writes on international development and socioeconomic transformation, with a particular focus on Africa.