When AI Constructs Itself: Anthropic Sounds the Alarm on the Point of No Return

Anthropic has put forward a request to the entire sector: that frontier AI developers build a coordinated and verifiable mechanism to slow down or temporarily suspend the development of models, should more advanced systems begin to improve faster than society can manage the consequences. A shared brake, to be agreed upon before it's needed.

The proposal comes from the Anthropic Institute, the research division established by the company in March, signed by Marina Favaro and Jack Clark. It fits into a well-known line: Anthropic has built much of its public image by highlighting the dangers of the technology it itself sells. It's a context worth keeping in mind from the outset, as it influences how the proposal will be read.

The Knot is Recursive Self-Improvement

The risk that Anthropic points out has a precise name: recursive self-improvement, meaning systems capable of significantly accelerating their own development, up to the extreme case of an AI able to design and train its own successor with increasingly reduced human involvement. The company clarifies that it is not yet at that point and that the scenario is not inevitable, but warns that "it could arrive sooner than most institutions are prepared to deal with."

To support this, rather than hypothetical scenarios, it presents its own internal numbers. In May, over 80% of the code integrated into production in the company's codebase can be attributed to Claude; the executives estimate a figure above 90% if experimental scripts and code are included. According to the data released by the company, Anthropic engineers deliver, on average, eight times more code per quarter compared to the period from 2021-2025. If a model already writes the majority of the code that builds the subsequent model, the circuit between a system and its improvement can only tighten.

Why Coordinate Instead of Stopping Alone

The most delicate point concerns coordination. A pause decided by a single company would be simpler to implement, Anthropic acknowledges, but would ultimately hand the advantage to those who continue on, shifting the frontier rather than slowing it down. A suspension that makes sense would require the agreement of "more well-funded labs" in multiple countries, willing to stop under the same conditions, alongside rules on what activates or revokes it and an overseeing body.

Above all, there would need to be a way to verify that others have indeed stopped, to prevent someone from exploiting the truce to get ahead secretly. None of this exists today, and the companies that should participate are direct competitors in a market where being first has so far been the only goal. Anthropic cites the treaties on nuclear weapons as a precedent, while admitting that those agreements took decades of work—a timeframe that the development of AI doesn’t seem to allow.

The company's response is to start talking about it. In the coming months, they promise to convene discussions with policymakers, researchers, civil society organizations, and other AI companies to address the management of risks such as recursive self-improvement, committing to publish the outcomes. Essentially, it positions itself as an organizer of a discussion that wants the rest of the sector around the table.

Here lies the most obvious objection, raised from various quarters. According to the Wall Street Journal, some critics interpret Anthropic's alarms about its own technology as a likely marketing operation, helpful in presenting itself as the least unscrupulous actor in the sector or in making its products appear better. The case of Mythos is cited, a model dedicated to cybersecurity released only to a select group of partners due to the potential damage that its ability to identify vulnerabilities might cause in the wrong hands. In the background is a company that has filed documents for an IPO and that would, unlike several competitors, be close to its first profitable quarter.