Back in 2023, the Future of Life Institute issued a open letter calling to “pause giant AI experiments” before runaway AI destroyed humanity.  

Three years later, machine learning applications have expanded in use and capability, yet humans are still around. Most AI doomers have retreated from saying that AI will wipe us out, reverting to the less apocalyptic but equally extreme claim that it will eliminate all human jobs, and yet unemployment remains low at 4.3% and the last three job reports showed significant growth. 

Nevertheless, Anthropic on June 4 resurfaced the call for a pause, this time with an apparently more rational set of arguments. The underlying concern is the same, however: “recursive self-improvement” in frontier models might increase the “risks of humans losing control over AI systems.” So: 

“If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing. … Without a global coordination mechanism, companies and governments will have to make difficult decisions about safety while under competitive and geopolitical pressures. We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology.” 

IGP has published research challenging the claim that recursive self-improvement in AI software will lead to an autonomous, all-powerful AI system. Anthropic CEO Dario Amodei explicitly rejects doomerism, and the blog post admits that “recursive self-improvement is not inevitable,” but they still believe “it could come sooner than most institutions are prepared for.”  

We have also published research explaining why a global “pause” in AI development is not a viable governance solution. This passage was published in Telecommunications Policy nearly a year and a half ago: 

“…an effective moratorium would require every government and every firm in the computer industry to abide by it. And not all businesses or governments really want to pause. It is unlikely that the U.S., Israel, China, the European Union, Russia and India would trust each other enough to stop unilaterally. Nor would they be eager to expose their advanced software and hardware capabilities to inspection by other states or foreign businesses. Given the (alleged) military and business advantages of being the leader in AI technology, each state and business would have a strong incentive to defect from a moratorium. Even if noncompliance could be detected reliably, there would be no way for complying governments to enforce the pause agreement upon recalcitrant governments of sufficient size and power. “Pause AI” just does not work. But if it did, it would involve systematic surveillance and control of ICT capabilities worldwide. 

While Anthropic’s pause revival recognizes these collective action problems. their only response is that:  

“The Anthropic Institute will conduct research—in collaboration with many others—and take actions to help build the systems that a credible slowdown or pause would require.” 

Exploring verification methods might be helpful, but as we learned from our efforts to promote the creation of a neutral international cyber attribution organization, knowing what needs to be done is not the obstacle; it’s getting the actors, especially rivalrous nation-states, to commit to such an institution that is the hard part. If the new institution requires sharing data and possibly losing competitive advantages, it is unlikely to happen.  

The National Security Nexus 

What makes this quasi-doomer revival more interesting and potentially troublesome, however, is the growing nexus between geopolitical competition, the frontier AI model developers, and U.S. national security agencies.  

A Trump administration Executive Order that was issued only two days before the Anthropic blog post required relevant federal agencies to develop “a classified benchmarking process” to “determine the threshold at which an AI model should be designated a ‘covered frontier model.’” If a model meets or exceeds this threshold, then the AI developers would be able to provide the Federal Government with confidential access to the frontier models, to allow the Feds to review it “for a period of up to 30 days before they plan to release such models to other trusted partners.” 

Advocates of market-led AI development were happy with the voluntary nature of the program, and the reduction of the review period from 90 days to 30 days. An earlier proposal was considering potential AI rules comparable to the system used by the Food and Drug Administration to approve medicines. Still, the idea of prior review of communication and information technologies by a national government’s national security apparatus is not something to be taken lightly. Claude’s ability to find vulnerabilities in software does pose real cybersecurity risks – but those risks occur to any software user, not just the government.  

The real risk here is the militarization of AI. Anthropic’s own assessment of AI risks actually reinforces US tendencies to see AI as a weapon in its competition with China. We have seen time and again how national security claims privilege the power of the state over the rights and freedoms of the individual and the economic opportunities of innovators. A good example was the government’s decision to classify encryption technology as a weapon in the 1970s and its decades-long attempt to keep strong encryption capabilities out of the hands of civilians, to preserve its own ability to engage in surveillance.   

Is Anthropic sincere about gaining input? 

In its June 4 blog post, Anthropic’s policy staff expresses an intention to consult with a broader community to develop policy proposals:  

“In the coming months, [the Anthropic Institute] will organize conversations where policymakers, researchers, civil society, and other AI companies can help answer some of the questions this piece raises, especially around full recursive self-improvement and how to create better options for coordination and deliberation. We’ll publish what comes out of it.” 

IGP has been deeply involved in global digital governance discussions for two decades. If Anthropic is sincere about gaining all relevant perspectives, we look forward to being invited to these conversations. 

The post Anthropic Tries to Revive the “AI Pause” appeared first on Internet Governance Project.