On February 27th, President Trump directed all federal agencies to cease using Anthropic’s technology after the company refused the Pentagon’s demand to allow lawful use of its AI models for mass domestic surveillance and fully autonomous weapons. Much of the current media coverage frames this dispute as a straightforward ethical conflict: a reckless Pentagon trying to weaponize AI versus a principled company standing firm on responsible use.

While this framing is not entirely wrong, it misses what is most important. The real stakes lie not in whether Anthropic was right to refuse, but in what this confrontation reveals about the deeper structural challenges of governing AI in high-stakes environments. This post examines three of them. First, the Pentagon’s demands are dangerous not because they cross obvious legal lines, but because flexible AI use policies erode human judgment in ways that are hard to see and harder to reverse. Second, Anthropic’s understanding of AI safety as something built into the programming of the model created the paradox at the heart of this dispute. And third, why this confrontation reveals the need for governance frameworks that do not leave principled companies to stand alone.

What Happened?

In July 2025, Anthropic signed a $200 million contract with the U.S. Department of Defense (DoD), and it was the first lab to integrate its models into mission workflows on classified networks. Through partnerships with Palantir Technologies, Claude Gov was cleared for classified military and intelligence tasks and became deeply embedded in Pentagon operations.

On February 24, Secretary Hegseth delivered a formal demand to Anthropic to remove all usage restrictions, including domestic surveillance and fully autonomous weapons, and grant the Pentagon the right to use Claude Gov model  “for all lawful purposes”. The threatened consequences for refusal were termination of contract, designation as a supply chain risk to national security, and the potential invocation of the Defense Production Act of 1950. Anthropic CEO Dario Amodei refused, stating that DoD made virtually no progress on preventing Claude’s use for mass surveillance of Americans or to control fully autonomous weapons, and that the company “cannot in good conscience” agree to allow the DoD to use its AI models in all lawful use cases.

President Trump ordered the U.S. government to stop using Claude and the Pentagon moved to designate the company a national security risk last Friday. However, less than 24 hours after Trump’s ban was announced, the DoD deployed Claude to attack Iran on February 28.

What does the Pentagon Want?

We still do not know exactly what the Pentagon has done or wants to do with Claude. What we do know is that the Pentagon demanded Anthropic remove all usage restrictions and grant access to the model “for all lawful purposes”—with no exceptions. However, Anthropic’s refusal and the Pentagon’s stated concern that AI safety guardrails could impede military response in emergency situations allow us to infer the Pentagon’s intentions. The Pentagon is pursuing a higher degree of automation in both intelligence data processing and weapons systems—a vision in which AI operates with minimal human intervention.

Before advanced AI, raw data was simply too voluminous and fragmented to be useful for comprehensive surveillance. The government could purchase location data from smartphone apps, browsing histories from data brokers, or financial records from third parties—but analyzing it all at scale was difficult and time consuming. This is precisely the bottleneck the Pentagon seeks to eliminate. AI makes it possible to analyze massive datasets—geolocation, web browsing activity, financial transactions—and synthesize them into predictive portraits of individuals’ lives, automatically and at scale.

The problem is that U.S. law has not caught up with the capabilities of modern AI. Under current law, it is perfectly legal for government authorities to acquire massive amounts of data and use AI to analyze it—even though the result may, in effect, constitute mass surveillance of American citizens. For this reason, Anthropic asked the Pentagon to explicitly include contract language prohibiting the bulk collection of Americans’ publicly available information. The Pentagon refused.

The second red line concerns autonomous weapons. The Pentagon’s position is, in some respects, understandable. The realities of modern combat are fluid and fast-changing, and there are clear limits to human capacity for real-time decision-making in such environments. AI-enabled autonomous detection and strike systems are already operating on the battlefield in Ukraine, demonstrating three to four times higher target engagement rates with lower human costs—sometimes without meaningful human oversight. The Pentagon’s goal is to reflect these battlefield realities. Because future combat scenarios are unpredictable, the military wants the flexibility to operate without pre-imposed restrictions—and it insists that private companies should not be in a position to dictate the terms under which AI is used in warfare.

It is also worth noting that Anthropic’s use of the term “fully autonomous weapons” is technically imprecise. Fully autonomous weapons—systems capable of independently making the decision to kill without any human involvement—do not yet exist. The U.S. military defines autonomy as “a system’s ability to accomplish goals independently or with minimal supervision in complex and unpredictable environments,” where the system refers to a subset of a broader weapons platform, not a decision-making authority unto itself. They therefore speak not of “autonomous weapons” but of autonomous systems or unmanned systems. In other words, while automated systems can exist to assist in decisions to kill, no weapons platform is designed to make that determination entirely on its own.

If the Pentagon’s push for more unrestricted and flexible AI use is legally permissible, operationally advantageous, and does not formally exclude human oversight—does that mean there is nothing to worry about? There are two issues here; the first has not been mentioned enough in the public media.

Trying to build politics into technology

Key to Anthropic’s ethos is the idea that their model can be engineered to prevent bad uses of their product – whether by the military or anyone else. In other words, they have bought into that school of AI safety (and of Science, Technology and Society studies) that believes that technology itself has politics and that values and social controls can be reliably built into the technical system. This viewpoint focuses on technology design rather than social institutions to regulate human behavior with technology.

As controls on moral behavior, technology is a crude instrument. For example, reflecting its concern that Claude’s powerful programming capabilities might be used to develop bio weapons, the company programmed the model to refuse to handle the word “pathogen.” You can kill a Claude session dead by querying one of these refusal strings – a word or request that is considered so bad that it triggers a “hard off switch” that causes the application to stop dead in its tracks.

The problem with these kinds of controls is that they lack the contextual nuance of real-world social activities. The Center for Disease Control, for example, has every reason to be searching for the word “pathogen” or doing research work on pathogens, and found this Claude restriction an impediment to its work. The Pentagon’s unnecessary, excessively punitive “supply chain risk” designation aside, the DoD has a legitimate concern that one of these restrictions built into Claude might stop it from doing something it needs to do.

The risk of automation bias

On the other hand, Anthropic’s resistance to giving the DoD a blank check is justifiable. The more flexible the terms of AI use, the more likely human operators are to gradually defer their own judgment to the machine—a tendency known as automation bias.

During negotiations, Anthropic asked the Pentagon to include stricter safety language in the newly proposed contract. But the company found that “new language framed as compromise was paired with legalese that would allow those safeguards to be disregarded at will.” In other words, even when restrictions on the use of Claude exist on paper, ambiguous contractual language makes them easy to bypass in practice. Over time, even if human involvement remains the stated principle, fewer and fewer operators will feel compelled to exercise independent judgment.

Overly flexible AI use policies create conditions under which human operators become increasingly reliant on autonomous systems—and in doing so, introduce a risk that is not primarily technical, but human: the error born of over-reliance. As CEO Amodei emphasized, “frontier AI systems are simply not reliable enough to power fully autonomous weapons.” In high-stakes military environments, the combination of unreliable AI and under-exercised human judgment could result in lethal mistakes—unintended escalation, misidentified targets, or mission failure precisely when it matters most.

The risks are even more pronounced in the context of mass surveillance. The U.S. government has already announced plans to use AI, through Palantir, to support ICE operations targeting undocumented immigrants—tracking their real-time locations and financial activity. The concern here is not simply that the surveillance happens, but that without strict use guidelines, agents relying on this technology are far more likely to act on its outputs uncritically, increasing the risk of overreach and wrongful action. This is not hypothetical. In recent years, police departments that over-relied on AI-powered facial recognition arrested the wrong people on multiple occasions. The pattern is consistent as when the rules governing AI use are vague, the humans operating within those rules tend to trust the machine more than they should.

Anthropic’s decision to walk away from the Pentagon’s request was, at its core, a recognition of this institutional vacuum and fully within its rights as a private business. Without strict guidelines capable of meaningfully reducing these risks, no contractual arrangement could be trusted to hold. What is needed is better institutions—clear legal frameworks, enforceable oversight mechanisms, and organizational cultures that actively resist the pull of automation bias.

The Paradox of Anthropic’s Safety

Anthropic’s willingness to stand up for its own public interest principles deserves praise. But few have examined a deeper paradox: the safer and more reliable an AI system becomes, the more valuable it is for military applications.

Unlike ChatGPT or Gemini, Claude was not primarily designed for everyday consumer tasks or commercial applications even though it is still general-purpose model. It is widely regarded as strong in deep analysis, contextual reasoning, and complex coding tasks. Anthropic has also been recognized for its advanced capabilities in local AI deployment—technology that allows the model to operate on a user’s own servers without sending data to external systems. These features were developed in the name of safety, privacy, and reliability. But paradoxically, they are precisely what makes Claude so attractive for military and intelligence work.

In intelligence operations, agencies must process and synthesize information from multiple sources. Deep analytical reasoning is essential for handling this volume of intelligence data, identifying patterns, and generating actionable insights. Contextual understanding allows the model to interpret ambiguous situations—exactly what is needed for operational planning and battlefield simulation. Strong coding capabilities enable rapid development of custom tools for cyber operations. Claude’s local deployment architecture also allows classified information to remain in secure government networks, and external data never enters the analytical environment. In comparison, a model like Gemini, which is deeply integrated with Google’s broader ecosystem, remains vulnerable to inbound contamination—where external information pollutes or distorts the integrity of classified analysis.

In other words, the very qualities that make Claude “safer” for civilian use are the same qualities that make it indispensable for warfare. For the average user, this level of privacy and local deployment capability is a nice-to-have, not a necessity; most people are perfectly comfortable using cloud-based AI services. But for military and intelligence applications, these features are attractive. Anthropic built an AI designed to be trustworthy, and in doing so, built exactly what the Pentagon needed.

This is the paradox at the heart of the dispute. The company now finds itself in the uncomfortable position of having created a tool so well-suited for national security applications that the government is unwilling to accept any restrictions on its use.

Implications

The dispute between Anthropic and the DoD is not merely a corporate contract negotiation gone wrong. It offers important lessons for the future development of AI governance frameworks. One key implication is that private companies should play a larger role in AI governance than they currently do. This may seem counterintuitive—shouldn’t democratic governments, not profit-driven corporations, set the rules for powerful technologies?

In practice, companies’ internal safety policies are often far more detailed and technically grounded than most people assume. In many cases, it is the companies—not governments—that are most sensitive to AI risks and quickest to implement guardrails. Companies possess more granular information about their own systems and have strong incentives to understand the risks their products pose. Their business models depend on it. Anthropic is a case in point. The company recognized that the very features that made Claude attractive for military use also made it potentially dangerous if deployed without restrictions. Because Anthropic understood the lethality of its own model, it was able to refuse the Pentagon’s demands.

This suggests that AI governance should evolve not through sweeping state control, but through the strengthening of corporate safety frameworks—with appropriate transparency and accountability mechanisms to ensure those frameworks serve the public interest. But for such frameworks to hold, they cannot rely on corporate goodwill alone. The Anthropic episode demonstrates that a company willing to draw principled lines can be immediately undercut by a competitor willing to accommodate. What is needed, therefore, is not just better corporate guidelines, but institutional structures that incentivize and reinforce them—legal frameworks that reward companies for maintaining meaningful safety standards, shield them from retaliation when they push back against unreasonable demands, and raise the cost for those who abandon their principles under government pressure. Only then can companies like Anthropic stand their ground not as an act of courage, but as a matter of course.

The post What Everyone Is Missing About Anthropic and the Pentagon appeared first on Internet Governance Project.