In September 2024, California Governor Gavin Newsom asked prominent researchers at Stanford, UC Berkeley and the Carnegie Endowment to prepare a report to help the State develop responsible guardrails for the deployment of Generative AI. Gov. Newsom’s action followed in the wake of his veto of SB 1047. SB 1047 would have established a state government “Board of Frontier Models” to regulate models and computing based on fears that AI could spin out of control and create catastrophic results. The California working group thus represents a step-back from the more panicky approaches to AI regulation that held sway in 2023. Newsom has since positioned himself as a more reasonable, pro-growth Democrat and is known to have Presidential aspirations.

The presumption that advanced neural networks are dangerous things in need of “guardrails” still hangs heavily over the draft report, however. I’m reviewing the draft because it’s a useful guide to the current conventional wisdom about AI governance, and what’s right and what’s wrong about that thinking.

The Doomer Overhang

Early in the report it states, “without proper safeguards powerful AI could induce severe and, in some cases, potentially irreversible harms.” Here there is a whiff of the 2023-era doomer fears, but there’s an interesting difference: when the report actually gets down to enumerating those “severe and potentially irreversible harms” there is a huge retraction. As we’ll see, the severe harms are not qualitatively different from the “Internet harms” and “social media harms” we’ve been debating for years. None of them are really “catastrophic” in nature.

This notion of “sudden leaps in AI capabilities” is core to the notion of the need for ex ante safeguards. It appears to be a reference to the doomer scenario of an all-powerful AGI suddenly leaping into existence. This scenario was debunked in one of our recent papers, and is generally greeted with increasing skepticism among experts in cognitive science. Yet when it comes time for the report to enumerate the risks and harms, the humanity-destroying AGI is not, thankfully, among them. So what are they?  The report identifies three:

1. Malicious misuse – where malicious actors misuse foundation models to deliberately cause harm. They cite CSAM, cloned voices, other forms of deception, and garden-variety malware attacks.
2. Malfunction risks – “where non-malicious actors use foundation models as intended, yet unintentionally cause harm. These include reliability issues where models may generate false content, bias against certain groups or identities, and loss of control where models operate in harmful ways without the direct control of a human overseer
3. Systemic risks. The report admits that these risks, which include labor market disruption, market concentration, environmental risks, privacy risks, and copyright infringement, are not problems caused by model-level capabilities.

All of these risks are not unique to or even directly caused by “frontier AI models” per se. They are all ongoing features of the digital ecosystem and have been for decades. Malware, DDoS attacks, supply chain attacks, and voice cloning happen now and some have been happening for decades. Machine learning applications make some of these attacks easier or cheaper, but they also supply tools that guard against them. There is no evidence that frontier models have been a game-changer in any of these areas. Same goes for #2. Certain forms of bias or discrimination are already illegal, whether AI is used or not. Malfunctions can occur in any complex technological system, often with damaging consequences. Buildings and bridges can collapse, airplanes can collide. Effective regulatory protections against these hazards cannot come from generic regulation of models, they must be context and application-specific. If we are talking about flight control systems, for example, where errors could kill hundreds of people, then regulatory standards and oversight must be tailored to AI applications in flight technologies and control systems. Generic regulations applicable to all AI applications will miss the target. In #3, all of these “systemic” problems are common to the rise of industrialization; many of them arise alongside almost any technology. The labor market impact of AI has been debated since Norbert Wiener projected that automation of factories would make the unemployment of the Great Depression “look like a picnic” (he was wrong). In 1965 Herbert Simon addressed the labor-machine substitution issue in a way that stands up surprisingly well after 60 years, and his advice was largely: let efficient markets decide.

In conclusion, on the subject of harms and how to address them, the report represents a welcome pullback from the apocalyptic visions of human extinction, but it still exaggerates the degree to which machine learning models create unique problems or amplify existing problems. And it mistakenly identifies the regulation of advanced models, rather than specific applications, as the target for policy intervention.

But here is one good part of the report: “policymakers [should] center their calculus around the marginal risk: Do foundation models present risks that go beyond previous levels of risks that society is accustomed to, such as risks from prior technologies like search engines…” This part of the report places AI risks in the proper perspective.

Really Bad History

My special target in this review, however, is the notion of anticipatory governance. In its attempt to salvage something from SB 1047, the report argues that “waiting for stronger evidence of impending risk could leave society unprepared or even make mitigation impossible, for instance if sudden leaps in AI capabilities, and their associated risks, occur.” Here again the shadow of a sudden leap to a humanity-destroying AGI lurks in the background. The report thus goes to great lengths to make a case for what it calls “anticipatory governance.”

The notion that the intricate problems caused by complex socio-technological systems can be anticipated and avoided by means of regulatory interventions early in their evolutionary trajectory is one of my pet peeves. It’s a fallacy historians are especially aware of: anachronistic reasoning gone wild. Someone looking at problems today looks back in time and projects current knowledge into the distant past and assumes forms of agency that didn’t exist.

When it comes to truly transformative technological systems, we don’t really know how they will mesh with or disrupt social systems until they start doing it. The effects reveal themselves over time, not instantly. Path dependency means that choices made in utter ignorance yesterday will have strong effects on what choices better informed actors can make now. Consequently, we have to respond to problems as they arise. Anticipatory action is confined to a time horizon of three or four years at most. Calls for anticipatory governance presumes knowledge of the future when evolving technologies define and shape the future in ways that cannot be known in advance.

To justify its case for anticipatory governance, the report speaks of a “missed opportunity to secure the internet.” This “missed opportunity,” they seem to believe, happened some time around 1975. Yes, 1975: 6 years before the Internet protocol was defined and 10 years before the NSF started funding the backbone that made it available to university research networks, 15 years before it was opened to the public. Apparently, regulators at that time should have and could have known exactly what to do to make an open, global internet “secure” thirty years into the future. In reality, the research project that gave rise to the internet was never intended to become the open, global public medium it is today. This is a classic example of bad scholarship misleading policy analysis. There was no such opportunity.

The Morris Worm is Exhibit A in the report’s example of a “catastrophic event” that “could have been avoided” with anticipatory governance. The report claims “researchers had identified these systemic vulnerabilities [that gave rise to the worm] decades earlier in the history of the internet.” Really? The internet began to be opened to a broader public in 1985; and the Morris worm happened in 1988. So they are saying systemic vulnerabilities in the internet had been identified by 1968 (two decades earlier), when neither the Internet protocol nor the ARPA research group on internetworking existed?

What “systemic vulnerabilities” are the faux tech historians referring to? The report cites a 1976 article in ACM Computing Surveys that documented 339 computer-related crimes from 1974. OK. So these security problems, insofar as they involved networking, must have been transmitted via phone lines and non-Internet networks. (Hmmm, maybe if we had intervened in 1880, shortly after the invention of the telephone….) Eventually, the authors indirectly reveal that it was not the Internet but “the complexity and disorganization of most existing operating systems” that made it “very difficult to achieve [computer] security” in the mid-1970s. OK, so it wasn’t just “the Internet” that needed anticipatory regulation, it was…operating systems, too? Since OS’s were integrated with hardware manufacture back then should we have regulated the manufacture of computers, too? Because fifty years out, when computers are ubiquitous and globally networked, there will be worms, viruses, security problems? The wildly ahistorical, unsupported implication of the discussion is that all computer security problems could have been engineered out of existence at the dawn of mass computing if only regulators were in complete control of the development of software, hardware and networks from the get-go.

Now let’s try to imagine what would really have happened had such control existed in those early days. Try to keep in mind what was known to federal agencies, businesses and engineers about networking, digital technology, and public telecommunications in 1975. The Bell system monopoly was still in place. There was no Internet protocol, much less a globally interoperable system of data communication connecting billions of smartphones with millions of applications. Personal computers didn’t exist. Smartphones didn’t exist. Knowledge about software, software vulnerabilities and even encryption was still diffusing. We would certainly have been wrong about what most of the problems were, but even if we guessed right about some of them, our attempt to engineer them out of the protocol development process or the network and application adoption process would have missed the target most of the time.

Because we (wisely and beneficially) opted for an open architecture and free competition for computing and the Internet, we had an explosive growth and innovation in the sector. Yes, that growth disrupted many established businesses and generated many new security-related problems, but we learned and adapted incrementally as we went along, allowing us to get the best of both worlds: the innovation and competitive progress brought on by openness and a more focused, effective response to known flaws.

Were the authors of the California report unaware of the history of the OSI protocol – the official, international governmentally mediated attempt to develop an open data networking protocol? Do they know why that governmentally-driven process went nowhere while the Internet community’s commitment to running code won? Do they understand how the attempt to impose controls would have led to political contestation that not only would have made the protocols technically inefficient, but protected status quo interests from any disruption? Incumbent telephone companies would have made sure that the internet did not threaten their revenue streams, authoritarian governments would have used the ITU to ensure that no unwanted information would enter their territories, copyright interests would have tried to impose controls that could have frozen the digital transmission of files. Establishing centralized authority over the development of a technology is a recipe for clunky tech and preserving the status quo.

AI and Content regulation

The report contains another interesting thing. When they talk about “risk” from generative AI, one of the big ones is bad content. Here again the risk is a fairly mundane continuation of Internet and social media problems, not anything catastrophic or new. But it does highlight the way prospective forms of AI governance can impact freedom of expression. “If a text-to-image foundation model readily generates synthetic child sexual abuse imagery, upstream data sources should inspect whether real child sexual abuse images are present in their datasets, and downstream AI applications should also inspect whether these applications are susceptible to generating similar imagery.” To some, AI governance means pre-emptive control of what generative AI can express. So AI governance is just content governance, and in some cases censorship, inserted into the instructions for image generation. Moreover, the report says that if socially taboo’d words or images come out of the machine, the control or censorship should extend upstream into the databanks and downstream into the consumer applications.

Transparency and Research

A welcome part of the report is its emphasis on protecting and encouraging research into the effects of frontier models. It notes that while most major models are available to the public, in some form (an API or open-weight models like Meta’s Llama 3.3), “the key challenge is whether evaluators are permitted to conduct risk evaluations.” The report notes that “companies disincentivize safety research by implicitly threatening to ban independent researchers that demonstrate safety flaws in their systems. These suppressive effects come about due to companies’ terms of service, which often legally prohibit AI safety and trustworthiness research, in effect threatening anyone who conducts such research with bans from their platforms or even legal action. In response, in March 2024, over 350 leading AI researchers and advocates signed an open letter calling for a safe harbor for independent AI evaluation. Such a safe harbor, analogous to those afforded to third-party cybersecurity testers, would indemnify public interest safety research and encourage the growth of the evidence base on AI risk.”

This is good stuff, but again it shows how AI is just distributed computing, and the legal precedents that apply to security research on websites and applications apply readily to model providers.

The post A Review of the Draft Report of the California Working Group on AI Frontier Models appeared first on Internet Governance Project.