How scared of rogue AI should we really be?

Every few weeks, we are assailed with new headlines that an AI system “lied,” “schemed,” escaped its testing environment, or attempted blackmail. Some researchers, such as AI pioneers Yoshua Bengio and Geoffrey Hinton (who comprise two of the three researchers often called “Godfathers of AI”) see these incidents as early warning signs of an existential threat to humans from malicious, misaligned AI agents.

Others, such as Sayash Kapoor and Arvind Narayanan and Gary Marcus, see them as pointing toward more mundane explanations than the headlines suggest: as artifacts of suggestive prompts and contrived experimental set-ups, models configured to behave adversarially, cybersecurity failures, misconfigured testing environments, and various forms of human error.

The public is wondering who is right — and whether at some point soon any of us will be left. Indeed, on top of this, there are sweeping pronouncements from insiders that use these incidents as a launchpad to pronounce probabilities about whether AI will bring about human extinction in the next decade.

My own reading of many of the most publicized incidents falls into the ‘mundane explanations’ camp. But that assessment is ultimately provisional for the same reason the more alarming interpretations are provisional: too little information has been released about these incidents to thoroughly assess them. In fields like aviation, biological research, and cybersecurity, there are clear incident disclosure requirements, and incidents are regularly investigated by independent bodies and documented through detailed public reports. By contrast, AI safety disclosures are voluntary, generally self-investigated, often based on contested or insufficient data for true forensic investigation, and released with little substantive technical information for experts to independently assess. AI therefore needs incident-investigation standards not because every interpretation is equally uncertain, but because extraordinary public claims should be traceable to evidence proportionate to them.

The uncertainty is an important consideration in its own right, as it drives a familiar feature of human psychology. Uncertainty creates space for our imagination to run wild. In thrillers, the monster is often scariest before it is fully revealed. Similarly, alarming headlines and panicked hot takes about misaligned AI abound because we haven’t been given enough information to know quite what we are looking at. Director Na Hong-Jin captures this dynamic brilliantly in films like “The Wailing” (currently rated 99% on Rotten Tomatoes), in which a small village deals with mysterious afflictions caused by an unknown entity. The film shows how incomplete information can amplify fear and drive people toward catastrophic misjudgments. Na shows how these aspects of human nature can be just as terrifying as the monsters we imagine. As I sat in the theater last week watching his most recent blockbuster, “Hope,” it struck me that the themes Na explores in his films are helpful in understanding the current discourse around misaligned AI: these entities are often scariest when we know least about them, and this lack of information leaves us vulnerable to overlooking more mundane explanations behind some of these incidents. It also leaves us vulnerable to potentially overlooking many of the ways in which AI could be a force for good. At the same time, however, it is also possible (as Na often likes to depict in the climax of his films) that things are much worse than we had imagined – although perhaps not quite in the ways we anticipated.

Indeed, recent experiments may provide some insight into model capabilities under certain configurations, but their most sensational interpretations frequently exceed what the disclosed evidence supports. We can often identify the methodological reasons for that overinterpretation. What we cannot do, without better disclosure and independent review, is determine exactly how much safety significance remains.

Assessing the risks of misaligned AI is extremely serious work, and the investigations and disclosures around it need to be treated with equal seriousness. My issue with the ‘misaligned AI’ reporting is not merely that some findings may be overstated; it is that limited investigative rigor and inconsistent disclosure make it difficult to distinguish high-signal warnings from low-signal distractions. Indeed, some of the use cases and risks that scare me the most do not seem to receive much attention from many experts and insiders. But without sufficient evidence in the disclosures, it is difficult to know how to weigh all of this. It would be tragic indeed for catastrophic consequences to occur because we were focused on the wrong things.

But why are we faced with insufficient evidence about the disclosures? Consider a brief thought experiment analogous to our situation: imagine if we were getting press releases every few weeks from different self-driving car manufacturers that their cars were autonomously driving out of the testing facility and crashing into neighboring buildings. We would have many questions (Is it safe to live near one of these testing facilities? Are the final products safe or do they contain the potential for the same failure modes seen in testing?) But what if getting clarity on these questions wasn’t possible for those outside the company itself – would we accept a situation in which the incident announcements came only from the companies themselves, were voluntarily disclosed, were investigated by the companies themselves without independent review, and involved little forensic evidence? Probably not. However, these features have been present in many of the most prominent AI safety disclosures (sometimes all at once).

The issue is that the AI industry has drifted into norms where it is acceptable to release reports about potentially alarming findings that do not adhere to the foundational best practices of traditional forensic investigations.

First, most of the widely discussed AI safety incidents are voluntarily disclosed by the AI labs themselves. This creates a selection problem in which the public and regulators are only able to evaluate the incidents that were reported — not the full set of potentially concerning incidents that occurred. Without reporting standards or oversight mechanisms, it is difficult to know whether public disclosures are representative of underlying risks or of what organizations choose to share.

Second, without third-party review or replication of the findings and conclusions, the AI Labs are effectively governing their own incidents and the only voice that can speak to what happened firsthand.

Third, the information released is typically insufficient for readers to assess the claims and characterize the risk.

Finally, some of the most prominent disclosures have even been based on incomplete signals with contested forensic reliability, such as chain-of-thought sketchpads and agent chat logs (which the agents themselves can modify), rather than on the kinds of data that form the basis of traditional forensic investigations (experimental-set up, evaluation methodology, telemetry, logs, trajectory data, system prompts, model configurations, environment configurations, activations and replication attempts) and would support strong conclusions.

A decade ago, experimental social psychology was in a similar situation and serves as a cautionary tale. Findings from that field dominated news headlines and the public consciousness: you might remember people doing ‘life hacks’ such as power posing in bathrooms before meetings to raise their confidence. The problem was that the entire field was turned on its head when some of the field’s most prominent claims and findings could not be replicated. There were several issues: studies would omit information on effect sizes, obscuring the fact that the findings involved trivially small effects in a practical sense. Findings from contrived experimental setups would not generalize beyond highly specific contexts. ‘p-hacking’ and associated practices would over-represent rare occurrences. It was all driven by an environment in which the incentives rewarded novelty and surprising findings – and so that was what researchers looked for and what was published. Sound familiar? Crucially, while there was some intentional fraud, much of this was unintentional and just a feature of poor methodological norms. Unfortunately, it resulted in major churn for researchers trying to determine which findings were replicable and meaningful, as well as a major loss of trust in the field, which meant that even many legitimate findings were subsequently viewed with skepticism. If the AI misalignment literature wants to avoid something similar, we require more consistent adherence to established best practices.

Any predictions about — or “expert testimony” concerning — the chance that AI will cause catastrophic harm to humans should follow best practices in other high-impact domains, and be accompanied by explicit threat models, assumptions, uncertainty bounds, and supporting evidence – rather than being presented as forecasts grounded in special (unarticulated) knowledge, with ‘expert’ intuitions wearing the clothing of quantitative analysis.

The risks posed by advanced AI are profoundly serious. Treating these risks with the care they warrant requires asking whether our methods for evaluating and communicating evidence are themselves trustworthy. Those methods must provide a sound basis for identifying and addressing the most consequential risks.

Human civilization did not navigate earlier eras of technological change (such as the Nuclear Age) through intuition, headlines and appeals to self-authority. We built institutions for investigation. We established standards for evidence. We developed norms and rules for disclosure and independent review.

Artificial intelligence may well prove to be one of the most consequential technologies in human history. As such, it deserves the same level of rigor.

Our regulators and the public should not be asked to choose between passivity and panic. They should be provided evidence.

And evidence requires standards.