The AI Safety Debate Has Lost the Plot. We Asked 10 Experts What the Real Nightmares Are

Save Story Save this story Save Story Save this story In early fall, I felt my anxiety rising from its usual baseline. Was it September’s back-to-school energy? The looming midterm elections? Nope, I realized with annoyance: It was AI—again. Only this time, instead of its purveyors warning of massive job displacement, they were foretelling mass extinction. Who needs jobs when we’re all dead? Curiosity soon replaced my knee-jerk irritation. The summer of 2026 was a clear turning point in the AI narrative. Less than four years after ChatGPT launched, OpenAI software broke out of a supposedly safe test environment and hacked into a major open-source platform, Hugging Face . These bits of software— AI agents —were assigned human attributes: They had “gone rogue,” displayed “bad behavior,” “cheated.” Soon after, former and current Anthropic researchers suggested that advanced AI might kill us all in the next decade. Is any of this real ? On the one hand, from my desk as a Silicon Valley correspondent, I often hear that workers in the AI labs are stunned by the advancements they’re seeing. On the other, the end of humanity seems extremely far-fetched. Amid the hyperbolic discourse, whose views can we trust? So WIRED set out to compile a guide to AI safety—a set of ideas that rewire our collective thinking around what safety really means and how it should work. I spoke with a panel of experts across disciplines, asking each of them: Is the world going to end? Once that was covered, I grilled them on the biggest risks they think we’re facing. Almost no one, it turned out, was Pollyannish. Nearly everyone I spoke to expressed genuine concerns about the threat of cyber and biological attacks. One expert who studies the risks of AI in nuclear operations said she’s stunned by how quickly error-prone AI systems are being adopted in defense. AI-enabled attacks in these areas could be catastrophic, if not fully apocalyptic. Most members of the panel were skeptical about any pause in development: They nearly all agreed that, with big AI companies so tightly cuffed to the trains of capitalism, a pause wouldn’t last long. “Democracy” and “governance” came up often—or anti-democracy, as more than one person suggested that AI’s current trajectory is tyrannical. Lawmakers have a big role to play, but many of my sources think governments will be too slow; one prominent technologist said it’s almost absurd to view the US courts as arbiters of global AI safety. Still, the famed computer scientist Yoshua Bengio urged the public to apply pressure to politicians, adding to me, “That’s your job too.” So, what—or who—will keep us safe from AI? Looking to regain some faith in humanity, I called a theologian who often meets with the AI labs. If there’s even a slim chance that we’re losing control of the essential institutions that undergird society, he said, then we should focus on foiling that. But AI raises other questions too. What does it mean to be conscious? To create? To commune with other people? The answers are “entirely up for grabs,” he said. My anxiety’s still kicking around, but I took some comfort in knowing that corporations don’t get to call all the shots on the future. The bigger questions facing humanity are in all our hands. BIGGEST CONCERN: Nuclear catastrophe I’ve been working hard to educate the nuclear community on the risks and lack of reliability of AI systems, and I’m concerned by the willingness of many people in the sector to integrate these models into their workflows, into nuclear operations. I think this is incredibly urgent. I’m not a doomer, but to me, if these AI models make it into the operations of nuclear systems, we’ll be introducing an incredibly faulty, unreliable system that could lead to catastrophic failures. MOST OVERBLOWN RISK: Extinction As true safety engineers, we don’t engage with that because there are no numbers to back it. This is a very cold way to say it, but when we are talking about how safe is “safe enough,” we’re examining how many people might die. There’s an entire science behind it. Our claims have to be falsifiable and verifiable. You can’t just say, “Humans are going to go extinct in 10 years. Believe me. It might be a 10 percent chance; it might be a 25 percent chance.” How should people think about AI safety? Some of the comparisons that people draw between AI and nuclear are quite frustrating. Despite constantly comparing AI to nuclear, they’re denying standard risk assessment techniques as being sufficient to control AI. I wrote an entire paper on this. Part of this field of risk assessment was born out of the Manhattan Project. Scientists were very concerned, and they developed rigorous standards around systems to limit harm to human life, the environment, and even financial and national security assets. They came up with methods for assessing safety that were ultimately based on answering a democratic question: How safe is safe enough? That is the question that any safety engineer has to ask, and is core to everything that follows. What we’re seeing today, instead, is that large AI labs are renegotiating these safety thresholds through the co-option of terminology, under the pretense that safety instead should be about this unsubstantiated, unscientific, and hypothetical concern. We have AI engineers telling us, no, no, no, that doesn’t matter. We get to determine what AI safety is. We get to draw what those thresholds are. It’s become a religious argument. And I find that very anti-democratic. BIGGEST CONCERN: Cybersecurity In the long run, I’m cautiously optimistic that this technology will allow people to improve their cyberdefenses more rapidly than attackers can exploit systems. But at least for the next few years our infrastructure will not have adopted new defenses. The technology might tilt toward the attackers. We’ve also seen AI deployed recklessly in military use. MOST OVERBLOWN RISK: Runaway intelligence There’s this broad, general notion that AI systems getting smarter is a risk in and of itself, as opposed to the many specific risks we need to worry about. I don’t buy into the notion that overall improvement in AI has an inevitable downstream harm. It’s better to first specify what downstream risks you’re worried about, because the defenses you come up with will be different. For example, cybersecurity has a well-established culture of bug bounties. If someone finds an attack vector or hacks into a system, they get paid a lot of money for disclosing it. In the next few years we might need defenses against biosecurity risks. We should think about screening for what people are allowed to order, like if they’re ordering DNA fragments, so that they’re not being used for harm. How often do your views clash with those of the people around you? Often these views are on two levels: What is going to happen with AI, and what are we going to do about it? On the first question, there’s a lot of disagreement. For example, the coauthors of AI 2027 expect that there will be this incomprehensible, almost indescribable amount of change. Whereas I think the impact will be dramatic, but not quite so dramatic. But on the question of what should be done, most agree there’s a need for transparency from AI companies and that they should face some liability when things go wrong. We also agree on the need for protections for whistle-blowers working from within the AI companies. From my perspective, this is just good industrial policy. Have you made any changes in your life in response to AI risks? I’ve tried to stop using apps where I don’t think developers have put in a lot of time and effort into making the cyberdefenses up to standard. BIGGEST CONCERN: Chemical, biological, and radiological threats The novel biological pathogen that someone releases, the 17-year-old who gets ahold of an extremely capable AI model and launches a cyberattack that takes down the inte