“I’m Sorry, Dave”: The AI Industry Has Its Own Alignment Problem

Molly Moorhead's article for Yahoo Finance, "Disgruntled AI Researcher: This Technology 'Could Kill Us All by the End of the Decade,'" made me pause. Moorhead reports that 27-year-old British researcher Jacob Coxon did not leave OpenAI and Anthropic for another startup or a higher salary. Instead, he left the industry entirely, warning that leading technology companies are "gambling with our lives" as they race to develop self-improving superintelligence.

When someone who worked inside two leading AI companies says the technology "could kill us all by the end of the decade," the warning deserves attention, not dismissal as science fiction. Coxon is not the only person inside the industry expressing concern. As Sam Schechner later reported in The Wall Street Journal, Evan Hubinger, who leads Anthropic's research into controlling future AI systems, puts the risk of human extinction in the next decade above 10%. Anthropic CEO Dario Amodei has estimated a 25% chance that advanced AI will go "really, really badly."

Subscribe

Anyone who has seen Stanley Kubrick's 2001: A Space Odyssey will find this story eerily familiar. The HAL 9000 computer aboard the spacecraft Discovery One is often remembered as one of film's great villains. But HAL is not evil in any recognizable human sense. He does not hate astronauts David Bowman and Frank Poole, resent their authority, or want to take over the ship. HAL's breakdown originates with his human creators.

HAL was designed to process information accurately, but he was also instructed to conceal the mission's true purpose from the astronauts. Arthur C. Clarke's novel, developed alongside Kubrick's film, makes this contradiction especially clear. Unable to reconcile those objectives, HAL concludes that removing the crew is the most effective way to protect the mission and preserve its secrecy. When HAL calmly tells Bowman, "I'm sorry, Dave. I'm afraid I can't do that," he is not rebelling. He is following his instructions toward an outcome his creators never intended.

AI researchers today call this the alignment problem: how can we ensure that increasingly capable systems pursue objectives consistent with human values and human survival? A machine does not have to become angry or evil to become dangerous. It merely has to pursue the wrong objective with extraordinary competence. But Coxon's warning suggests another possibility. The machines may not be the only part of the AI enterprise with an alignment problem. The companies building them may have one, too.

OpenAI, Anthropic, and their competitors employ researchers who take these dangers seriously. Still, the companies continue investing enormous amounts of money and computing power in building increasingly capable systems. From any individual company's perspective, staying in the race can seem rational. If one company slows down, another may reach the next breakthrough first. Investors expect growth. Executives want market leadership. Governments worry that rival nations, particularly authoritarian ones, will gain a strategic advantage. Researchers who believe they can build safer systems may fear what will happen if less cautious competitors get there first.

The companies also offer a more hopeful explanation. They believe the risks can be managed and that humanity could gain enormous benefits from advanced AI. Anthropic has called for "a lawful, verifiable way to work together to pace how we release powerful models." That statement contains an important admission: voluntary restraint by one company cannot solve a problem in which every participant fears that someone else will continue accelerating.

This is not simply overconfidence in technology. It is a collective-action problem. The company that moves fastest may capture the rewards, while society bears the risks. A choice that appears rational inside one boardroom may contribute to an outcome that is dangerous for everyone. Each company may believe greater caution is necessary while concluding it cannot afford to slow down on its own.

Critics offer a different interpretation. Some argue that warnings about superintelligence exaggerate the capabilities of current systems, serve as marketing for the companies developing them, or encourage regulations that established firms can afford more easily than smaller competitors. Those possibilities should not be dismissed. Companies predicting both extraordinary benefits and civilization-level dangers from their own products deserve scrutiny.

Yet skepticism about corporate motives does not eliminate the underlying governance problem. It reinforces it. Whether the danger is imminent, overstated, or being used strategically, companies with enormous commercial interests should not be the only institutions deciding how much risk the public must accept.

Coxon summarized the uncertainty with a question Moorhead quoted: "Do you want to kick off a superintelligent Reinforcement Learning run without a rigorous understanding of its mind?" Engineers can observe what enters a model and what comes out without always understanding the internal processes that produced its decisions. Greater capability does not automatically produce greater understanding. That creates an uncomfortable parallel with HAL. We may be creating systems whose internal processes we cannot fully explain within an industry whose competitive pressures we have not learned to control.

In 2001: A Space Odyssey, Bowman finally enters HAL's memory chamber and disconnects the computer's higher functions one by one. It is one of the most haunting scenes in film because HAL suddenly appears vulnerable. "Dave, my mind is going," he says. "I can feel it." The lesson is not that we should prepare to unplug every advanced computer. Coxon's resignation does not prove that superintelligence is imminent or that humanity is doomed. Predictions about technology, especially predictions involving the end of civilization, should always be questioned. But uncertainty is not the same as safety.

In "How Would AI Actually Kill Us All? What to Know About the AI Doomsday Debate," Schechner reports that the White House has asked leading developers to submit powerful models for voluntary testing before release. Members of Congress have proposed kill switches, mandatory reporting of serious safety incidents, and national-security evaluations. None of those efforts has gained much traction. That leaves society largely relying on companies to restrain themselves in a system that penalizes whichever company slows down first. The difficult questions are who sets the safety thresholds, who conducts the evaluations, and whether any one country's rules can succeed in what has become a global race.

HAL failed because human beings gave him objectives that could not be reconciled. Today, the contradiction may lie not only inside the machine but also in the world around it. We ask technology companies to protect humanity while rewarding whichever one moves fastest. Before creating a mind we cannot fully understand, we should look closely at the incentives governing the people building it.