As I have discussed previously, the global conversation about AI safety is focused almost entirely on the wrong problem. Not because alignment doesn’t matter – it does – but because alignment is an interior question being asked in place of an exterior one, and the substitution is not accidental.
Alignment asks: will the AI do what we want it to do? Will it hallucinate? Will it be biased? Will it follow instructions? These are important questions about tool calibration. They assume the institution holding the tool is legitimate, properly constrained, and accountable. They ask whether the instrument is well-tuned.
The structural question is mute: is it acceptable for any private institution to hold this kind of power at all, and if so, under what constraints that don’t depend on the goodwill of the people inside it?
The distinction matters because the alignment frame lets everyone – companies, regulators, the public – feel like the problem is being worked on. It provides the comforting narrative that smart engineers are making the AI safe, and that the remaining challenge is technical. It positions the human in the loop as the safety mechanism.
But humans are not a safety mechanism. Humans are the risk.
The Self-Destruction Problem
This is the part that most AI governance discourse refuses to engage with directly, so let me be explicit: the assumption that human oversight is the safeguard against AI misuse requires the assumption that human overseers will consistently act in the interest of the populations they affect. Everything we know about human behavior under conditions of concentrated power tells us this assumption is false.
This is not a moral judgment about individuals. It is an empirical observation with thousands of years of supporting evidence.
We are a species that invented nuclear weapons and then built enough of them to destroy civilization several times over. We are a species that, knowing the consequences of carbon emissions for decades, accelerated them. We are a species that built global financial systems so fragile that a handful of institutions could trigger worldwide economic collapse, and then, after it happened, rebuilt the same systems with minor modifications. We are a species currently watching the only nuclear arms control treaty between the world’s two largest arsenals expire without replacement, while the administration that let it lapse simultaneously threatens to resume nuclear testing.
The “human in the loop” is not a failsafe. The human in the loop is the entity most likely to override safety constraints under pressure, rationalize short-term decisions with long-term consequences, and prioritize self-preservation or institutional survival over the welfare of people affected by their choices. Putting a human in the loop of an AI system and calling that a safety measure is like installing a smoke detector that is also an arsonist. It may work most of the time. The failure mode is catastrophic.
When I wrote previously about the alignment problem and its misplaced focus, the argument was that the human capability framing, the idea that we need to make AI safe for humans, is the wrong orientation. The deeper problem is that we need governance structures that account for what humans do with power, because the historical record on that question is unambiguous and it is not flattering.
Acton’s Law as Engineering Principle
“Power tends to corrupt; absolute power corrupts absolutely.” Lord Acton wrote this in 1887. It is treated as a moral aphorism, something people quote at dinner parties. It is actually an engineering principle with more empirical support than most things we build critical infrastructure around.
Every functioning democracy on earth embeds this principle in its design. Separation of powers. Term limits. Judicial review. Constitutional constraints. Freedom of the press. Independent auditors. None of these exist because democratic theorists assumed leaders would be bad people. They exist because the assumption that leaders will remain good under sustained pressure is the single most dangerous assumption in governance design. The entire architecture of democratic accountability is built on the premise that character is not a durable safeguard.
Now apply this to AI.
Anthropic has a public benefit corporation structure and a Long-Term Benefit Trust. These are thoughtful governance mechanisms. They are also the only structural constraints between an autonomous cyber-offense capability and the pressures of $19 billion a year in burn rate, 230+ investors across multiple sovereign jurisdictions, sovereign wealth funds from Singapore, the UAE, and Qatar on the cap table, and an unpredictable U.S. administration that has already demonstrated its willingness to override institutional norms when it suits its purposes.
OpenAI had a nonprofit board that was explicitly designed to constrain commercial incentives and ensure the safety mission remained primary. In November 2023, we watched that structure fail in real time. The board attempted to exercise its oversight function, the thing it was literally designed to do, by removing CEO Sam Altman over concerns about candor, safety processes, and what board member Helen Toner later described as a pattern of withholding information and, in some cases, outright lying. Two executives had reported what they called “psychological abuse.” The board’s concerns were, by any governance standard, serious.
Within five days, Altman was reinstated. The board members who had voted to remove him were themselves removed. Microsoft, which had invested billions and was given one minute’s notice of the firing, applied massive pressure. Ninety-seven percent of employees threatened to quit. Investors mobilized. The governance structure that was explicitly built to prioritize safety over commercial interests folded in less than a week under the combined weight of capital, talent, and commercial momentum.
This is what happens when a governance structure designed to constrain power encounters the actual forces that power generates. The structure failed. Not because the people were wrong to try, but because the structural design was inadequate to the forces arrayed against it. Acton’s law is not about character. It is about the physics of power. It’s about the predictable deformation of institutions under pressure that no individual’s virtue can resist indefinitely.
The recent New Yorker profile of Altman raises further questions about how individuals handle the psychology of extreme power. But the profile matters less than the structural lesson: it does not matter whether the person at the center is a saint or a sociopath. The institution will deform under pressure regardless, unless the constraints are structural rather than characterological.
The Good Guy / Bad Guy Trap
We have largely exercised restraint with nuclear weapons, after seeing the impact in Hiroshima and Nagasaki, and in the Bikini Atoll where so much damage was done in testing. But this is largely because it was a power standoff. It was deterrence. It was the full awareness that nuclear war could cause civilizational collapse, the end of humanity, unfathomable misery to every live species on the planet. cultural framing becomes actively dangerous.
The emerging public narrative positions the AI narrative as a story about good companies and bad companies, much as the nuclear conversation unfolded. Anthropic is the safety-focused lab. OpenAI is the reckless commercializer. Dario Amodei is the careful steward. Sam Altman is the power-hungry disruptor. Pick your hero, pick your villain, and the problem becomes manageable, just make sure the good guys win.
This is exactly wrong. Not because the characterizations are inaccurate (they may or may not be), but because the framing implies that the solution is personnel when the problem is structure. If the “good” organization holds unchecked power, we’re told to relax. Acton’s point is that this is the moment when we cannot relax.
The good guy/bad guy frame does three things, all of them harmful. First, it lets people believe that the problem is solved if the right person is in charge, which history tells us is never durably true. Second, it deflects attention from the absence of structural constraints by substituting a narrative about character. Third, it makes criticism of the “good” actor feel disloyal or churlish. If Anthropic is doing its best, who are you to question whether their best is sufficient? It’s very difficult for us to escape good guy bad guy framing. Joseph Campbell drew on this and Jungian thinking, writing about the hero’s journey – and our minds need to go to a counterbalance. René Girard and mimetic theory posit the idea that communities need a scapegoat to maintain cohesion – if there’s a bad there must be a good and if there’s a good there must be a bad. This is an unfortunately unsubtle aspect of how we frame our stories and what we seek in narrative. Try to find an exception. It will be difficult to identify this dynamic lacking in any story that you’ve ever read, in any human tale that’s ever been told.
I am not questioning Anthropic’s intentions. I am questioning whether any private institution, regardless of intentions, should hold power of this magnitude without structural accountability that doesn’t depend on the continued goodwill of the people inside it. The answer from 2,400 years of political philosophy, from Plato through Montesquieu through the framers of every democratic constitution, is no.
The Cultural Rehearsal
Popular culture has been processing this exact question at massive scale for nearly two decades, and the trajectory of that processing tells us something important about where the collective unconscious has arrived, even if policy hasn’t followed.
The dominance of superhero narratives in film and television over the past fifteen to twenty years is not coincidental. It maps almost exactly onto the period when technology began concentrating power in individual actors and small groups at a scale that previously required nation-states. The Marvel Cinematic Universe launched in 2008, the same year the iPhone App Store opened and Facebook hit 100 million users, the year the infrastructure for platform power became visible. The peak of superhero saturation in popular culture coincides with the rise of the tech billionaire as a figure who genuinely holds outsized power over daily life.
And the fiction has been getting darker in lockstep.
The early MCU was optimistic. Tony Stark is a billionaire weapons manufacturer who chooses to be a hero, and the narrative rewards him. Power is personal, and personal virtue is sufficient. By the time we reach Captain America: Civil War, the narrative has fractured: the film is literally about whether superheroes need institutional oversight, and the Sokovia Accords are a fictional governance framework for unchecked private power. The film cannot resolve whether they are right, because the real world cannot either.
The Boys, which debuted in 2019 and just released the first two episodes of its final season, completes the inversion. Also based on a graphic novel, it’s superheroes are corporate products. Their heroism is a brand managed by Vought International. The question is not whether they will save you but whether the company that owns them will let them, and what the company’s real interests are. Power is institutional, not personal, and the institution is corrupt not because the people in it are monsters (though some are), but because the structure incentivizes corruption. This is Acton’s law dramatized as satire.
Alan Moore arrived at this destination in 1986 with Watchmen, which addressed the guardianship problem with a precision that political philosophy rarely achieves in fictional form. Dr. Manhattan is the closest analogue to what we now confront: a being whose mere existence reshapes geopolitics, regardless of his intentions or actions. He does not choose to destroy the world. He withdraws, exercising a restraint that humans have historically been loath to demonstrate. But his existence creates the vacuum that Ozymandias exploits to stage a mass-casualty event in the name of preventing a worse one. The power’s existence is the crisis. The character of the person holding it is, at best, a modulating factor. Much of the point of the watchmen is that the watchmen themselves are extremely flawed characters. They all have aspects of evil of them that they have fallen victim to, that they have allowed to be exploited, that they have exploited themselves.
But here is the update to the framework that Moore couldn’t have anticipated: the question is no longer quis custodiet ipsos custodes: who watches the watchmen.
There is no designated guardian class. No one was appointed to hold this power. No one was elected, vetted, trained, or given a mandate. A private company in San Francisco built something in the course of doing R&D, and now holds capabilities that can autonomously map and exploit the digital infrastructure of civilization. The power didn’t arrive through delegation. It arrived through emergence. And the entire governance tradition that asks “who watches the powerful” assumes the powerful were designated, which means they can be undesignated. What do you do when the power simply appeared inside a private institution that nobody appointed and nobody can recall? In ancient Rome Senators were forbidden from engaging in certain commercial activities. The Lex Claudia (218 BC) prohibited senators from owning large trading ships, to prevent conflicts of interest between commercial and governance roles and had a number of other operational structures that tried to ensure that they would not be easily corrupted. We saw this collapse very famously with Julius Caesar, who attempted to centralize power and was assassinated by his peers who saw this overreach as dangerous. Every human body of governance is subject only to the corruption of humans knows this visible more right now than the United States.
What Would Actual Governance Look Like?
I am an analyst, not a legislator. I am not going to draft model legislation from a hotel room in Siem Reap without a specific scenario. But I can identify structural principles that any serious governance framework would need to embody, based on what we know about power, institutions, and the historical record.
Separation of capability from control. The entity that builds the most powerful model should not be the sole entity that decides how it is used. This is separation of powers applied to AI, the principle that no single institution should hold all functions. Build, deploy, audit, and constrain should be structurally separate roles.
Mandatory transparency at capability thresholds. When a model crosses defined capability lines; say, the ability to autonomously find zero-day vulnerabilities in major operating systems, that fact should become public, not proprietary. The threshold itself should be defined through a process that includes democratic input, not solely by the company that crossed it.
Accountability that doesn’t depend on circumstance or character. Any governance structure that relies on the continued goodwill of the people inside it is, by Acton’s law, a structure that will fail under sufficient pressure. The constraints must be architectural, embedded in law, treaty, institutional design, not characterological.
International coordination that doesn’t depend on any single state. The NPT model is failing because it depended on the good faith of the states that created it, and that good faith has evaporated. Whatever replaces it for AI cannot be anchored to a single jurisdiction or a single geopolitical bloc. It must be structurally resilient to the defection of its most powerful members, which is the hardest design problem in international governance and the one we most need to solve.
Democratic input into deployment decisions with population-level impact. When a capability affects the security of every computer system on earth, the decisions about how to deploy it cannot be made solely by a corporate board, a trust document, and a partner list. There must be a mechanism through which the populations affected have standing — not as consumers, but as citizens.
The Clock
None of these principles are novel. Democratic theorists have articulated them for centuries. The problem is not that we don’t know what governance looks like. The problem is that nobody is building it at the speed the situation requires.
Anthropic says the window for defensive advantage with Mythos-class capabilities is measured in months. The regulatory conversation (the EU AI Act, the stalled Canadian AIDA, the absent U.S. federal framework) is measured in years. The gap between the speed of capability development and the speed of governance development is not closing. It is widening.
Meanwhile, the cultural preparation has been happening for decades. Audiences have spent twenty years absorbing narratives that move from “the powerful will be good” to “power corrupts institutions” to “power at this scale cannot be contained.” The public’s narrative intuition is, in many ways, ahead of policy.
The alignment conversation is asking whether the AI will behave. The question that matters is whether the power structure will behave. And the answer, from theory, from history, from the empirical record of every institution that has ever held unchecked power, is: not without structural constraints that we have not yet built and are not yet building.
The watchmen aren’t watching. There are no watchmen. There’s a company in San Francisco with a trust document. And a clock that is running out.
Jen Evans is the founder of Pattern Pulse AI. Her ongoing research series “Whose AI Runs the Government?” examines AI sovereignty dependency chains, particularly in Canadian federal and provincial infrastructure. She is the originator of Evans’ Law, the Nudgment framework, and AI Conversational Phenomenology.