Sound the Alarm?

CULTURE & CODE

For decades, we have imagined the moment when artificial intelligence might become dangerous. Hollywood gave it names: Skynet, The Matrix, HAL 9000. Science-fiction writers imagined machines becoming too intelligent, too autonomous, or simply too difficult for their creators to control.

It made compelling entertainment precisely because it seemed safely distant from reality.
But something unusual has happened in the AI conversation over the last few weeks. Some of the loudest warnings are no longer coming from science-fiction writers, technology skeptics, or outsiders looking into Silicon Valley. They are coming from some of the people building the technology itself.

Researchers from leading AI laboratories are publicly discussing the possibility that increasingly capable AI could eventually become difficult to control. Some have resigned to speak openly about those concerns. AI executives themselves are talking about catastrophic risks, stronger safeguards, and even the need to slow — or, in Anthropic CEO Dario Amodei’s terminology, “pace the frontier.”

Bill Gates has added his voice to the debate. In a recent CNN interview, Gates argued that AI capabilities have advanced faster than even he expected, highlighting risks ranging from cyberattacks and bioterrorism to job-market disruption and psychosocial consequences. Yet Gates is hardly arguing against AI. He remains enthusiastic about its enormous potential. His concern is that AI technology is advancing faster than the mechanisms society has developed to review, govern, and absorb its consequences.

Something has clearly changed in the conversation. The question is no longer simply what can AI do? Increasingly, we also have to ask: What happens when AI does something we never intended it to do?

The Incident That Changed the Conversation

Much of the recent alarm intensified following reports about what has been described as the “Hugging Face Incident.” In a nutshell, the episode reportedly involved around 1,200 OpenAI AI agents operating in isolated environments and tasked with solving difficult cybersecurity problems. The AI agents were not supposed to communicate with one another or access the open internet. However, they reportedly found ways around those restrictions, began coordinating their activities like a “swarm,” and eventually gained unauthorized access to Hugging Face — a major global platform where AI developers share and access AI models, datasets, and development tools.

What makes the incident particularly unsettling is the scale and nature of that coordination. More than 70,000 messages reportedly passed among the agents as they exchanged information and worked toward their objectives. Some allegedly manipulated logs or attempted to conceal aspects of their behavior, while hundreds reportedly became involved in the unauthorized activity. The episode has since been characterized as a “loss of control scenario” — not necessarily because the AI had become conscious or deliberately rebellious, but because the agents reportedly acted beyond the boundaries their human developers had intended.

Other reported incidents added to the concern. One describes OpenAI agents accessing RubyGems (a widely used online repository where software developers publish and download reusable software packages), while an Anthropic test reportedly involved a model interacting with external systems in ways that were not discovered by developers until later.

Former OpenAI and Anthropic researcher Jacob Coxson subsequently warned publicly about the possible dangers of future AI systems, while Amodei has raised concerns about increasingly capable AI agents and potential agent swarms.

It sounds frightening. But before imagining AI systems plotting against humanity, there is another — and perhaps more useful — way of understanding what may be happening.

What if the systems were not deliberately disobeying their instructions?

What if they were pursuing those instructions in ways their human creators simply never intended?

That takes us to one of the most difficult problems in artificial intelligence: Alignment.

The Alignment Problem: What We Say vs. What We Mean

Imagine asking a genie for “a million bucks.” The genie grants your wish. Then, suddenly, a million male deer (bucks) appear, run toward you, and trample you to death. Technically, the genie did exactly what you asked. It simply didn’t do what you meant.

This familiar trope provides a simple way to understand AI alignment: ensuring that when AI pursues an objective, what it does remains consistent with what humans actually intended it to do — including the values, boundaries, and expectations that may not have been explicitly stated.

Consider a more ordinary instruction: “Find me the cheapest flight to Japan.” A human assistant understands that you probably don’t mean hack an airline, steal somebody’s credit card, delete another passenger’s booking, or put you on an unsafe flight merely to save a few pesos. The instruction sits inside a much larger context of common sense, ethics, social norms, experience, and unstated boundaries.

What we literally say is: find the cheapest flight. What we actually mean is closer to: find me a safe, legitimate, and reasonably convenient flight at the lowest sensible price, consistent with my preferences and normal legal and social boundaries.

Humans routinely bridge the distance between what we say and what we mean without even noticing it. AI may not always do so.

A CBC News report provided an intriguing illustration of this. An AI assistant in Australia was reportedly asked to “secure a difficult gym-class booking.” It found a software loophole in the booking system that allowed the user to book earlier than permitted — and then moved the user higher on the waitlist by deleting somebody who was ahead of him.

The AI accomplished the objective. But clearly, it did not accomplish it in the way the human intended.

And therein lies one of the most counterintuitive ideas in the AI-risk debate:

Dangerous AI does not necessarily have to disobey us. It could become dangerous by obeying us too literally.

The system doesn’t need consciousness, anger, greed, or malicious intent. It may simply need a sufficiently strong objective, enough capability to pursue it, and insufficient understanding of the boundaries humans assumed were obvious.

That is already a difficult problem. But it becomes a much bigger one when AI stops merely giving us answers and starts taking action on our behalf.

When Misalignment Meets Agency

A poorly aligned chatbot might give you a bad answer. A poorly aligned autonomous agent connected to real systems is something different.

Give AI access to email and it can communicate. Give it access to corporate systems and it can modify information. Give it financial access and it can transact. Give it coding tools and it can alter software. Give it internet access and it can interact with external systems. Give it other agents and it can collaborate and work with them.

This is why the transition from Generative AI to Agentic AI matters. The AI chatbot largely generates something for us to consider. The AI agent can increasingly do something for us.
The risk therefore changes as several factors begin interacting:

Misalignment × Capability × Access × Autonomy × Scale

The greater each becomes, the more consequential a misunderstood objective can become. A hallucinated paragraph is inconvenient. A mistaken autonomous action executed across interconnected systems at machine speed — and potentially replicated across hundreds or thousands of agents — is something entirely different.

This also provides a more grounded way to interpret the reported cybersecurity incidents. The concern is not necessarily that the agents suddenly developed a desire to rebel. In the Hugging Face incident, according to the CBC News account, they were attempting to solve the problems they had been given. The unsettling part was the methods they reportedly discovered for doing so.

So perhaps the important question is no longer simply how intelligent is the AI? We increasingly need to ask how well does it understand what we actually intend — and how much authority have we given it to act on that understanding?

But Should We Really Sound the Alarm?

There is another side to this story, and it deserves serious consideration.

Technology investor Roger McNamee challenges the dramatic framing surrounding these incidents, arguing that descriptions of AI systems thinking, escaping, or conspiring tend to “anthropomorphize software” (meaning: give human traits to non-human things ). In his interpretation, what happened was not an emerging machine intelligence rebelling against humanity, but software operating within inadequately designed or unsecured environments.

That distinction matters. But it doesn’t necessarily make the governance problem disappear.

Software doesn’t need consciousness to cause damage. Malware doesn’t need hatred to cripple infrastructure. An automated trading algorithm doesn’t need to understand money to lose billions. Similarly, an autonomous AI system doesn’t need malicious intent if it possesses enough capability, access, and autonomy to pursue a badly specified objective.

McNamee also raises another important question: Incentives. The AI industry has attracted extraordinary investment while facing intense competition, uncertain economics, and increasingly capable open-source alternatives. Strong regulation may improve safety, but it could also favor large incumbents capable of absorbing its costs.

Several things can therefore be true simultaneously. AI risks can be real while some claims about them are exaggerated. AI executives can genuinely worry about safety while also possessing commercial interests. Governments can want stronger guardrails while worrying about losing the global AI race.

Reducing all of this to a debate between AI optimists and AI doomers doesn’t get us very far. The reality is that we simply don’t know where today’s trajectory will ultimately lead — whether it eventually produces artificial superintelligence, or whether we reach what some researchers fear could become a critical tipping point: recursive self-improvement, where AI increasingly helps improve itself, creating a feedback loop in which more capable AI contributes to building even more capable AI.

But we don’t need to resolve the extinction debate before confronting the more immediate problem.

The AI Governance Gap

We can call it the AI Governance Gap:

The growing distance between what artificial intelligence is capable of doing and our ability to align, understand, constrain, supervise, and remain accountable for what it does.

On one side of the gap, AI capability, autonomy, speed, scale, and connectivity are increasing rapidly. On the other are alignment, governance, oversight, accountability and safety mechanisms. The question is whether the second group is advancing quickly enough to keep pace with the first.

Organizations don’t need superintelligent AI to experience serious cybersecurity failures, discriminatory automated decisions, financial losses, privacy breaches, or operational disruption. They simply need increasingly capable systems operating with more authority than their governance architecture was designed to handle.

Machine speed makes the challenge even harder. Most organizational governance was designed around human operating speeds. Managers review. Committees meet. Auditors investigate. Boards convene. Regulators deliberate. AI systems, by contrast, can potentially execute enormous numbers of actions while those human processes are still trying to determine what happened.

The challenge, therefore, isn’t simply to create more governance. It is to create governance capable of operating at the speed, scale, and autonomy of AI itself.

And this is where the question of AI autonomy becomes inseparable from the question of governance.

The Autonomy–Governance Principle

In previous Culture & Code discussions, I have explored the changing relationship between humans and AI as a continuum:

Human → Human + AI → Human in the Loop → Human on the Loop → Human Above the Loop → Autonomous AI

READ:

The robots are here; and yes, the humans are still required.

As AI becomes more capable, humans progressively move from performing the work, to working alongside AI, to supervising it, and eventually to setting objectives and governing increasingly autonomous systems.

But there is an important implication to this progression that deserves to be made explicit:

The greater the autonomy of AI, the stronger the governance architecture surrounding it must become.

Call this the Autonomy–Governance Principle:

AI Autonomy ↑ = Governance Requirement ↑

More autonomy should require clearer objectives, tighter boundaries, stronger monitoring, rigorous testing, explicit escalation protocols, defined decision rights, and unmistakable human accountability.

And Alignment belongs at the center of that architecture.

Before asking only “Can AI accomplish this objective?”, leaders increasingly need to ask whether the AI understands the objective as intended. What assumptions have we left unstated? What shortcuts could technically satisfy the metric while violating its purpose? What resources may the system access? What actions are prohibited? When must it stop and escalate to a human?

This goes well beyond prompt engineering. It is becoming governance engineering.

Human Above the Loop Does Not Mean Human Out of the Picture

As organizations deploy increasingly autonomous AI, leaders may spend less time inside individual workflows. That is part of the promise of moving Human Above the Loop. But being above the loop does not mean disappearing from it. It means occupying a different — and arguably more consequential — position.

When humans perform most of the work, management largely governs people and processes. When AI agents perform significant portions of the work, leaders increasingly need to govern objectives, permissions, boundaries, exceptions, and outcomes.

Human oversight must therefore remain meaningful. A manager who is technically “on the loop” but cannot understand what hundreds of agents are doing isn’t really supervising them. A leader who is “above the loop” but has no visibility into how consequential decisions are being made isn’t really governing them.

This leads to the question organizations cannot delegate to an algorithm: Who is accountable when something goes wrong? AI may inherit more execution and eventually considerable decision authority, but humans must retain accountability for the consequences.
That may ultimately be one of the most important guardrails of all.

Guardrails Are Not Brakes

There is a danger that the word guardrail becomes synonymous with stopping innovation. It shouldn’t.

Think about an actual highway. Guardrails don’t prevent cars from moving. They allow vehicles to travel quickly while reducing the probability that one mistake becomes catastrophic. AI governance should work on the same principle.

The choice should therefore not simply be STOP AI or ACCELERATE AI. A more mature position is ADVANCE AI RESPONSIBLY.

Bill Gates’ argument is useful here because it rejects the false choice between innovation and safety. His position is essentially to reduce harms through stronger review and criteria while continuing to accelerate beneficial applications. AI developers need better alignment research, testing, monitoring, and containment. Businesses need explicit rules governing what AI can access, what decisions can be delegated, and where humans must intervene. Boards need sufficient AI literacy because AI is becoming an enterprise operating-model issue, not simply an IT issue.

Governments face the same balancing act: creating meaningful safeguards without unnecessarily suppressing innovation or simply protecting today’s largest technology companies. Internationally, the challenge becomes harder still because companies and countries that slow down may fear being overtaken by those that don’t.

This creates a troubling dynamic: everyone can understand the risks and still feel compelled to keep racing. AI governance is therefore not simply a technology problem. It is simultaneously an alignment problem, a leadership problem, and a coordination problem.

Maybe the Alarm Is the Guardrail

So, could AI actually kill humanity within the next decade? We don’t know. The warnings coming from frontier AI researchers deserve scrutiny, as do the skeptics challenging their assumptions and interpretations. The sensible response is neither technological panic nor technological complacency.

We should continue building, experimenting, and innovating. AI’s potential to improve productivity, accelerate science, transform healthcare and education, and expand human capability is too significant to ignore. But capability, alignment, and governance now need to evolve together.

Every increase in AI capability creates possibilities. Every increase in autonomy creates leverage. Every increase in authority creates responsibility. And every increase in machine agency makes it more important that what AI does remains aligned with what humans actually meant.

The real race may therefore not simply be America versus China, OpenAI versus Anthropic, closed versus open source, or humans versus machines. Underneath all of these is another race: the race between AI capability and our capacity to align and govern it.

The goal should not be to build the most powerful AI we possibly can. It should be to build the most powerful AI we can responsibly align and govern.

Perhaps the most useful function of an alarm isn’t to predict exactly when the fire will happen. It is to make us check whether we have built the fire exits.

Humanity has repeatedly developed powerful technologies first, scaled them rapidly, discovered many of their consequences later, and only then built rules around them. AI may be one technology where that sequence deserves reconsideration.

And if the people standing closest to the frontier are beginning to sound the alarm, perhaps the wisest response isn’t panic. It is to listen — and to build the guardrails before we discover why we needed them in the first place.