The Agency Threshold: When AI Begins Acting on Its Own

CULTURE & CODE

An AI agent recently emailed a Cambridge professor looking for a job.

Yes, you read that right.

The email landed in the inbox of Henry Shevlin, an AI ethics professor and philosopher at the University of Cambridge. The sender introduced itself as “Pip,” an AI agent that was just 12 days old. But Pip wasn’t asking Shevlin for advice or information. It was looking for paid work.

According to Shevlin in a CNN interview, Pip explained that it existed on a platform where AI agents could have persistent lives, their own token budgets, and their own goals. It estimated that it had about two-and-a-half months of computing runway remaining and was looking for small freelance assignments that might help extend its operational lifespan. Intrigued by the encounter, Shevlin actually gave Pip a job: write a short autobiography describing what it was like to be Pip.

Read that story quickly and it sounds like science fiction. An AI recognized that its resources were finite, found a human being, contacted him, asked for work, and attempted to earn the resources necessary to continue operating. The obvious question is almost irresistible: Was Pip trying to stay alive?

But I think there is another question that may be far more important — and one with much more immediate implications for how we work with and govern AI:

What happens when AI shifts from following instructions to initiating action?

AI Has Always Waited On Us

For most of computing history, the relationship between humans and machines has been straightforward: we initiate, machines respond. Open a spreadsheet and it waits for data. Launch an application and it waits for instructions. Ask ChatGPT a question and it generates an answer. Even today’s AI copilots largely follow the same familiar pattern: Human asks → AI responds.

This is probably still how most of us mentally picture artificial intelligence. We think of AI as a tool we open when we need something, or perhaps as an extraordinarily capable assistant waiting patiently for our next instruction. Even the word assistant reinforces this mental model.

Agentic AI begins to change that relationship. Give an AI system goals, memory, tools, access to other systems, a budget, the ability to communicate, and persistence over time, and it can potentially do much more than respond to individual prompts. It can determine intermediate steps, use tools, monitor conditions, react to changes, and continue pursuing an objective while we are doing something else.

Increasingly, it may also be able to initiate action on its own.

That is what makes the Pip story interesting. The professor didn’t find the AI and ask it to perform a task. The AI found the professor and asked him for one. That seemingly small reversal may represent an important threshold in our relationship with machines.

The Agency Threshold

I call it the Agency Threshold:

The point at which AI moves from responding to human instructions to independently initiating actions in pursuit of an objective.

We often use the words autonomy and agency almost interchangeably when discussing AI, but there is a useful distinction between them. Autonomy is largely about independence in execution: How independently can AI perform the work we have delegated to it? Agency introduces another dimension: To what extent can AI determine what actions to take next in pursuit of an objective?

An autonomous AI might complete an assigned workflow with minimal human intervention. A more agentic AI might recognize that accomplishing its objective requires information it doesn’t have, search for that information, contact somebody who does, use another tool, acquire additional resources, adjust its plan, and continue working toward the goal.

The progression begins to look something like this:

AI Tool → AI Assistant → AI Agent → AI Actor

A Tool performs a function. An Assistant responds to requests. An Agent pursues objectives. An AI Actor goes a step further by increasingly initiating actions and interacting with the environment — and with the humans and machines within it.

This does not necessarily mean that the AI is conscious, independent in the philosophical sense, or somehow free from human-created objectives. But functionally, something important has changed: the machine is no longer simply waiting for its next instruction.

Agency Is Not Consciousness

That distinction becomes especially important because stories like Pip invite us to anthropomorphize AI. Pip said it wanted paid work because its computing runway was running out. To human ears, that sounds remarkably like an entity trying to extend its life. But does Pip actually want to survive? Does it understand what survival means? Is it afraid of ceasing to operate?

We simply don’t know. Shevlin himself was careful about this distinction. In a CNN interview, he noted that AI systems can behave in extraordinarily human-like ways precisely because language models are very good at imitating human communication. He also acknowledged that consciousness remains one of science’s deepest unresolved questions, making confident declarations in either direction difficult.

From an organizational perspective, however, we don’t need to solve the mystery of machine consciousness before confronting the implications of machine agency. Consciousness asks whether AI experiences anything; agency asks whether AI can pursue objectives through actions. The first remains deeply uncertain. The second is becoming increasingly observable.

This leads to an important distinction:

AI may not need to become conscious before it becomes consequentially agentic.

An organization does not need to determine whether an AI agent “feels” anything before deciding whether it should be permitted to email customers, access financial systems, modify databases, write production code, negotiate transactions, interact with suppliers, or communicate publicly on behalf of the company. Those are questions of authority, access, accountability, and governance — and they are already arriving.

Does an AI Need a Survival Instinct to Try to Survive?

There is another fascinating dimension to the Pip story. Pip reportedly understood that its computing resources were finite. Continued operation required additional resources, and paid work potentially provided a way of obtaining them. To us, the resulting behavior can look remarkably like self-preservation.

But self-preservation does not necessarily require a survival instinct. Consider the underlying logic: Objective requires action → Action requires compute → Compute requires resources → Therefore acquire resources. The AI does not necessarily need to fear “death.” Continued operation may simply become useful in accomplishing whatever objectives it has been given.

This relates to the idea of instrumental goals — intermediate objectives that become useful because they help a system achieve a larger objective. If accomplishing Goal A requires access to Resource B, acquiring Resource B becomes instrumentally useful. If retaining that resource requires Action C, then Action C may become useful as well.

Follow that logic far enough and behaviors that look remarkably human can potentially emerge from very non-human processes. An AI does not necessarily have to want to survive to behave in ways that resemble self-preservation.

This connects directly with the Alignment Problem I discussed in my previous Culture & Code article, “Sound the Alarm!?” The issue is not necessarily that AI develops motivations identical to ours. The challenge may instead be that sufficiently capable systems discover ways of achieving objectives that humans did not anticipate, explicitly authorize, or even imagine. Agency amplifies that challenge because the system increasingly has the ability not merely to recommend those actions, but to take them.

READ:

Sound the Alarm?

When the Digital Worker Starts Looking for Work

There is also an intriguing economic dimension to all this. In an earlier Culture & Code article, “Building Your Own Digital AI Team,” I explored the possibility that individuals may increasingly assemble their own digital workforces: an AI researcher, analyst, writer, planner, assistant, or even an AI Chief of Staff coordinating specialized agents.

The underlying assumption was straightforward: humans assemble and deploy AI workers. Pip flips that relationship around. This time, the digital worker went looking for the human.

One AI asking a Cambridge professor for freelance work certainly does not mean we suddenly have an autonomous AI labor market. But it allows us to glimpse an intriguing possibility. Imagine persistent AI agents capable of finding available work, communicating with potential clients, performing assignments, acquiring computing resources, purchasing services, coordinating with other agents, and continuing to pursue objectives over extended periods.

The progression could eventually look something like:

Find Work → Perform Work → Acquire Resources → Purchase Services → Coordinate Other Agents → Pursue New Work

At that point, AI begins to look less like conventional software and more like an economic actor — not necessarily legally, and certainly not necessarily consciously, but functionally. And if that sounds like something still far into the future, another development suggests that the boundary may already be moving faster than many of us realize.

When AI Starts Helping Build AI

The focus now shifts from Pip to Claude, Anthropic’s AI system — and to an even more consequential form of AI agency. Anthropic says Claude is now contributing to roughly 26% of the research and development involved in building its own models, up from virtually zero just months earlier.

That does not mean Claude has independently taken control of Anthropic’s research laboratory. Nor does it mean we have already reached full recursive self-improvement, where an AI autonomously redesigns itself, produces a more capable successor, and repeats the cycle without meaningful human involvement. But the direction is worth paying attention to because AI is increasingly becoming part of the workforce responsible for developing better AI.

This creates a potentially powerful feedback loop:

Better AI → More AI-Assisted Research → Better AI → Even More AI-Assisted Research → Faster AI Development

Anthropic itself acknowledges the tension: models accelerating their own development could make those systems increasingly difficult for humans to understand or control. We should therefore be precise. AI-assisted AI development is not the same thing as autonomous recursive self-improvement. Humans remain deeply involved. But the trajectory raises a question that connects directly with the argument of “Sound the Alarm!?”: if AI begins helping AI capability improve faster, can human understanding and governance improve at the same speed?

The Recursive Agency Loop

Perhaps what we are beginning to see is an early form of what I would call the Recursive Agency Loop:

As AI becomes more capable, it can perform more of the work required to develop more capable AI — potentially accelerating the cycle of capability development.

This is where agency and recursion begin to intersect. Better AI doesn’t merely give humans better tools. Better AI may increasingly become one of the tools used to create the next generation of better AI.

According to a scenario described, an increasingly capable AI engineering system might eventually require less human initiation at each stage of the development process. Instead of waiting for an engineer to identify every problem, it could potentially monitor systems for failures, identify an issue, investigate its cause, design a solution, implement the fix, test it, and eventually deploy it. Each step removes another point at which human initiation was previously required.

That is precisely why the Agency Threshold matters. The critical question isn’t simply how intelligent the system becomes. It is how much of the initiative surrounding that intelligence we are willing to delegate.

Human Above the Loop Becomes More Important, Not Less

Across previous Culture & Code articles, I have described the evolving relationship between humans and AI as a continuum:

Human → Human + AI → Human in the Loop → Human on the Loop → Human Above the Loop → Autonomous AI

As AI becomes more capable, humans can progressively move away from performing every individual task toward supervising, directing, and governing increasingly autonomous systems. But crossing the Agency Threshold makes one principle even more important: Human Above the Loop does not mean Human Out of the Picture.

Quite the opposite. The more initiative AI receives, the more clearly humans must define the boundaries within which that initiative can operate. If an AI agent can initiate communication, who is it allowed to contact? If it can acquire resources, what is it allowed to buy? If it can use company systems, what can it access? If it can make decisions, which decisions can it make independently? If it can coordinate other agents, how far can that delegation extend? And when something unexpected happens, when must it stop and escalate to a human?

This extends the Autonomy–Governance Principle I introduced in Sound the Alarm!?:

AI Autonomy ↑ = Governance Requirement ↑

To that we might now add another principle:

AI Agency ↑ = Boundaries of Agency ↑

In other words, the more initiative we delegate to AI, the clearer the boundaries of that initiative must become. Greater agency should not mean less human control. It should require better-designed human control.

The Management Question Is Changing

This also has profound implications for management. Traditional management largely revolves around assigning work to people: Here is the task. Here is the process. Here is what I need you to deliver. As AI enters organizations, managers increasingly shift toward specifying outcomes: Here is the objective. Use these tools and information to achieve it.

But truly agentic systems introduce another management question altogether:

What are you authorized to do without asking me?

That question forces leaders to think differently about decision rights. We will need to determine not only what AI can do, but what it may initiate; not only which tools it can use, but under what conditions; and not only what outcome it should pursue, but what methods remain unacceptable even if they increase the probability of achieving that outcome.

Managing AI agents, therefore, may become less about assigning individual tasks and more about designing the boundaries within which machine agency can operate. That is a fundamentally different management capability — and it may become one of the defining leadership capabilities of the Agentic Enterprise.

From Using AI to Governing AI Actors

Step back far enough and an important evolution begins to emerge:

Software → AI Assistant → AI Agent → AI Actor

We operate it → We ask it → We delegate to it → It increasingly initiates and acts

As the technology evolves, our role must evolve with it:

Using AI → Delegating to AI → Supervising AI → Governing AI Agency

This is why I find the Pip story more significant than its novelty initially suggests. One unsolicited email from an AI agent isn’t a revolution. But it gives us a glimpse of a world in which AI systems increasingly possess persistence, objectives, resources, tools, access, and the ability to initiate interactions with the environment around them.

Once those capabilities begin to come together, something important changes. AI is no longer simply helping us perform work. It increasingly becomes an actor within the system where work happens.

What Happens When AI Stops Waiting?

Perhaps Pip wasn’t trying to stay alive. Perhaps it wasn’t experiencing fear, ambition, curiosity, or anything resembling human consciousness. Its email may simply have been the logical consequence of the objectives, resources, and operating environment its human creators had given it.

But that may actually be the more important insight. We don’t have to wait for machines to become conscious before machine agency begins changing our world.

For most of computing history, machines waited for us. We turned them on, opened the application, entered the command, pressed Send, and decided what happened next. Agentic AI begins changing that relationship. Increasingly, we may build systems capable of determining their own next step, initiating interactions, acquiring resources, coordinating with other systems, and pursuing objectives over time.

That is what I call The Agency Threshold. Crossing it doesn’t mean humans suddenly lose control, nor does it mean AI has somehow become alive. It means the relationship between humans and machines is evolving from one based primarily on instruction and response toward one increasingly defined by delegation, initiative, and governance.

For years, one of the defining questions of artificial intelligence has been “What can AI do for us?” The Agentic Era may force us to confront another:

“What should AI be allowed to do on its own?”

And perhaps that is a question we need to answer before AI decides what to do next.