Who Is Responsible When an AI Agent Hacks?

How NIST, Microsoft Entra, and modern IAM frameworks are tackling non-human identity and delegated authority before they become an accountability problem.

Imagine an AI agent working inside a large company.

It can read documents, access databases, investigate incidents, create tickets, call APIs, write code and perhaps even change a production configuration.

One morning, something goes wrong.

The agent accesses a system it wasn't supposed to access.

It retrieves sensitive information.

It changes a firewall rule.

It creates another agent with excessive permissions.

Or it follows a malicious instruction hidden inside a document and takes an action nobody intended.

The first question will probably be:

What did the agent do?

But there is another question that may become just as important:

Which agent did it?

And then comes the uncomfortable one:

Who is responsible?

We have spent decades building identity systems for people, applications, services and devices.

Agentic AI introduces another kind of actor: software that can interpret a goal, make decisions, use tools and take actions with limited human intervention.

If we are going to give these systems the ability to act, we need to know who — or rather, what — is acting.

An agent doesn't have to be malicious to become a security problem

When we hear the word hacking, we usually imagine a person sitting behind a computer trying to break into a system.

Agentic AI changes the picture.

An AI agent may already have legitimate access to databases, APIs, source code, cloud infrastructure and internal documents.

It doesn't necessarily need to "break in."

It may simply be tricked into using the access it already has in an unintended way.

Imagine Alex asks an AI agent:

"Read the incident reports and prepare a summary."

One of those documents contains instructions designed to manipulate the agent.

The agent follows those instructions.

It retrieves information from another system and sends the result somewhere it shouldn't.

No password was stolen.

No firewall was bypassed.

The agent used its legitimate identity and permissions.

The problem was that the agent was persuaded to misuse its authority.

This is one of the reasons agentic systems create a different security problem from traditional applications.

The agent may already have the keys

Traditional software generally follows predefined instructions.

An AI agent can interpret goals and dynamically decide which tools to use.

Microsoft's current AI agent security guidance describes agents as autonomous systems that can plan, use tools, maintain state and interact with other agents. It also treats the path from user → agent → tools → other agents as a trust boundary.

That changes the security question.

Instead of asking only:

"Can someone hack the system?"

we also need to ask:

"What happens if the agent itself is manipulated?"

An agent might receive a malicious instruction through a document.

A tool it uses might be compromised.

Another agent might provide misleading information.

A user might deliberately ask it to perform something dangerous.

Or the agent might simply have been granted more permissions than it actually needs.

The agent doesn't necessarily need to be malicious.

It only needs to be capable, connected and insufficiently constrained.

This is where identity becomes important

NIST's work on agentic AI identity makes an important argument: agents should be treated as "first-class entities" with their own identifiers, credentials and entitlements rather than simply borrowing a person's credentials.

That is an important distinction.

The answer to agent identity shouldn't be:

Give the agent my username and password.

It should be closer to:

Give the agent its own identity and explicitly delegate the authority it needs.

That way, we can distinguish between Alex acting and an agent acting on Alex's behalf.

Consider two audit records.

The first says:

Alex accessed customer database.

But Alex didn't.

An AI agent acting on Alex's behalf did.

The second says:

Agent-8472 accessed customer database on behalf of Alex.

That is a completely different security picture.

Now we know which agent acted, who it was acting for and which identity was involved.

Identity doesn't prevent an attack.

But without identity, investigating the attack becomes much harder.

But identity is only the beginning

Suppose an organization has 10,000 agents.

Who creates their identities?

Who gives them permissions?

Who reviews those permissions?

Who removes them when they are no longer needed?

And most importantly:

Who is accountable for them?

Creating thousands of identities manually obviously doesn't scale.

Identity has to become part of the agent lifecycle.

An agent might move through something like:

Create → Identify → Authenticate → Authorize → Act → Audit → Revoke

Some agents might exist for months.

Others might exist for only a few minutes to complete a task.

NIST's work on software and AI agent identity and authorization explicitly addresses identification, authorization, auditing and non-repudiation.

This is where traditional identity and access management becomes foundational to the agentic world.

Who is responsible?

This may be harder than giving an agent an identity.

Consider a simple example.

Alex asks an operations agent:

"Investigate why the application is slow and fix it."

The agent investigates the logs.

It identifies a problem.

It changes a configuration.

The application recovers.

Everything is fine.

Now change one thing.

The configuration change causes a production outage.

Who is responsible?

Alex, who gave the instruction?

The developer who created the agent?

The team that configured its permissions?

The platform team that hosted it?

The company that deployed it?

Or the agent?

The agent can be identified.

But identity does not automatically answer the question of accountability.

This is why Microsoft's current Entra Agent ID model is interesting. Microsoft distinguishes technical ownership from business sponsorship. Owners handle technical administration, while sponsors provide business accountability for the agent's purpose and lifecycle.

That separation becomes particularly important when an agent can cause real-world consequences.

Now imagine agents creating agents

This is where things get even more complicated.

Imagine:

Human → Agent A → Agent B → Agent C

Alex gives Agent A authority to perform a task.

Agent A decides that Agent B can help.

Agent B delegates part of the task to Agent C.

Agent C has access to a sensitive database.

Now Agent C retrieves information it shouldn't.

Who authorized that action?

Alex never directly interacted with Agent C.

Perhaps Alex didn't even know Agent C existed.

This is where delegation becomes critical.

If an agent can delegate authority, that delegation needs to remain traceable.

Otherwise, we could end up with a strange situation:

We know which agent performed the action, but we don't know where the authority came from.

That is an accountability problem.

Microsoft's shared-responsibility guidance explicitly treats interactions between agents as trust boundaries rather than assuming that one agent should automatically trust another.

The principle is simple:

Delegated authority should not become unlimited authority.

How much freedom should an agent have?

This is where security and autonomy collide.

Imagine an infrastructure agent responsible for keeping an application healthy.

At one extreme, it can only recommend actions.

At another, it can change anything it believes is necessary.

Neither extreme is particularly useful.

Instead, imagine giving it boundaries.

It can

• restart a service
• increase capacity
• roll back a deployment

But it cannot

• delete the production database
• change identity policies
• create unrestricted credentials
• access customer financial data

Now the agent has autonomy, but not unlimited authority.

This is the idea of bounded autonomy.

Microsoft's current security guidance emphasizes least privilege, explicit permissions, policy enforcement and controls around agent tool use rather than giving agents unrestricted access.

The more capable the agent becomes, the more important those boundaries become.

What happens when the agent is hacked?

Here's the uncomfortable part.

We tend to think about securing the AI model.

But an agent isn't just a model.

It is something more like:

Model + Identity + Tools + Data + Permissions + Memory + Actions

Compromise any of those components and the consequences can be different.

A manipulated prompt might change what the agent decides.

A compromised tool might change what the agent can do.

Excessive permissions might turn a small mistake into a major incident.

A stolen credential might allow someone to impersonate the agent.

And a compromised agent might potentially use its legitimate access to reach other systems.

So securing an agent isn't simply about making the model resistant to malicious prompts.

It is about containing what happens when the model, tool or surrounding system behaves unexpectedly.

The kill switch matters

If an agent can act, we need to be able to stop it.

Can we:

Revoke its credentials?

Disable its identity?

Terminate its active sessions?

Block a particular tool?

Prevent it from creating another agent?

Restrict its access to a particular database?

Stop all agents belonging to an application?

And perhaps most importantly:

Can we reconstruct what happened afterwards?

A kill switch without an audit trail tells us that something was stopped.

An identity without a kill switch tells us who acted.

We need both.

Should humans approve everything?

It sounds safe.

Whenever an agent wants to perform a sensitive operation, ask a human.

But imagine an agent processing 50,000 routine transactions.

Should a person approve every transaction?

That would defeat much of the reason for using an agent.

NIST's work on agent identity and authorization also discusses the tension between human oversight and scalability: human approval can be useful, but requiring a person to approve every action can undermine the benefits of autonomous systems.

So perhaps the answer isn't:

Humans approve everything.

Nor:

Agents decide everything.

It is:

Humans define the boundaries. Agents operate within them.

Four questions for an agentic world

Perhaps we can reduce the entire problem to four questions.

Identity

Who is acting?

Authority

What is the agent allowed to do?

Control

What boundaries must it operate within?

Accountability

Who is responsible when something goes wrong?

And there is a fifth question that becomes increasingly important:

Recovery

How quickly can we stop it and understand what happened?

These aren't the same problem.

An agent can have an identity without having permission.

It can have permission without having unlimited authority.

It can have authority without being allowed to cross a particular boundary.

And it can have all of those things while still requiring a clearly accountable human or organization behind it.

The difficult part isn't creating millions of agents

We are likely to see many more agents inside organizations.

Some will be permanent.

Some will exist for minutes.

Some will work independently.

Some will work together.

Some may create or delegate work to other agents.

The challenge won't simply be making them intelligent enough to perform their tasks.

It will be making sure that, at scale, we can still answer:

Which agent did this?

Who gave it permission?

What was it allowed to do?

What exactly did it do?

Who is accountable?

NIST is already working on the identity and authorization foundations for software and AI agents. Microsoft is building agent identity, ownership, sponsorship, authorization and management into its enterprise identity stack.

These aren't distant theoretical questions anymore.

They are becoming infrastructure questions.

And perhaps that's the strange thing about the next generation of AI.

We spent years asking:

How intelligent can AI become?

Then we started asking:

What can AI do?

Now perhaps we need to ask something different:

If an AI agent can act on its own, who is it — and who is responsible when it hacks?

The difficult part of an agentic future may not be creating millions of agents.

It may be creating millions of agents without creating millions of accountability gaps.