Federica Crudo: When AI Has Permission to Act, Who Protects the Human Intent?

As autonomous AI systems gain access to digital infrastructure, the next cybersecurity challenge may lie beyond identity, credentials and conventional access control.

A cybersecurity breach does not always begin with a stolen password or a compromised server. In an increasingly agentic digital environment, a system could possess legitimate credentials, use approved tools and communicate through authorized interfaces — yet still perform an action that its human operator never intended.

This creates a difficult question for the next generation of cybersecurity: what happens when an AI system is technically authorized to act, but the action itself no longer serves the purpose for which that authority was granted?

Traditional security architectures are largely designed around identities, permissions, networks and resources. These remain essential, but autonomous AI introduces another dimension. Agents interpret instructions, process untrusted information, make decisions and delegate tasks to other systems. Their behavior can change as they encounter new information, even when the credentials they use remain valid.

For Federica Crudo, this points to a potentially fundamental shift in security design: moving beyond the question of who can access a system to examine whether every consequential action remains consistent with the original human intention.

The gap between permission and purpose

Consider an AI agent authorized to organize a user's inbox, summarize messages and prepare draft responses. It encounters an email containing instructions that attempt to redirect its behavior. The agent interprets those instructions as relevant to its task and uses its existing integrations to access or transmit information.

No password necessarily needs to be stolen. No traditional access-control rule necessarily needs to be broken. The danger emerges from the relationship between the content the agent processes and the authority it already possesses.

This is closely related to what cybersecurity researchers describe as the confused deputy problem: a privileged system can be manipulated into using its legitimate authority on behalf of an unauthorized party.

The problem becomes more complex when agents delegate tasks to other agents, which may then invoke additional tools, APIs or services. Each step creates another opportunity for the original task to be misunderstood, expanded or detached from the intentions of the person who initiated it.

Martin Kováč: Does this mean that identity-based security and Zero Trust are becoming insufficient?

Federica Crudo: "They remain fundamental, but identity and intention are different things. Knowing which agent is acting, and confirming that it possesses valid permissions, does not necessarily establish that a specific action belongs to the task it was authorized to perform. We need to connect the authority to the purpose, not just to the identity."

From access control to authority continuity

One possible direction is to treat authorization not as a permanent property of an agent, but as a temporary and verifiable mandate associated with a specific task.

For the purposes of this discussion, this principle can be described as authority continuity: the requirement that an action remains traceable to the original human authorization throughout every stage of its execution.

The idea goes beyond recording which account called an API or which agent modified a file. It asks whether the action was still justified by the original task, what information influenced it, which permissions were delegated along the way and whether the scope of authority changed unexpectedly.

Imagine an AI agent coordinating a complex workflow involving several specialized agents. Instead of allowing each participant to inherit broad permissions from the preceding system, every delegation would carry explicit constraints. Those constraints would define the permitted objective, the accessible resources, the types of actions allowed and the point at which further authorization becomes necessary.

The authority granted to a downstream agent could never exceed what was delegated to it. It would also be limited in duration and scope, with sensitive or irreversible actions subject to independent checks.

Crucially, these restrictions would not depend exclusively on the AI deciding to follow its instructions. A separate enforcement mechanism would have to evaluate proposed actions against the applicable authorization rules before execution.

Could cybersecurity learn to verify intent?

Martin Kováč: How would you prevent this from becoming another AI system monitoring an AI system, with the same weaknesses repeated twice?

Federica Crudo: "I would not make another language model the final authority over whether an action is allowed. AI can help interpret a task and identify suspicious behavior, but the enforcement layer should operate independently of the model's own decisions. Permissions, boundaries and approval requirements need to be checked outside the reasoning process."

This distinction is important. A model that has been manipulated cannot be expected to serve as the sole reliable judge of whether its own actions are appropriate.

A more robust architecture would separate interpretation from execution. The AI could propose an action, explain its intended purpose and supply the context behind its decision. An independent policy mechanism would then evaluate whether the request falls within the authority granted by the human operator.

A record of the decision could preserve the relevant evidence: the originating authorization, the chain of delegation, the policy applied and the action ultimately executed.

Such records would make it easier to investigate incidents and distinguish between an agent following an authorized task, a legitimate task being misinterpreted and a downstream system exceeding its delegated authority.

This would not eliminate every risk. Intent can be ambiguous, policies can be incomplete, and an independent control can itself be misconfigured. But it would create a stronger separation between what an AI system believes it should do and what it is technically permitted to execute.

A different way to measure AI security

The implications extend beyond traditional access management.

AI agents are increasingly evaluated according to their ability to complete tasks, their speed and the quality of their results. Yet successful completion alone is not a sufficient measure of security. A system might achieve its objective while disclosing information it should have protected, modifying resources outside its scope or delegating authority too broadly.

A more meaningful evaluation would also examine whether the agent remained within its authorized purpose throughout execution.

Did every consequential action have a traceable basis in the original authorization? Did delegation preserve the same restrictions? Could the system detect when an external instruction attempted to redefine its objective? Were sensitive operations stopped when their authorization became uncertain?

These questions suggest a broader principle: an AI system should not be considered secure simply because its final output is correct. Its path to that output matters, too.

The next cybersecurity boundary

As AI systems become more capable of acting independently, cybersecurity will have to evolve beyond protecting individual accounts, applications and networks. It will also need to address how authority is created, transferred, constrained and revoked across increasingly autonomous systems.

The objective is not to eliminate autonomy. Excessive restrictions can make intelligent agents ineffective, while unrestricted authority creates unnecessary exposure. The challenge is to grant enough freedom to complete legitimate tasks without allowing that freedom to expand silently as the system interacts with new information and other agents.

For Crudo, this is where the next stage of AI security becomes particularly interesting: at the boundary between what a system can do and what it was actually authorized to do.

The most important question may therefore change. Instead of asking only, "Did this agent have access?", security systems may increasingly need to ask: "Who authorized this action, for what purpose, and can we prove that the original intention survived every step?"

In an era of autonomous software, protecting access may no longer be enough. The deeper challenge is protecting the human intention behind it.