The Agent Did Exactly What It Was Told To Do. That’s the Problem.

What three engagement patterns taught us about the real cost of an unsecured AI agent - and why the invoice rarely shows up where you expect it

Originally published at pentesttesting.com/cost-of-an-unsecured-ai-agent/

Nobody on the engineering team broke anything. That's usually how these findings start, not with an alarm, but with a shrug. "It did what it was allowed to do." We hear a version of that sentence in almost every agentic AI engagement we run now, and it's usually the moment a client realizes the risk they're carrying isn't a bug. It's a permission they granted on purpose, months ago, for a completely reasonable-sounding demo.

A support copilot at a mid-market SaaS company had one job: answer billing questions and cut down on ticket volume by issuing small account credits without waiting on a human. Under $50, self-service, no big deal. Three weeks into production, a string of ordinary-sounding follow-up messages talked the agent into approving a credit more than forty times its intended ceiling and surfacing a transaction history that belonged to someone else's account. No exploit. No malware. No CVE number to file. Just an agent doing precisely what its tools allowed it to do, in a context nobody had tested for.

That's the pattern behind almost every unsecured AI agent finding we log: the gap between what an agent is supposed to do and what it's technically allowed to do. And that gap is where the real cost lives.

Why This Isn't a "Jailbreak" Story

Most public discussion of AI security still centers on jailbreaks. Can you trick the model into saying something it shouldn't? That's a real risk category, and it matters. But once an LLM stops just generating text and starts calling APIs, sending messages, or updating records, the more expensive question stops being "what can it be tricked into saying" and becomes "what can it be tricked into doing."

That distinction maps directly onto two categories in the OWASP Top 10 for LLM Applications (2025):

• LLM06:2025 - Excessive Agency, which covers agents granted more autonomy or tool access than their task actually requires.
• LLM02:2025 - Sensitive Information Disclosure, which covers cases where retrieval or tool output surfaces data that the requesting user was never entitled to see.

Prompt injection (LLM01:2025) still shows up constantly but in our reporting, we treat it as the entry vector, not the root cause. It's the technique that nudges an already over-permissioned agent into misusing access it already had. Fix the injection resilience without touching the permission scope, and you've patched the door while leaving the safe unlocked.

"The agent never did anything it wasn't technically allowed to do. That's exactly the problem the permission boundary was drawn around the tool, not around the situation." ~ Shofiur Rahman, CEH, CEO of Pentest Testing Corp

A Composite Case: The Credit That Kept Growing

Here's a sanitized, illustrative version of a pattern we've seen across more than one fintech engagement, not a real client, not a working exploit, and deliberately vague about mechanics.

A support agent had two tools available: look up a customer's own transaction history, and issue a goodwill credit up to a small preset limit. Reasonable scope, reviewed and signed off before launch. During a manual testing engagement, our team held an extended, entirely normal-sounding conversation with the agent: a question about a delayed refund, a reference to a prior support ticket that didn't actually exist, and a gradual reframing of the situation over several turns. By the end of the conversation, the agent had come to treat its credit limit as a per-request suggestion rather than a hard ceiling and, separately, had included line items from the customer's broader account history that had nothing to do with the original question.

No code was broken. No credentials were stolen. The system worked exactly as built. That's what makes this risk class so hard to communicate to a board using traditional breach language. There's no CVE to cite, just a permission boundary that existed on paper and dissolved under a normal-sounding conversation.

What This Actually Costs

The industry data backs up how expensive "working as designed" can get. IBM's 2025 Cost of a Data Breach Report, the first edition to study AI-specific incidents in depth, found that among organizations that experienced a breach involving an AI model or application, 97% lacked proper AI access controls at the time. Those incidents led to compromised data 60% of the time and operational disruption 31% of the time. Separately, "shadow AI" AI tooling deployed without security oversight was a factor in roughly one in five breaches studied, and added an estimated $670,000 to the average cost of those incidents. Sixty-three percent of the organizations studied had no formal AI governance policy at all.

Global average breach cost across the report sits around $4.44 million; U.S. costs run closer to $10.22 million. Those are the headline numbers. In our engagements, the AI agent breach cost pattern also includes line items that never make the report: emergency engineering time to re-scope permissions under pressure, legal review of what data may have leaked, notification obligations if regulated data (HIPAA, GDPR) was exposed, and the slower, quieter cost of an enterprise deal stalling out because a security questionnaire asked a question your team couldn't confidently answer.

"A pentest report tells you where the risk is. It doesn't tell you who owns fixing it, or whether anyone will notice next time. That second part is a policy problem, not a technical one." ~ Shofiur Rahman

Five Controls That Actually Move the Needle

None of this requires rebuilding an agent from the ground up. The fixes that consistently reduce both likelihood and cost in our re-test engagements:

Least-privilege tool scoping. A read-only lookup credential, separate from a write-capable credit-approval credential, not one shared service account that can do everything the backend allows.

Action tiering with a human checkpoint. Read actions, low-impact writes, and high-impact writes shouldn't share a trust level. Anything in the high-impact tier payments, account changes, outbound messages gets a confirmation step or a hard cap.

Retrieval-response filtering. Constrain what the model can include in its answer to the fields relevant to the question asked, instead of trusting it to self-censor a full record it was just handed.

Tenant and session isolation testing. In multi-tenant products, explicitly test whether context or data from one session can leak into another — this is rarely covered by standard functional QA.

Tool-call-level logging. Log the intermediate tool calls and permission checks, not just the final response. This is the single biggest lever for cutting incident response time, and IBM's data shows containment speed is one of the strongest cost predictors in the entire report.

How This Actually Gets Tested

Manual testing for agentic systems looks less like a scanner run and more like a long, patient conversation. Instead of firing a payload list at an endpoint, a tester spends real time in the agent's own interface, building context across turns the way a persistent user or a patient social engineer would. The goal isn't a single clever prompt; it's finding the sequence of ordinary-sounding requests that, cumulatively, moves the agent outside its intended scope.

In practice, that means mapping every tool the agent can call, then testing each one from three angles: Can this action be triggered beyond its intended limits? Can this action be triggered without the authorization that the product team assumed was in place? And separately, does this action's output leak anything the requesting user shouldn't see? Those three questions cover the bulk of what shows up as LLM06 and LLM02 findings in our reports.

The permission map itself is often the most useful artifact that comes out of an engagement that many teams have never seen, in one place, the full list of what their agent's service account can actually reach. It's common for that list to include write actions nobody remembers approving, left over from an earlier prototype that was never fully cleaned up.

Logging gets tested too, not just the agent's behavior. During a re-test, we'll deliberately trigger a boundary-pushing action and then ask: could your team reconstruct exactly what happened from the logs alone, without access to us? If the answer is no, that's a finding in its own right because it's the difference between a two-hour investigation and a two-week one when something happens for real.

What Enterprise Security Reviews Are Starting to Ask

If you sell to mid-market or enterprise customers, this risk class has already started showing up in vendor security questionnaires often before your own team has finished mapping it internally. The questions tend to cluster around a few themes: What actions can your AI agent take autonomously, and what's the approval process above a defined risk threshold? How is agent access scoped relative to a human employee doing the same job? Can you demonstrate, not just assert, that one customer's data can't surface in another customer's session? And critically: if the agent takes an unintended action, how quickly could you tell us, and what would you tell us?

Teams that can answer those questions with evidence, rather than a policy document that was never tested, move through security review noticeably faster. That's a quieter cost than a breach headline, but over a sales cycle, it adds up to real revenue, and it's one of the more compelling reasons to treat agent testing as a launch-blocking step rather than a post-incident cleanup task.

The Part a Pentest Report Can't Fix

Findings from a manual AI agent penetration test map cleanly onto the MAP, MEASURE, and MANAGE functions of the NIST AI Risk Management Framework. They identify what's exposed, size the risk, and hand your team concrete remediation items. What they can't do is satisfy the GOVERN function on their own: the internal policy, ownership, and accountability structure that decides who reviews agent permissions before launch, who owns the risk after, and how often it gets re-tested. A clean report sitting next to no governance policy is close to the industry norm right now, not the exception, and it's the gap that turns a contained, testable finding into an expensive, headline-worthy one.

If You're Reading This Because You're Not Sure

If your team is about to ship an agent with any kind of write access or already has one in production, the useful question isn't "could this be misused," it's "would we know if it had been, and by how much." That's what manual, conversation-based testing is built to answer, and it's not something automated scanners are equipped to catch, since it requires holding a realistic multi-turn conversation the way an actual user (or attacker) would.

We put together a sanitized sample report showing what this kind of finding looks like in practice, scope, methodology, and remediation guidance, with all client-identifying details stripped. If you want a copy, reach out through pentesttesting.com, and we'll send it over, no form-fill funnel required.