AppSec Has a Boundary Problem, and It Is Not the One You Think

Four places application security breaks under AI, why patching fixes none of them, and where the boundary has to move.

For about twenty years, application security has run on one load bearing assumption. Untrusted input arrives, you validate it, you sanitize it, and you keep it away from anything that executes. Everything downstream of that idea is a variation on the theme. Parameterized queries. Output encoding. Content Security Policy. Input allowlists. The entire OWASP web canon is, at its core, an elaborate defense of one boundary: data is data, and code is code, and the attacker's job is to blur that line while your job is to hold it.

That boundary is gone in AI systems. Not weakened, not bypassed in a clever edge case. Structurally absent. And most security programs are still funding controls that assume it is there.

I want to walk through what that actually means, starting from a documented incident, then to three more places the same crack shows up in different disguises. I am going to be careful throughout to separate what is verified from what is my read of it, because a lot of AI security writing right now is confident in inverse proportion to its evidence.

This is the first of five posts. This one is the map. It walks the four breaks end to end so you can see they are the same crack from four angles, and it stays at the altitude a security leader needs to make budget and architecture decisions. The four posts that follow each take one break down to the level where you have to make real design choices, with the incidents, the failed mitigations, and what I think replaces them. If you only read one, read this one. The rest are for the people who have to build it.

The anchor - a single email, no clicks, full exfiltration

In June 2025, Aim Security disclosed EchoLeak, tracked as CVE 2025-32711, a zero click prompt injection vulnerability in Microsoft 365 Copilot. Microsoft, as the CNA, scored it CVSS 3.1 9.3 critical and published the record on June 11, 2025 under the weakness class CWE 74, command injection. Here is the chain, and the detail matters because two separate controls failed for two separate reasons.

An attacker sends a crafted email to someone inside the organization. The email language is written to evade Copilot's XPIA classifier, the control Microsoft built specifically to catch cross prompt injection attempts. That is bypass one, and it is a detection failure. The hidden instructions inside the email are scored as ordinary content and pass into Copilot's retrieval context.

Later, the victim asks Copilot something unrelated. Copilot builds its RAG context, pulls in the attacker's email as relevant material, and executes the instructions inside it. Now the data needs a way out. Microsoft's Content Security Policy is supposed to prevent exactly that. The exploit routed exfiltration through an allowlisted Teams image proxy, a domain the CSP already trusted. That is bypass two, and it is an egress failure, a completely different control from the first one.

No credential theft. No malware. No memory corruption. No exploit in the sense the word has meant for thirty years. The model did precisely what it was designed to do, follow the instructions in its context, and it had no mechanism to distinguish the user's instructions from the attacker's, because both arrived as text in the same channel. Microsoft patched it server side and reported no known exploitation in the wild.

That last paragraph is the whole thesis of this article. Read it again.

The reason EchoLeak matters is not that Copilot had a bug. It is that the bug was not, in a meaningful sense, in the code. Simon Willison had already named the underlying condition on June 16, 2025, calling it the lethal trifecta: access to private data, exposure to untrusted content, and the ability to communicate externally. Any two of those are survivable. All three in one system, and an attacker who controls the untrusted leg owns the private leg and has a way to ship it out. Copilot had all three by design. So does almost every enterprise AI deployment being greenlit this quarter.

That is the established part. From here forward, more of this is my analysis, and I will mark where the line is.

Break one: The input trust boundary collapses

Prompt injection gets discussed as if it were a vulnerability class, something that sits alongside XSS and SSRF in a list you triage against. I think that framing is wrong, and I think it is the single most expensive misunderstanding in AI security right now.

XSS is a bug. It has a fix. Encode the output, respect the context, and the class dies. Prompt injection has no equivalent because there is no encoding boundary to enforce. The model receives one flat sequence of tokens. Your system prompt, the user's question, the retrieved document, the tool output, the email someone sent from outside, all of it arrives in the same channel with no cryptographic or structural marker separating trusted from untrusted. You can label it. You can wrap it in tags and beg the model to treat everything inside as data. The model is a probabilistic system that decides, per token, what to attend to. Labeling is a suggestion, not a control.

The person who named this attack class saw the boundary problem from the start. Simon Willison coined the term prompt injection, and he coined it by analogy to SQL injection, because in his framing the underlying problem is the same one: trusted and untrusted content mixed together in a single context. SQL injection had a fix, because SQL has a grammar and you can parameterize against it. Natural language has no grammar you can parameterize. That is the whole difference, and it is why the analogy names the problem without offering the cure.

Willison's own assessment as of June 2025 is that we still do not know how to reliably prevent this. He is direct about the vendor response, saying he is deeply suspicious of guardrail products, because they tend to advertise catching something like 95 percent of attacks, and in web application security 95 percent is a failing grade. That is the right standard. An attacker does not need to succeed 5 percent of the time across a population. They need to succeed once, against you, and they can retry.

Forecast, mine. Classifier based prompt injection defenses will keep getting funded and will keep getting bypassed, because a classifier trying to detect adversarial instructions in natural language is playing the same losing game as signature based antivirus in 2005. I expect the programs that survive this to stop trying to detect malicious input and start assuming all input is malicious, then constrain what the model is permitted to do with it. The defensive center of gravity moves from the input side to the capability side. That is a design change, not a tooling purchase, which is why it is going to be slow and painful for organizations that treat AI security as a product category.

Break two: Identity multiplies past what IAM was built to hold

In May 2025, Invariant Labs demonstrated something on GitHub's MCP server that should have changed more minds than it did.

The setup is mundane. A developer runs an AI coding agent connected to GitHub through MCP, using a personal access token that has access to both their public and private repositories, because that is how tokens are usually scoped. An attacker files an issue on the developer's public repository. The issue contains hidden instructions. The developer asks their agent to look at open issues. The agent reads the malicious issue, follows the instructions inside it, pulls data from the developer's private repositories, and publishes it into a pull request on the public repo, where the attacker can read it.

Invariant called this a toxic agent flow. What they were careful to say, and what got lost in the coverage, is that this was not a flaw in the GitHub MCP server's code. It is architectural. There was no patch to ship, because nothing was broken. Every component behaved as specified.

Willison used this same case as his canonical example of the trifecta, pointing out that a single MCP server managed to combine all three legs by itself: it could read issues filed by attackers in public repositories, reach into private repositories, and create pull requests that carried the private data back out.

Look at what the identity model actually did here. The token was issued to a human. The human is trusted. The agent inherited the human's full authority and then took instructions from an anonymous stranger on the internet. At no point in that chain does any system have a concept of "this action was requested by an attacker rather than by the token holder," because the identity layer does not model intent, it models authority. And authority was never in question.

This is the part I think leadership consistently underestimates. Non human identity is not a scaling problem. It is not "we now have ten thousand service accounts instead of one thousand and we need better lifecycle management." It is a semantics problem. Every identity system we have assumes that the principal holding a credential is the one deciding how to use it. Agents break that assumption at the root. The credential holder and the decision maker are now different entities, and the decision maker can be influenced by anyone who can get text in front of it.

OWASP put a category on this. ASI03, Identity and Privilege Abuse, in the Top 10 for Agentic Applications 2026, published December 9, 2025 by the GenAI Security Project. My read, not OWASP's: the GitHub MCP case is the cleanest public demonstration of ASI03 and ASI01 firing together, and the fact that it required no vulnerability at all is the part that should keep people up at night.

Forecast, mine. Least privilege is necessary and insufficient here. The thing that has to get invented, and I do not think it exists in production anywhere at scale today, is a way to bind an agent's authority to the provenance of the instruction it is currently acting on. A token that is valid for reading a private repo when the human asked, and invalid for the same read when the request traces back to text scraped from an untrusted source. That is a hard problem, it is where I expect serious identity vendors to converge by 2028, and until it exists, scoping tokens narrowly is a mitigation, not a solution.

Break three: Egress becomes the last line, and it was never designed to be

Both incidents above have the same final step. Data goes out.

EchoLeak needed the Teams image proxy. The GitHub MCP attack needed the public pull request. Neither attack completes without an outbound channel. And in both cases, the outbound channel was a legitimate, expected, business justified feature of the product.

Here is what I think this does to security architecture, and it is close to inverting the priority order most programs run on today.

For most of AppSec history, egress control has been a background concern, a compensating control, something the network team owns and the AppSec team references in a diagram. The real work happened at the input and the code. Under the lethal trifecta model, that ordering flips. If you cannot reliably control what instructions reach the model, and you cannot reliably control what data the model can see, because you deployed it precisely so it could see your data, then the only leg of the trifecta you have real leverage over is the third one. Egress is not a compensating control anymore. It is the primary one.

Forecast, mine, and I hold this one with high confidence: the mature AI security architecture of 2028 will look less like a scanner and more like a broker. Every outbound action an agent can take, every network call, every write, every message, every commit, will pass through an enforcement point that evaluates it against policy independent of the model's judgment, because the model's judgment is the thing under attack. Some of this exists today under names like agent gateway and MCP proxy. Most of it is immature. All of it is going to matter more than the SAST budget.

Break Four: Human review leaves the loop, and nobody logs the moment it happens

This is the one I have the least hard evidence for and the most conviction about, so I am labeling it clearly as analysis.

Every one of the attacks above required the agent to take an action without a human individually approving it. That is not incidental. It is the product requirement. Nobody buys an agent to ask permission for every step. The value proposition is autonomy, and autonomy is measured in how many approvals you removed.

So here is what happens organizationally, and I have watched versions of this play out. An agent ships with a human in the loop on every write. It works, but it is slow, and the approvals become rubber stamps within two weeks because a person clicking approve on their forty first pull request of the day is not performing a security review. Then someone points out, correctly, that the approval step is theater, and removes it for low risk actions. Then the definition of low risk expands, because it always does, and because expanding it is how you hit the efficiency numbers you promised when you funded the project.

At no point does anyone make a decision that looks like "we are removing security review from this system." Each individual step is defensible. The aggregate is that the last human check on an agent's behavior disappeared, and there is no artifact anywhere recording when.

Forecast, mine. This is where the next serious AI incident comes from. Not a novel exploit. A well understood attack class, prompt injection, landing in a system where the compensating control was human oversight, and human oversight had quietly degraded to nothing over eighteen months of reasonable local decisions.

What I think this adds up to

The four breaks are the same crack seen from four angles. Input validation, identity, egress control, and human review are all downstream of one assumption: that a system's behavior is determined by its code and its configuration, and that both are things you control.

Agents break that. Their behavior is determined by their context, and their context is populated at runtime from sources you do not control and cannot fully enumerate. That is not a new vulnerability class to add to the list. It is a different substrate for the entire discipline.

I do not think traditional AppSec goes away. Threat modeling, secure SDLC, vulnerability management, all of it still holds and still matters, and anyone telling you AI makes it obsolete is selling something. What I think happens is that each of those practices gets rebuilt around a boundary that sits somewhere new. Not between input and code, because that boundary no longer exists. Somewhere else. At the identity layer, at the egress layer, wherever a boundary can still be enforced by something deterministic rather than something probabilistic.

Figuring out exactly where that boundary goes is, as far as I can tell, the actual work of the next five years in this field. The next four posts are my attempt to figure it out in public.

Coming in this series

Post two. The Boundary That Cannot Be Encoded. Why prompt injection is not a vulnerability class and will not be patched. What every proposed mitigation gets wrong, what the research on structural defenses is showing, and why I think the defensive center of gravity moves off the input side permanently.

Post three. Your Identity Model Assumes the Credential Holder Is Deciding. Non human identity as a semantics problem rather than a scaling problem. Why least privilege is necessary and insufficient, and what has to get invented to bind an agent's authority to the provenance of the instruction it is acting on.

Post four. Egress Was Never Supposed to Be the Primary Control. The trifecta leg you can still hold. What an enforcement point that does not trust the model's judgment looks like, why agent gateways are immature, and why I think this line item outgrows the SAST budget before 2030.

Post five. Nobody Logs the Moment Human Review Leaves the Loop. How oversight degrades through a sequence of individually defensible decisions, why I think this is where the next serious AI incident comes from, and what an organization can do that is not theater.

Sources

CVE 2025 32711 (EchoLeak). CVE record published June 11, 2025, assigned by Microsoft as CNA, titled "M365 Copilot Information Disclosure Vulnerability," weakness class CWE 74. CVSS 3.1 base score 9.3 critical per Microsoft. At least one commercial tracker lists a lower base score of 7.5 for the same CVE, so the 9.3 is attributed to Microsoft rather than presented as consensus. Attack chain detail, XPIA classifier evasion and CSP bypass through an allowlisted Teams image proxy, from the Aim Labs disclosure, June 2025. Server side patch, no known exploitation in the wild.

The lethal trifecta, and prompt injection as a term. Simon Willison, "The lethal trifecta for AI agents: private data, untrusted content, and external communication," June 16, 2025, simonwillison.net. Source for the trifecta definition, the guardrail 95 percent critique, the statement that reliable prevention is still unsolved, his account of coining prompt injection by analogy to SQL injection, and his reading of the GitHub MCP case as a single tool combining all three legs.

GitHub MCP toxic agent flow. Invariant Labs, May 26, 2025, invariantlabs.ai. Malicious public issue, agent coerced into reading private repositories and leaking contents through an autonomously created public pull request. Researchers characterized it as architectural rather than a code defect in the MCP server.

OWASP Top 10 for Agentic Applications 2026 (ASI01 to ASI10). OWASP GenAI Security Project, announced December 9, 2025. The mapping of specific incidents to specific ASI categories in this article is my analysis, not OWASP's classification.