Let me describe a class of vulnerability that's been quietly living inside LLM applications since the first wave of production deployments. It doesn't require a sophisticated attacker. It doesn't exploit some obscure edge case in the model's training. In many cases, the developer who introduced it never thought it was a security decision at all.

It's called Improper Output Handling, LLM05 in the OWASP LLM Top 10 for 2025, and the reason it keeps appearing in architecture reviews is that most engineers building LLM-powered features think of model output as text. It is text. It's also data that gets processed by other systems. And that distinction is everything.

The Old Problem Wearing a New Hat

If you've spent any time in application security, the pattern here will feel familiar. Injection vulnerabilities, SQL injection, command injection, XSS all share the same root cause: data that an attacker can influence flows into a system that interprets it as instructions, and the system doesn't adequately distinguish between the two.

For decades, the data in question was user input: form fields, URL parameters, HTTP headers. The lesson security engineering absorbed, slowly and painfully, was that user input must be treated as untrusted and sanitized before it reaches any system that interprets it as structured data or executable instructions.

Now add a language model to the architecture. The model generates output. That output gets forwarded to a rendering engine, a database, a template system, a code execution environment, an email client, a browser. Does your sanitization logic still sit between the data source and the interpreter? Or did someone assume that because the model generated the content, it must be safe?

That assumption is the vulnerability.

What "Downstream Systems" Means in Practice

When people talk about LLM applications, they tend to picture a chat interface: user types something, model responds, response appears on screen. That's accurate for some deployments. It's increasingly rare for production systems.

In practice, LLM components are embedded in pipelines. A customer support bot doesn't just respond to queries, it retrieves data from a CRM, generates a response, logs the interaction to a ticketing system, and potentially triggers follow-up workflows. A coding assistant generates code that gets executed. A document processing system generates structured output that gets parsed and stored. A marketing tool generates content that gets rendered in a browser.

Each of those downstream handoffs is a potential injection point. And the model's output is the data crossing that boundary.

Consider a scenario where an LLM application retrieves customer emails, processes them for intent classification, and generates responses that include extracted data. An attacker sends an email that contains carefully crafted content, something like a hidden instruction embedded in normal-looking text. The model, trained to be helpful and to follow instructions, incorporates the attacker's directive into its output. That output then flows to the ticketing system, which interprets certain patterns as commands. The attacker has just achieved indirect prompt injection that propagates into a downstream system.

This is not theoretical. MITRE ATLAS documents this class of attack. The technique involves an adversary embedding adversarial content in data that the model will encounter, not in a direct conversation with the model, but in an email, a document, a web page, a database record that the model retrieves as context. The adversarial payload travels through a legitimate data source and achieves execution when the model incorporates it into output that reaches an interpreter.

The Sensitive Data That Leaks Through the Output Channel

Improper Output Handling doesn't only concern injection. It has a second, quieter failure mode: information disclosure. LLM02: Sensitive Information Disclosure describes how LLMs can leak confidential data, PII, or system details through their responses and the output handling layer is often where the controls should live but don't.

A model trained on or given access to sensitive data will, under the right conditions, reproduce that data in its responses. The conditions don't require an attacker. They can be as simple as a user asking a question that happens to be adjacent to sensitive information in the model's context window. Retrieval-augmented systems are particularly vulnerable here: the retrieval step fetches documents the user might need; the generation step may include details from those documents that the user shouldn't have.

I've seen this pattern in internal knowledge base applications where the retrieval system returns the ten most semantically similar documents to a query, and the LLM then generates a response that synthesizes all ten including the two documents the querying user didn't have permission to see. The access control logic was implemented at the retrieval layer but not enforced at the generation layer. The model had no concept of the user's permission level; it simply used everything in its context.

The NIST AI 100–2 document, published in January 2025, catalogs this as a failure of output governance, the absence of controls that verify what the model's response is allowed to contain before it reaches the user. It's a technical problem with an organizational dimension: someone needs to own the question of what is this model permitted to output, and how do we verify compliance at runtime?

The Three Places Teams Miss the Problem

In my experience reviewing LLM architectures, Improper Output Handling tends to slip through at three specific points:

At the rendering layer. Teams implement input sanitization, they strip dangerous characters from user queries before sending them to the model. They don't implement equivalent output sanitization before the model's response is rendered in a browser or passed to a document system. If the model can be manipulated into including JavaScript in its output, and that output is rendered without sanitization in a web context, you have stored XSS via an AI assistant. The attack surface is user-controlled data that the model might incorporate into its output, which is effectively unlimited in a general-purpose assistant.

At the code execution boundary. Coding assistants and AI-powered development tools generate code that often gets executed, either directly, as part of a CI/CD pipeline, or by a developer who trusts the model's output without review. The model doesn't have a concept of "this is a test environment" versus "this is production." It generates code based on its training and context. An attacker who can influence what's in that context can influence what code gets generated. Output that reaches an execution environment without review is a direct injection risk.

At the inter-service boundary. Microservice architectures that incorporate LLM components often pass model output between services as structured data JSON, XML, templated strings. Each of those formats has its own injection risks. A model that generates JSON output that gets parsed by a downstream service may produce output that's structurally valid but semantically malicious if the attacker can influence the model's output in the right way. Teams who handle SQL injection carefully in their application layer sometimes forget to apply the same thinking when the data source is a language model rather than a user form field.

What the Fix Actually Looks Like

The remediation pattern for Improper Output Handling draws on established secure development practices, because at its core, this is the same problem that's always existed: don't trust data from external sources, even when that source is your own AI component.

Treat model output as untrusted input to every downstream system. This is the mental model shift that matters most. The model may be your model, trained on your data, running in your infrastructure. Its output is still external data from the perspective of every system that receives it. Apply the same validation, sanitization, and encoding rules you'd apply to any other external data source.

Implement output filtering at the boundary, not inside the model. You cannot rely on the model's training to prevent it from reproducing sensitive information or generating output that could cause injection in a downstream system. Those constraints need to be enforced at the output layer: before the response reaches the user, before it reaches a rendering system, before it crosses a service boundary. Content classifiers, regular expression filters for PII patterns, output schema validation, these are the controls that actually work at scale.

Enforce access control at the output layer, not just at retrieval. For RAG systems and knowledge base applications, the permission model needs to persist all the way to generation. If a document is outside a user's access scope, the model should not be able to include information from that document in its response, not because the model knows the user's permissions, but because the output layer validates what's allowed before returning the response.

Log output, not just input. Most teams log user queries for monitoring and debugging. Far fewer log the model's responses at the same level of detail. Output logging is essential for detecting information disclosure incidents after the fact, for auditing what the model actually communicated to users, and for identifying injection attempts that may have succeeded. The NIST AI RMF's Measure function specifically includes monitoring AI outputs as part of ongoing risk assessment, you cannot measure what you don't record.

The Framing That Makes This Tractable

The teams I see handling this well have internalized one framing that makes the problem tractable: the LLM is an external service, even when it's internal.

Your language model is not part of your application logic in the way a function call is. It's a component with nondeterministic behavior, influenced by inputs you may not fully control, capable of producing outputs you cannot fully predict. Treating it with the same trust assumptions you'd apply to a third-party API rather than the trust you'd extend to your own application code, leads naturally to the right security posture: validate its outputs, don't expose it directly to interpreters, and never let its responses flow unexamined into systems that can take real-world actions.

That framing also makes the governance question answerable. If the LLM is an external service, then the question of what it's permitted to output is a service interface contract question, not a training question. And interface contracts are something security and engineering teams know how to define, enforce, and audit.

The OWASP LLM Top 10 exists because these problems are real, recurring, and often underestimated. Improper Output Handling isn't the most dramatic entry on the list. It doesn't have the conceptual novelty of adversarial ML attacks or the visceral alarm of a data poisoning incident. But it's the category most likely to bite a team that knows what it's doing in traditional security, simply because it requires applying familiar principles in an unfamiliar context.

The context is new. The principles aren't. That should make this one solvable.

This article is part of a series on AI application security, drawing on the OWASP LLM Top 10, MITRE ATLAS, and the NIST AI Risk Management Framework.