November 19, 2026
AI Security : Your AI Agent Can be Your Biggest Security Vulnerability
The hidden vulnerabilities, attack surfaces and security challenges behind todayβs rapidly evolving AI ecosystem.

By Raghunath G
9 min read
Building an AI application is easier than ever. Building one that is secure, reliable and ready for real-world use is a very different challenge.
AI has dramatically lowered the barrier to building software.
With AI coding assistants, powerful foundation models, open-source frameworks and agentic development tools, developers can now build applications in days that once took weeks or months.
Even people without extensive programming experience can create functional applications using natural language instructions.
But there is an unknown reality we need to discuss.
A working AI application is not necessarily a secure AI application.
An AI assistant might perform perfectly during a demonstration. It might answer questions accurately, retrieve information from documents and execute tool calls as expected.
But what happens when it encounters a malicious document? What if it retrieves information belonging to another user? What if an agent has permission to delete records when it only needs to read them?
These aren't hypothetical design questions. They are security problems that emerge when AI systems interact with real data, users, APIs and business workflows.
The challenge is no longer just securing the model.
It is securing the entire system built around it.
1. Why AI security deserves more attention
Traditional application security focuses on protecting code, infrastructure, identities, APIs, databases and networks.
AI introduces another layer of complexity.
Modern AI applications often combine:
- Large Language Models (LLMs).
- Retrieval-Augmented Generation (RAG).
- Vector databases and embedding models.
- External APIs and third-party services.
- AI agents with tool access.
- Model Context Protocol (MCP) integrations.
- User data, enterprise documents and business systems.
Each integration introduces another potential attack surface.
Consider a simple enterprise AI assistant.
An employee asks it to retrieve information from internal documents. The application searches a knowledge base, passes relevant content to an LLM and generates a response.
Now imagine someone has inserted malicious instructions into one of those documents.
The model may interpret those instructions as part of its context and behave in an unintended way.
If the assistant also has access to enterprise tools, the consequences could extend beyond an incorrect response.
The security boundary is no longer just the application. It extends across the entire AI execution path.
2. Six AI security risks every builder should understand
The OWASP GenAI Security Project published its 2026 Top 10 for LLM Applications in August 2026.
It reflects the changing threat landscape as AI applications become more connected, autonomous and integrated into business workflows.
Here are six areas worth understanding before deploying an AI application.
2.1. Prompt injection: When AI follows the wrong instructions
Prompt injection occurs when malicious input influences an AI model to behave in unintended ways.
The attack can be direct or indirect.
A direct attack comes from a user attempting to override the model's instructions.
An indirect attack can be hidden inside a webpage, email, PDF or document that the AI system processes.
For example:
An enterprise AI assistant is designed to summarise internal documents. An attacker inserts instructions into a document asking the assistant to reveal confidential information.
The user simply asks for a summary. However, the retrieved document contains additional instructions that attempt to manipulate the model.
This is particularly concerning for RAG applications and AI agents that interact with external content.
What builders should do:
- Treat external content as untrusted input.
- Separate trusted instructions from retrieved information.
- Restrict access to sensitive resources.
- Validate outputs before passing them to other systems.
- Require human approval for sensitive actions.
There is no single prompt filter that completely eliminates prompt injection.
2.2. Sensitive information disclosure: Protecting data in AI workflows
AI applications frequently process sensitive information.
This could include customer records, financial information, employee data, credentials, internal documents or proprietary business information.
The problem isn't limited to what the model returns.
Sensitive information can also be exposed through prompts, application logs, retrieval systems, debugging tools and third-party services.
Imagine an AI-powered customer support assistant.
If the application retrieves customer information without enforcing user-level authorisation, one customer might receive information belonging to another.
The model doesn't need to be compromised for this to happen. The application itself may have an authorisation weakness.
What builders should do:
- Enforce authorisation at the application and data layers.
- Apply least-privilege access to data sources.
- Avoid placing credentials and secrets in prompts.
- Mask or redact sensitive information where appropriate.
- Review logging, retention and third-party data handling.
Never rely on a prompt instruction to enforce data security.
2.3. Excessive agency: When AI has too much power
This is one of the most important risks as organisations move from AI assistants to autonomous agents.
An AI agent can interact with external tools, query databases, send emails, create tickets, modify records and execute workflows.
That makes agents useful.
It also creates a much larger security responsibility.
Consider an AI assistant designed to manage project documentation.
It needs permission to read documents and perhaps create new ones.
But what if it also has unrestricted access to delete files, modify permissions or send documents externally?
The problem is excessive agency: giving an AI system more functionality, permissions or autonomy than its task requires.
What builders should do:
- Apply least-privilege permissions.
- Use narrowly scoped credentials.
- Separate read and write permissions.
- Require approval before consequential actions.
- Maintain audit logs of agent activity.
- Restrict agent access to specific tools and resources.
An agent should only be able to perform actions that its assigned role requires.
2.4. RAG security: Protecting the knowledge layer
RAG has become one of the most common approaches to building enterprise AI applications.
It allows an LLM to retrieve relevant information from external knowledge sources instead of relying entirely on its training data.
However, adding a vector database doesn't automatically make an application secure.
Consider an enterprise knowledge assistant used by multiple departments.
Finance employees have access to financial documents. HR employees have access to employee records. Engineering employees have access to technical documentation.
If the retrieval layer doesn't enforce these permissions, the model could receive information that the requesting user isn't authorised to access.
This creates a cross-user or cross-department data exposure risk.
Another concern is data poisoning, where malicious or manipulated content enters the knowledge base and influences the model's responses.
What builders should do:
- Enforce permissions before retrieving documents.
- Preserve access-control metadata.
- Isolate tenants and sensitive data.
- Validate documents before ingestion.
- Monitor changes to knowledge sources.
- Test for unauthorised cross-user retrieval.
Remember:
The model should never be responsible for deciding what information a user is allowed to access.
That decision belongs to the application.
2.5. AI supply chain and MCP security
An AI application rarely depends on a single model.
It may rely on external models, open-source libraries, datasets, plugins, APIs, deployment platforms and third-party tools.
Each dependency creates potential security exposure.
The emergence of Model Context Protocol (MCP) adds another important consideration.
MCP allows AI applications to connect with external tools and data sources through a common protocol.
This makes integrations easier, but it also means that developers must understand what those tools can access and execute.
A poorly configured MCP server or an untrusted tool integration can expose sensitive data or create opportunities for unintended actions.
What builders should do:
- Use trusted model and package sources.
- Review third-party dependencies.
- Verify MCP server identity and permissions.
- Avoid unrestricted token sharing.
- Review tool descriptions and capabilities.
- Monitor dependency and configuration changes.
Convenient integrations should never replace security reviews.
2.6. Unbounded consumption, misinformation and unsafe outputs
Not every AI security problem involves data theft or unauthorized access.
AI applications can also be exposed to excessive resource consumption, unreliable information and unsafe downstream execution.
For example:
- Repeated requests can drive up model API costs.
- Malicious inputs can trigger expensive processing.
- An AI assistant can confidently generate incorrect information.
- Generated SQL or code can create vulnerabilities if executed without validation.
- An application can expose internal context through its responses.
The OWASP 2026 framework explicitly includes unbounded consumption, misinformation, hidden context exposure and improper output handling.
What builders should do:
- Implement rate limits and usage budgets.
- Validate generated SQL, code and structured outputs.
- Apply output encoding for the destination system.
- Monitor token consumption and API usage.
- Test factual reliability and unsupported responses.
- Avoid executing generated commands without appropriate controls.
Generated content should be treated as untrusted input, especially when it enters another application or execution environment.
3. Tools that can help you test AI security
Understanding the risks is only the starting point.
You also need to test how your application behaves under adversarial conditions.
Here are four tools worth exploring.
1. Garak
An open-source LLM vulnerability scanner that runs probes against supported model interfaces. It can help identify weaknesses involving prompt injection, data leakage and other undesirable model behaviours.
Useful for structured model-level testing.
2. Microsoft PyRIT
An open-source framework for automated and human-led red teaming of generative AI systems.
It supports repeatable attack scenarios and multi-turn testing.
Useful for teams looking to build more systematic AI red-teaming practices.
3. Giskard
Provides vulnerability scanning and testing capabilities for AI applications, including agentic workflows.
Useful for exploring adversarial scenarios and identifying potential quality and reliability issues.
4. Promptfoo
Documentation β Promptfoo Red Teaming
Supports automated red teaming and repeatable evaluations of LLM applications, including custom RAG and agent workflows.
Useful for integrating security evaluations into application development and testing.
One important distinction: these tools don't replace application security testing.
A model scanner cannot independently establish whether your database authorization, API permissions or deployment configuration is secure.
Test the complete application, not just the model.
4. Frameworks worth bookmarking
If you're building or managing AI applications, these resources provide a useful foundation.
OWASP GenAI LLM Top 10β2026
A practical reference for understanding the most important security risks associated with LLM applications.
OWASP Securing Agentic Applications V2
Released in November 2026, this guide covers secure architecture, identity and access controls, runtime protection, supply chain security and assurance testing across single-agent and multi-agent systems.
OWASP GenAI Data Security Best Practices V2
The November 2026 updates provide additional guidance on protecting data throughout generative AI workflows.
MITRE ATLAS
A knowledge base of adversarial tactics and techniques targeting AI-enabled systems. Useful for threat modelling and understanding how attackers approach AI environments.
NIST AI Risk Management Framework
Provides a structured approach to managing AI risks through four functions: Govern, Map, Measure and Manage.
Its Generative AI Profile, NIST AI 600β1, provides additional guidance for generative AI systems.
These frameworks serve different purposes. Use them as complementary resources rather than treating any single framework as proof that an application is secure.
5. A practical learning path for AI builders
You don't need to become a cybersecurity specialist before building your first AI application.
But you should understand the security implications of the systems you create.
Here is a practical approach.
Week 1: Understand application security fundamentals
Learn authentication, authorization, API security, input validation, secrets management and dependency security.
Week 2: Build a small AI application
Create a basic chatbot or RAG assistant using sample data.
Understand how prompts, models, embeddings, retrieval and APIs work together.
Week 3: Explore security weaknesses
Experiment with direct and indirect prompt injection, unauthorized data retrieval, unsafe outputs and excessive agent permissions.
Use applications and environments you own or are authorized to test.
Week 4: Introduce security testing
Explore Garak, PyRIT, Giskard and Promptfoo.
Create repeatable test cases and document what happens when the application encounters unexpected or malicious inputs.
Beyond Week 4: Build security into development
Add security checks whenever you introduce a new model, data source, tool integration or agent capability.
Security should evolve alongside the application.
6. The bigger picture: AI security is also a delivery responsibility
There is another dimension that deserves attention, particularly as enterprises move AI from experimentation into production.
AI security cannot remain isolated within the cybersecurity team.
It needs to become part of the entire software delivery lifecycle.
When an organization introduces an AI-powered application, several teams become responsible for different parts of its security:
- Product teams define intended AI behaviour, boundaries and user expectations.
- Architecture teams establish secure integration patterns and trust boundaries.
- Development teams implement authorization, validation and secure tool access.
- Data teams protect information quality, classification and access permissions.
- Security teams conduct threat modelling, adversarial testing and risk assessments.
- Delivery leaders ensure security requirements, testing, approvals and operational readiness are built into delivery plans.
This becomes even more important with agentic AI.
An agent may interact with multiple systems, make decisions and execute multi-step workflows.
Who approves its actions?
What permissions does it have?
How are its decisions monitored?
What happens when it behaves unexpectedly?
Where does human intervention become mandatory?
These questions cannot be left until the application is ready for production.
They need to be addressed during architecture, development, testing and deployment.
A useful principle is to establish explicit security gates for every major AI capability:
- Data access and privacy review.
- Model and prompt security evaluation.
- Tool and agent permission review.
- Adversarial and abuse-case testing.
- Human approval for high-impact operations.
- Production monitoring and incident response.
The goal isn't to slow down AI development.
It is to avoid building systems that are fast to deploy but difficult to trust, operate and govern.
Final thoughts
AI has changed how quickly we can build software.
But speed of development doesn't automatically translate into security, reliability or trust.
A chatbot that answers correctly is not necessarily secure.
A RAG application that retrieves relevant documents is not necessarily protecting information correctly.
An agent that successfully executes a workflow is not necessarily operating within acceptable permissions.
As AI moves deeper into business applications, the security conversation must move beyond the model itself.
The real question is no longer just: Can we build it with AI?
It is: Can we build it, secure it, test it and trust it when it operates in the real world?
That is a skill every serious AI builder needs to start developing.
And increasingly, it is a responsibility every technology delivery leader needs to understand.