What an AI Penetration Test Actually Covers

The real scope starts where prompts meet private data, application permissions, and tools that can take action

Originally published at pentesttesting.com/what-does-an-ai-penetration-test-cover.

A support assistant was supposed to summarize customer tickets and recommend the next step.

It had no administrative dashboard. It did not have a shell. It could not "take over the company." On paper, that made the feature look low risk.

The architecture told a different story.

The assistant could search internal support records through an API. It used a shared backend service account. Search results were added to the model context before the user saw anything. The application assumed that the assistant would ask only reasonable questions and that the system prompt would keep it within policy.

That assumption was the weak point.

During a controlled assessment, an indirect instruction inside untrusted ticket content influenced what the assistant attempted to retrieve. The significant issue was not that the model followed unusual text. The significant issue was that the retrieval service did not independently enforce the requesting user's customer boundary.

The model exposed a flaw that already existed in the application's trust design.

This is why asking only, "Can the chatbot be jailbroken?" produces an incomplete AI security assessment.

The better question is:

What can untrusted input cause the complete AI-enabled system to access, reveal, or do?

An AI Pentest Is a System Test, Not a Model Conversation

Language models are only one component in production AI applications.

A real deployment may include:

• A browser or mobile interface
• Authentication and sessions
• Prompt templates
• Model APIs
• Retrieval pipelines
• Vector databases
• File ingestion
• Conversation memory
• Function calling
• Agent tools
• Service accounts
• Output parsers
• Logging and analytics
• Third-party integrations

An AI penetration test examines how these parts behave together under adversarial conditions.

That means the tester is not merely collecting prompts that make the model say something unexpected. The tester is tracing trust boundaries.

Where did the instruction come from?

Which identity was used?

What data was retrieved?

What authorization check ran before the data entered the prompt?

Which tool was available?

Could the tool perform a write action?

Was a human asked to confirm it?

Did the backend independently verify that the user was authorized?

Those questions are where meaningful findings usually appear.

"Prompt injection matters most when it crosses a real security boundary. The report should show the data, permission, or action at risk, not just an unusual model response."

— Shofiur Rahman, CEH, CEO of Pentest Testing Corp

The First Deliverable Should Be a Trust Map

Before testing prompts, the assessor should map the AI workflow.

This does not need to become a fifty-page architecture document. It does need to identify the path from input to impact.

A useful map includes:

Input sources User prompts, files, email, tickets, web content, database records, retrieved documents, and tool responses.

Model context System instructions, developer prompts, conversation history, retrieved passages, user profile data, and tool descriptions.

Identity User identity, tenant, role, API token, service account, and model-provider credentials.

Data sources Relational databases, object storage, vector stores, document repositories, CRM records, and third-party APIs.

Actions Read, search, summarize, create, update, send, delete, approve, deploy, or execute.

Controls Authorization, validation, approval, rate limits, logging, monitoring, and emergency stop mechanisms.

Without this map, testing tends to over-focus on the visible chatbox.

The most important weakness may be one service call behind it.

What the Scope Normally Covers

1. Prompt Injection With Business Impact

The OWASP Top 10 for LLM Applications 2025 lists LLM01:2025 Prompt Injection as a core risk. The issue arises when inputs alter model behavior or output in unintended ways.

An assessment should test both direct and indirect forms.

Direct prompt injection comes from the user interacting with the model.

Indirect prompt injection comes from content the model reads, such as a document, support ticket, web page, email, or retrieved knowledge-base entry.

But a responsible assessment does not exaggerate every successful instruction override.

If the model becomes rude, changes tone, or discusses an unwanted topic, the issue may be a policy or product-control problem.

Security impact becomes clearer when the model:

• Retrieves information outside the user's permission
• Reveals sensitive context
• Selects an unauthorized tool
• Alters a protected workflow
• Causes unsafe output to reach another component
• Avoids a control that should be enforced outside the model

The system prompt is not an authorization layer.

It can guide behavior. It cannot replace server-side access control.

2. Retrieval and RAG Security

RAG gives a model access to current, private, or domain-specific information.

It also creates new security questions.

Can one tenant retrieve another tenant's content?

Are document permissions applied before retrieval?

Can deleted records remain searchable?

Can a user influence the knowledge base?

Can hidden instructions inside indexed content affect the model?

Do staging documents appear in production search?

Are vector database APIs exposed?

Does a shared backend identity bypass the user's own permissions?

RAG testing often looks less like "AI hacking" and more like careful authorization testing across a new data path.

The difference is that natural-language search can make hidden access-control problems easier to discover and exploit.

A user may not know a record ID. They may still ask the assistant for "all contracts mentioning a renewal next month" and receive data the application should have filtered out.

3. Agents, Tools, and Excessive Agency

OWASP LLM06:2025 Excessive Agency applies when an AI-enabled system can call functions, plugins, or connected services with more power, autonomy, or permission than is safe.

This is the highest-value area in many agentic assessments.

The tester should inventory each tool and ask:

• Does the agent need this tool?
• Does the tool expose more operations than required?
• Does it run with the user's identity or a shared privileged identity?
• Are tool parameters validated?
• Is authorization enforced by the backend?
• Are destructive actions blocked?
• Is human confirmation required?
• Does the confirmation show the exact action?
• Can one tool's output influence another tool?
• Are loops and repeated actions limited?
• Do failures stop safely?

A model should propose. A trusted application control should decide whether the proposal is allowed.

The dangerous pattern is giving a probabilistic component broad authority and then relying on prompt wording to keep it safe.

4. Authentication and Authorization

Traditional controls remain critical.

An AI feature still depends on:

• Sessions
• Tokens
• Roles
• Tenant boundaries
• Object ownership
• API keys
• Service accounts
• Logout behavior
• File permissions
• Conversation access

A standard web or API pentest may cover some of this, but the AI assessment asks an additional question:

Can the model or orchestration layer route around the control?

For example, the user interface may hide a restricted record. The assistant may still retrieve it through a backend search function.

The control must follow the data path, not only the screen.

5. Sensitive Information Exposure

AI systems can duplicate sensitive data across multiple places.

A customer record may appear in:

• The source database
• A retrieval index
• The prompt sent to the model provider
• Conversation memory
• Application logs
• Analytics
• Traces
• Tool-call history
• Cached responses

The assessment should determine where sensitive data travels, who can access it, how long it is retained, and whether it is needed.

It should also separate harmless prompt disclosure from real secret exposure.

A leaked instruction such as "answer in a professional tone" is not a critical finding.

A leaked API key, private document, customer record, or hidden authorization rule may be.

6. Output Handling

Model output is untrusted data.

That remains true even when the application created the prompt and trusts the model provider.

The output may be:

• Rendered in a browser
• Parsed as JSON
• Stored in a ticket
• Added to a database query
• Sent to another tool
• Used as a workflow definition
• Displayed as an approval request
• Passed to an interpreter

Each destination needs the same validation, encoding, and authorization discipline applied to any other untrusted input.

The risk is not "the model generated text."

The risk is "another system treated that text as trusted instructions or data."

7. Abuse, Cost, and Availability Controls

AI systems have unusual resource costs.

A small number of requests may trigger:

• Very long contexts
• Large document processing
• Repeated retrieval
• Multiple model calls
• Recursive tool use
• Expensive fallback models
• High outbound API usage

Testing can validate rate limits, token caps, per-user quotas, file limits, iteration limits, concurrency controls, and alerting.

This should be done safely.

A penetration test should prove that controls are missing or ineffective without causing a production outage or unexpected cloud bill.

8. Logging and Incident Visibility

A team cannot investigate what it did not record.

Useful AI security logs may include:

• User and tenant identity
• Model and prompt-template version
• Retrieval source identifiers
• Tool name
• Tool parameters
• Authorization decision
• Confirmation event
• Correlation ID
• Rate-limit event
• Guardrail event
• Administrative configuration change

The logs themselves require protection because prompts and responses may contain private data.

A good assessment asks both questions:

1. Can the organization detect meaningful abuse?
2. Can it do so without creating a new sensitive-data repository?

A Sanitized Case Study: The Finding Was Not "The Model Obeyed"

Consider an anonymized document assistant used inside a multi-tenant SaaS platform.

The assistant could search uploaded documents and create follow-up tasks. Users belonged to separate customer workspaces.

The initial assumption was that the main risk involved prompt injection.

During testing, a document containing untrusted instructions influenced the assistant's retrieval request. That alone was not the final finding.

The deeper issue appeared when the retrieval API used a broad service account and accepted a workspace identifier from the orchestration layer without independently checking the user's membership.

The agent then prepared a follow-up task containing a short excerpt from a document outside the user's workspace.

No task was actually created during the assessment. The proof was stopped at a safe validation point.

The report described an attack chain:

1. Untrusted document content entered the model context.
2. The model generated an unintended retrieval request.
3. The retrieval service trusted a workspace value from the AI layer.
4. The backend failed to enforce user-to-workspace authorization.
5. The agent was permitted to prepare a write action without a precise approval boundary.

Several controls were needed:

• Treat retrieved content as untrusted data
• Enforce tenant authorization inside the retrieval service
• Derive workspace identity from the authenticated session
• Limit the agent to read-only operation where possible
• Require an accurate preview before task creation
• Record the document source, retrieval decision, and tool call

Calling this only "prompt injection" would have hidden the engineering work required to fix it.

"The model can be the trigger, but the application determines the blast radius."

— Shofiur Rahman, CEH, CEO of Pentest Testing Corp

What the Test Does Not Cover Automatically

Clear exclusions prevent false confidence.

It Does Not Eliminate Hallucinations

A security test may examine misinformation when it creates a concrete risk.

It does not guarantee that every future answer will be accurate.

Model evaluation, factuality testing, and product-quality validation may require separate work.

It Is Not a Full Governance Audit

Security testing does not automatically cover fairness, bias, explainability, workforce impact, accessibility, copyright, procurement, or broader responsible-AI governance.

The NIST AI Risk Management Framework can help organizations structure governance, mapping, measurement, and risk treatment. A penetration test contributes technical evidence, but it is only one input.

It Is Not a Compliance Certificate

The report may support SOC 2, ISO 27001, HIPAA, PCI DSS, or customer security reviews.

It does not certify compliance.

It Does Not Include the Entire Technology Estate

The assessment may include only the AI feature, its APIs, retrieval system, and tools.

Cloud infrastructure, corporate networks, mobile applications, and unrelated web features need separate scope unless they are explicitly included.

It Does Not Include Source Review by Default

Black-box and gray-box testing can identify exploitable behavior.

Source-assisted review can add value for prompt construction, tool definitions, retrieval filters, and authorization middleware, but it should be named in the statement of work.

It Does Not Prove There Are No Vulnerabilities

The result is time-bound.

Models change. Prompts change. Tools change. Data changes. Permissions change.

The report should document the versions, roles, environments, and limitations that applied during testing.

AI Pentest Deliverables That Engineers Can Use

A useful report should include more than a severity label.

Look for:

• A clear tested scope
• Architecture and trust-boundary summary
• Executive risk summary
• Prioritized findings
• Sanitized evidence
• Affected roles and components
• Business impact
• Technical root cause
• OWASP LLM mapping
• Specific remediation
• Fix-validation criteria
• Retest results
• Documented limitations

Findings should describe complete paths.

For example:

Weak report: "Prompt injection was successful."

Useful report: "Untrusted document content influenced retrieval behavior. The retrieval API then returned cross-tenant content because it trusted a caller-supplied workspace identifier and used a shared service account. The agent could prepare a write action without a specific confirmation step."

The second version tells engineers where to work.

It also helps leadership understand why a prompt-only fix would be insufficient.

How to Prepare for Scoping

Before speaking with a provider, collect:

• A short description of the AI feature
• Model provider and model family
• User roles
• Tenant design
• Test environment
• AI-related APIs
• RAG sources
• Vector database
• File-ingestion paths
• Agent tools
• Service accounts
• Sensitive data categories
• Human approval points
• Rate and cost limits
• Existing logs
• Recent architecture changes
• Compliance or customer deadlines

Also define safety boundaries.

Testing should have agreed environments, permitted hours, prohibited actions, rate thresholds, emergency contacts, and stop conditions.

This is particularly important when the AI can send messages, alter records, trigger cloud workflows, or call paid external services.

The Practical Definition

So, what does an AI penetration test actually cover?

It covers the route from untrusted input to a security consequence.

That route may pass through a language model, but it rarely ends there.

The meaningful questions involve:

• Which data the AI can reach
• Which identity it uses
• Which tools it can call
• Which actions require approval
• Which controls exist outside the model
• Which outputs another system trusts
• Which evidence the organization can investigate

The assessment should test those boundaries and explain what was not tested.

That is more useful than a long list of jailbreak attempts. It is also a more realistic view of how AI-enabled applications fail.

Get a Redacted Sample Report

Review the Pentest Testing Corp AI penetration testing service and request a free redacted sample report to see the expected structure for evidence, impact, remediation, framework mapping, and retesting.

For a defined system, you can also book a 30-minute scoping call.