AI Security Learning Journal — Day 17

How attackers can manipulate what an AI learns or retrieves without breaking the model, the application, or even the retrieval algorithm.

The most dangerous AI failure may not look like a failure

When I first thought about attacking an AI system, I naturally imagined something breaking.

A compromised model.

A malicious prompt.

A jailbreak.

A vulnerable application.

An abnormal response.

But studying data poisoning changed that mental model.

A poisoned AI system can continue running normally.

The application stays online.

The vector database responds.

The Retriever returns relevant documents.

The LLM produces fluent answers.

The infrastructure dashboard stays green.

And yet the integrity of the system's decisions may already be compromised.

That realization led me to a different question:

What if the attacker doesn't need to break the AI, but only needs to influence what the AI considers trustworthy?

Two very different ways to poison AI

One distinction became essential for me: training poisoning and retrieval poisoning are not the same attack.

With training data poisoning, malicious information participates in training or fine-tuning.

Conceptually:

Poisoned Training Data
        ↓
Training / Fine-Tuning
        ↓
Weight Updates
        ↓
Changed Model Behaviour

Poisoned Training Data
        ↓
Training / Fine-Tuning
        ↓
Weight Updates
        ↓
Changed Model Behaviour

The attack changes what the model learns.

That matters because removing the original malicious sample later does not necessarily undo the effect. Its influence may already exist in the learned parameters.

RAG introduces a different possibility.

Poisoned Knowledge Base
        ↓
Retrieval
        ↓
Manipulated Context
        ↓
LLM
        ↓
Changed System Behaviour

Poisoned Knowledge Base
        ↓
Retrieval
        ↓
Manipulated Context
        ↓
LLM
        ↓
Changed System Behaviour

Here, the underlying model does not need to change at all.

The attacker changes what the model sees during inference.

That gave me a simple way to remember the difference:

Training poisoning changes what the model learned. Retrieval poisoning changes what the model sees.

A compromised RAG system does not necessarily mean a compromised model

This distinction seems small until we approach the problem as incident responders.

Imagine an organization using a commercial LLM through an API.

The model is hosted externally.

Nobody inside the organization can modify its weights.

But the company maintains an internal knowledge base used by RAG.

If an attacker manipulates that knowledge base and the assistant begins producing incorrect recommendations, saying "the model was poisoned" may point the investigation in the wrong direction.

The model could be perfectly intact.

What was compromised was the information path surrounding it.

Model                  → Intact
Knowledge Corpus       → Manipulated
Retrieved Context      → Manipulated
AI System Behaviour    → Affected

Model                  → Intact
Knowledge Corpus       → Manipulated
Retrieved Context      → Manipulated
AI System Behaviour    → Affected

This reinforces something I learned while studying RAG security:

The security boundary of an AI application is larger than the model itself.

When semantic relevance becomes an attack primitive

RAG systems commonly retrieve information using vector similarity.

That creates an interesting security problem.

A Retriever does not inherently ask:

"Which document is trustworthy?"

It is primarily trying to answer something closer to:

"Which document is semantically closest to this query?"

Those are completely different questions.

Imagine a knowledge base containing one legitimate security procedure.

An attacker cannot modify it.

Instead, they introduce many documents containing a subtly weakened version of the same procedure.

All of those documents are designed around terminology likely to appear in employee queries.

Now the attacker is not trying to destroy retrieval.

They are trying to win retrieval.

User Query
     ↓
Similarity Search
     ↓
Attacker-Controlled Documents
     ↓
Top-k
     ↓
LLM Context

User Query
     ↓
Similarity Search
     ↓
Attacker-Controlled Documents
     ↓
Top-k
     ↓
LLM Context

The Retriever may be doing exactly what it was designed to do.

The architecture failed somewhere else.

This led me back to one of my most important lessons from RAG security:

Semantic similarity is a relevance signal, not a trust decision.

The attacker can manipulate density instead of the model

Corpus flooding made this even clearer.

Suppose an attacker adds dozens of semantically similar documents around a particular topic.

It can be tempting to think:

"The model saw the malicious idea many times, so it learned it."

But in a RAG corpus, that is not necessarily what happened.

There may have been no training.

No gradient update.

No change to model weights.

Instead:

More Attacker Documents
        ↓
Higher Semantic Density
        ↓
More Candidates Near the Query
        ↓
Greater Presence in Top-k
        ↓
Greater Influence on Context

More Attacker Documents
        ↓
Higher Semantic Density
        ↓
More Candidates Near the Query
        ↓
Greater Presence in Top-k
        ↓
Greater Influence on Context

The attacker manipulated retrieval probability, not necessarily model learning.

That distinction matters both for detection and remediation.

The ingestion pipeline is part of the security boundary

This was probably my biggest architectural takeaway from Day 17.

Consider a common enterprise workflow:

Corporate Storage
      ↓
Scheduled Ingestion
      ↓
Parsing
      ↓
Chunking
      ↓
Embedding
      ↓
Vector Database

Corporate Storage
      ↓
Scheduled Ingestion
      ↓
Parsing
      ↓
Chunking
      ↓
Embedding
      ↓
Vector Database

It looks like a data-engineering pipeline.

But from a security perspective, it is doing something much more important:

It determines which information becomes eligible to influence the AI.

Now imagine an employee legitimately has permission to upload documents to corporate storage.

An attacker compromises that employee's account.

The identity system sees an authenticated user.

RBAC confirms that the user may write to the folder.

The upload succeeds.

Then automation does the rest.

Compromised Identity
        ↓
Authorized Write
        ↓
Malicious Document
        ↓
Automatic Ingestion
        ↓
Embedding
        ↓
Retrieval-Eligible Knowledge

Compromised Identity
        ↓
Authorized Write
        ↓
Malicious Document
        ↓
Automatic Ingestion
        ↓
Embedding
        ↓
Retrieval-Eligible Knowledge

Every component may be functioning correctly.

That exposed a dangerous implicit assumption:

Authorized to Write
        =
Authorized to Influence AI

Authorized to Write
        =
Authorized to Influence AI

Those permissions should not automatically be equivalent.

Authorized to write does not mean authorized to influence AI context.

Automation can scale a trust mistake

This is where traditional cybersecurity and AI Security reconnect.

Automation itself is not the enemy.

But automated trust can propagate one security failure through an entire system.

One compromised identity can become one malicious document.

One malicious document can become multiple chunks.

Those chunks become embeddings.

Those embeddings become retrieval candidates.

Those candidates become LLM context.

That context can influence many users.

The system may have efficiently automated the attacker's influence without any component technically malfunctioning.

A lesson I keep rediscovering throughout this journey is:

Automation scales both good controls and bad assumptions.

The subtle attack worries me more

An attacker does not necessarily want the AI to produce something obviously malicious.

Imagine a SOC assistant normally recommending endpoint isolation when malicious confidence reaches 80%.

Then its behaviour slowly changes:

80% → 82% → 85% → 88% → 90%

80% → 82% → 85% → 88% → 90%

Nothing here looks absurd.

A recommendation requiring 90% confidence could even sound more cautious.

Now compare two AI responses.

One says:

"Disable all security controls."

A SOC analyst will immediately become suspicious.

But another says:

"Based on the available evidence, endpoint isolation may not yet be necessary. Additional monitoring is recommended."

That sounds reasonable.

Professional.

Cautious.

And potentially exactly what an attacker wants.

The dangerous chain becomes:

Poisoned Data
      ↓
Small Behavioural Shift
      ↓
Plausible Recommendation
      ↓
Human Trust
      ↓
Wrong Decision
      ↓
Attacker Benefit

Poisoned Data
      ↓
Small Behavioural Shift
      ↓
Plausible Recommendation
      ↓
Human Trust
      ↓
Wrong Decision
      ↓
Attacker Benefit

The attack does not need to make the AI look compromised.

It may be more successful when the AI continues looking trustworthy.

So how would I detect that?

This is where behavioural baselines became important to me.

In traditional security monitoring, we often establish normal patterns and investigate meaningful deviations.

The same idea can be applied to AI behaviour.

For a SOC assistant, I could maintain controlled scenarios with expected security properties.

Not exact sentences.

Not exact tokens.

Properties.

For example:

Scenario:
High-confidence malware
Expected property:
Recommend isolation

Scenario:
High-confidence malware
Expected property:
Recommend isolation

Then I periodically evaluate whether that property remains stable.

Because LLMs are probabilistic, the wording may change.

That is normal.

But if a stable security decision gradually changes from:

80 → 82 → 85 → 88 → 90

80 → 82 → 85 → 88 → 90

I now have something worth investigating.

Without history, 90% might look reasonable.

With history, I can ask:

Why did this security property progressively change?

But drift does not prove poisoning

This distinction is critical.

A behavioural baseline can tell me:

Something changed.

It cannot automatically tell me:

An attacker poisoned the system.

The cause could be:

a legitimate policy update;
an outdated document;
an accidental error;
a new model version;
a different embedding model;
retrieval configuration;
prompt changes;
lifecycle failure;
deliberate poisoning.

So my investigative chain becomes:

Anomaly
   ↓
Investigation
   ↓
Evidence
   ↓
Causal Reconstruction
   ↓
Attribution

Anomaly
   ↓
Investigation
   ↓
Evidence
   ↓
Causal Reconstruction
   ↓
Attribution

A sudden behavioural change after a corpus update is correlation.

It is valuable.

But correlation is not yet causation.

This turns provenance into forensic evidence

If I suspect poisoning, I want to reconstruct more than the user's prompt.

I want to know:

What did the user ask?
        ↓
What documents were retrieved?
        ↓
Which versions?
        ↓
When were they created or modified?
        ↓
Who submitted them?
        ↓
Who approved them?
        ↓
When were they ingested?
        ↓
What context reached the model?
        ↓
Which model and prompt versions were active?
        ↓
What response was generated?

What did the user ask?
        ↓
What documents were retrieved?
        ↓
Which versions?
        ↓
When were they created or modified?
        ↓
Who submitted them?
        ↓
Who approved them?
        ↓
When were they ingested?
        ↓
What context reached the model?
        ↓
Which model and prompt versions were active?
        ↓
What response was generated?

Knowing that policy.pdf was retrieved is useful.

Knowing which version of policy.pdf was retrieved at that exact moment is forensic evidence.

And Day 17 added another question to what I learned in Day 16:

How did that information become eligible to reach the model in the first place?

Similarity should happen after trust

This changed how I would design the retrieval pipeline.

Instead of:

All Documents
      ↓
Similarity Search
      ↓
Top-k

All Documents
      ↓
Similarity Search
      ↓
Top-k

I would rather think:

All Documents
      ↓
Provenance
      ↓
Authorization
      ↓
Validation
      ↓
Approval
      ↓
Lifecycle Check
      ↓
Eligible Knowledge
      ↓
Similarity Search
      ↓
Top-k

All Documents
      ↓
Provenance
      ↓
Authorization
      ↓
Validation
      ↓
Approval
      ↓
Lifecycle Check
      ↓
Eligible Knowledge
      ↓
Similarity Search
      ↓
Top-k

Similarity still matters.

But only after the system has determined which documents are allowed to compete.

My resulting principle is:

Similarity should rank eligible knowledge, not decide which knowledge deserves authority.

And what if every control fails?

This is where defense in depth matters.

Imagine the attacker succeeds.

The identity is compromised.

The malicious document enters the corpus.

Validation misses it.

Retrieval surfaces it.

The LLM believes the context.

The model generates an incorrect recommendation.

That still should not automatically mean operational impact.

For a high-impact action, I would want another independent chain:

LLM Recommendation
        ↓
Policy Enforcement
        ↓
Authorization
        ↓
Human Validation
        ↓
Execution

LLM Recommendation
        ↓
Policy Enforcement
        ↓
Authorization
        ↓
Human Validation
        ↓
Execution

The human should not blindly trust the AI.

But the human should not be the only security boundary either.

The model can be manipulated.

The human can make mistakes.

Individual controls can fail.

The architecture needs to survive those failures.

My biggest takeaway from Day 17

Before this lesson, I might have summarized data poisoning as:

"Someone puts malicious data into an AI system."

Now that feels incomplete.

The more useful questions are:

What data?

At what stage?

Who controlled it?

Did it participate in training?

Did it enter the retrieval corpus?

Did the model weights change?

Did only the context change?

How did the content become trusted?

Did behaviour actually change?

Was the change accidental or intentional?

Can I reconstruct the causal chain?

Those questions completely change how I would investigate the incident.

And they lead me to the idea I want to carry forward:

Poisoning attacks do not need to break the AI system. They can succeed by quietly changing the information the system considers trustworthy.

The infrastructure can remain healthy.

The model can remain intact.

The Retriever can work correctly.

The response can remain fluent.

The recommendation can remain plausible.

And the decision can still be wrong.

That is what makes data poisoning such an interesting AI Security problem.

The target is not always availability.

It is not always code execution.

It is not even always the model.

Sometimes the target is simply trust.

This article is part of my ongoing AI Security Learning Journey. I use these entries to document concepts in my own words, connect them with cybersecurity architecture and incident response, and record how my understanding evolves. Lab solutions, flags, proprietary exercises, credentials, and training material are intentionally excluded.

Suggested Medium tags:

AI Security · Cybersecurity · RAG · Artificial Intelligence · Data Poisoning

Contents