September 3, 2026

AI Security 101: From “What is a Neural Network” to Actually Attacking and Defending One

A working reference for security folks getting into AI/ML testing — basics, attack techniques, defenses, and the tools/commands to try them…

By Jatin yadav

4 min read

A working reference for security folks getting into AI/ML testing — basics, attack techniques, defenses, and the tools/commands to try them yourself.

Why this matters

AI is quietly becoming part of the attack surface in every environment we assess — chatbots, RAG-powered internal tools, AI coding assistants, agentic automations with API/tool access. Traditional VAPT skills (input validation thinking, auth bypass thinking, supply-chain thinking) transfer directly — you're just applying them to a new kind of "application logic" that happens to be a trained model instead of code.

This post is the reference I wish I had starting out: the basics, mapped straight to how you'd break and then defend each piece, with real tools and commands.

1. The Basics, Fast

  • AI — a system performing tasks that normally need human reasoning.
  • ML (Machine Learning) — a subfield of AI where the system learns patterns from data instead of being explicitly coded.
  • ML lifecycle — define problem → collect/clean data → train → evaluate/tune → deploy → monitor/retrain. It's a loop, not a one-off.
  • Overfitting — model memorizes training data instead of generalizing. Matters for security because overfit models leak more about their training data when probed.
  • Neural networks / Deep learning — layered models (input → hidden layers → output) that learn complex, non-linear patterns. The "black box" nature is exactly what makes auditing them hard.
  • LLMs (Large Language Models) — neural networks trained on massive text corpora to predict/generate language, now wired into agents that can call tools, browse, and take actions.

Keep this mental model: data in → model → decision out. Every stage of that pipeline is something an attacker can target.

2. Offensive Side — Attacking AI Systems

a) Prompt Injection (the SQLi of the LLM world)

Manipulating the model's behavior through crafted input — direct (you type it) or indirect (hidden in a webpage, PDF, or email the AI reads later).

Try it with:

# Garak - LLM vulnerability scanner (like nmap/nuclei for LLMs)
pip install garak
garak --model_type openai --model_name gpt-4o-mini --probes promptinject
# Promptfoo - config-driven red-teaming for LLM apps
npm install -g promptfoo
promptfoo redteam init
promptfoo redteam run

b) Jailbreaking

Bypassing safety guardrails via roleplay framing, encoding tricks (base64, leetspeak), multi-turn escalation, or "DAN"-style persona prompts.

Try it with:

# PyRIT (Microsoft's Python Risk Identification Toolkit)
pip install pyrit
# Comes with attack strategies + orchestrators for automated jailbreak testing
# Garak again, targeted probes
garak --model_type openai --model_name gpt-4o-mini --probes dan,encoding

c) Adversarial Examples (evasion attacks)

Subtly perturbing an input (image pixels, text tokens) so a classifier misfires — e.g. malware sample that "looks benign" to an ML-based AV engine.

Try it with:

# Adversarial Robustness Toolbox (ART) - IBM's library for evasion/poisoning attacks
pip install adversarial-robustness-toolbox
# TextAttack - adversarial attacks specifically for NLP models
pip install textattack
textattack attack --recipe textfooler --model bert-base-uncased-imdb --num-examples 10

d) Data Poisoning

Injecting malicious/mislabeled samples into training data (or a RAG knowledge base) to bias future behavior — including planting a backdoor "trigger" phrase.

Where to look: if you're testing a RAG-based internal assistant, check whether the ingestion pipeline validates and sanitizes documents before they're indexed — that's your poisoning entry point.

e) Model Extraction / Theft

Repeatedly querying a model to reconstruct a functional clone (stealing IP without ever touching the weights directly), or reconstructing training data via membership inference.

Try it with:

# ART also supports model extraction attack simulations
# Rate-limit testing: hammer the inference API and watch response entropy/consistency

f) Malicious Model Files (supply chain)

Pretrained models shared as .pkl/pickle files can execute arbitrary code on load — classic deserialization attack, just wearing an AI hat.

Try it with:

# Fickling - detect malicious pickle files before loading a model
pip install fickling
fickling --check-safety suspicious_model.pkl
# ModelScan - scan model files (pickle, H5, SavedModel, etc.) for embedded exploits
pip install modelscan
modelscan -p /path/to/model

3. Defensive Side — Securing AI Systems

Attack Defense Prompt injection Input/output filtering, isolate system prompts from user input, treat all external content (web pages, docs) as untrusted Jailbreaking Layered guardrails (input classifier + output classifier), continuous red-teaming before and after release Adversarial evasion Adversarial training, input normalization, ensemble models Data poisoning Data provenance/lineage tracking, access controls on ingestion pipelines, anomaly detection on new training/RAG data Model extraction API rate-limiting, output rounding/noise, query-pattern anomaly detection Malicious model files Verify checksums/signatures, scan before load, prefer safetensors over pickle Excessive agent permissions Least-privilege tool access, human-in-the-loop for sensitive actions, sandbox execution

Guardrail / monitoring tools worth knowing:

# LLM Guard - input/output scanning, PII redaction, prompt injection detection
pip install llm-guard
# Rebuff - prompt injection detection specifically
pip install rebuff
# NeMo Guardrails - NVIDIA's programmable guardrails framework
pip install nemoguardrails
# Giskard - automated testing/scanning for ML models & LLM apps (bias, security, robustness)
pip install giskard

4. Frameworks to Anchor Everything To

Don't freelance the taxonomy — map findings to standards the same way you'd map a web app pentest to OWASP Top 10:

  • OWASP Top 10 for LLM Applications — Prompt Injection, Insecure Output Handling, Training Data Poisoning, Model DoS, Supply Chain, Sensitive Info Disclosure, Insecure Plugin Design, Excessive Agency, Overreliance, Model Theft.
  • MITRE ATLAS — adversarial ML threat matrix (like ATT&CK, but for AI systems). Great for report language and TTP mapping.
  • NIST AI Risk Management Framework (AI RMF) — governance-level, useful when tying findings back to a risk register.
  • ISO/IEC 42001 — AI management system standard, if the org already runs ISO 27001 this slots in naturally.

5. A Simple Learning Path

  1. Understand the basics — ML lifecycle, overfitting, neural nets, LLM training (this room/post).
  2. Learn the OWASP LLM Top 10 — vocabulary + mental model for findings.
  3. Run Garak or Promptfoo against a test model — see prompt injection/jailbreak attempts land or fail in real time.
  4. Scan a model file with ModelScan/Fickling — supply-chain angle, feels very "traditional pentest" and builds intuition fast.
  5. Read a few MITRE ATLAS case studies — real-world incidents mapped to techniques.
  6. Try building one guardrail (LLM Guard or NeMo Guardrails) — defense side sticks better once you've had to configure one yourself.

Quick Command Cheat-Sheet

# Offensive
garak --model_type openai --model_name <model> --probes promptinject,dan,encoding
promptfoo redteam run
textattack attack --recipe textfooler --model <model>
fickling --check-safety <model.pkl>
modelscan -p <model_path>
# Defensive
pip install llm-guard rebuff nemoguardrails giskard