October 29, 2026

Harness Engineering: Engineering Autonomous AI Agents for Offensive Security

Why building powerful AI agents is not enough and how harness engineering can change the future of autonomous security testing.

By VETTRIVEL

9 min read

1. Introduction: The Problem Is Not Always the Model

Artificial intelligence is changing how we approach offensive security.

Today, we can use large language models (LLMs) to understand source code, analyses HTTP requests, identify suspicious execution paths, generate test cases, investigate vulnerabilities and assist with penetration testing.

We can also connect these models to tools such as Burp Suite, Nmap, browser automation frameworks, source code analyser's, debuggers, and custom security tools via MCP.

At first glance, building an autonomous penetration testing agent seems straightforward.

Give an AI model access to security tools, provide it with a target, and ask it to find vulnerabilities.

But anyone who has experimented with AI agents for real security work will quickly discover a problem.

An agent that can perform a task once is not necessarily an agent that can perform the same task reliably.

An agent might identify an interesting vulnerability in one execution and miss it in another. It might misunderstand an HTTP response, repeat the same failed test, forget an important observation, or claim that a vulnerability exists without sufficient evidence.

Sometimes, it selects the wrong tool. Sometimes, it understands the vulnerability but cannot construct a valid test. Sometimes, it performs the correct test but interprets the result incorrectly.

Increasing the model's size does not automatically solve these problems. Fine-tuning the model only does not automatically solve them either.

Adding more tools can make the situation worse if the agent does not understand when and how to use them.

This raises an important question:

What if the biggest limitation of an AI security agent is not the model itself, but the engineering system surrounding it?

This is where harness engineering becomes important.

Harness engineering is the practice of designing the complete environment in which an AI model operates. It includes the tools the model can access, the instructions it receives, the state it maintains, the way it executes tasks, the evidence it collects, the errors it handles, and the rules it must follow before reaching a conclusion.

In offensive security, this means building a system that can investigate a target systematically, maintain its findings, recover from failures, verify vulnerabilities, and produce conclusions supported by evidence.

The objective is not simply to make an AI model smarter. The objective is to make the entire system more reliable.

2. What Is Harness Engineering?

To understand harness engineering, we first need to understand the difference between a model, an agent, and a harness.

The model

The model is the reasoning engine.

It interprets information, generates plans, analyses source code, proposes hypotheses and decides what to investigate next.

For example, when examining a web application, a model might identify a possible server-side request forgery (SSRF) vulnerability based on how the application processes user-controlled URLs.

However, the model's conclusion is only a hypothesis until the behaviour can be verified.

The agent

An agent uses the model to work towards a goal.

It can receive a task, select tools, inspect results, make decisions, and continue investigating.

For example, an agent might:

  1. Inspect an API endpoint.
  2. Identify a URL parameter.
  3. Examine the relevant source code.
  4. Form an SSRF hypothesis
  5. Select an appropriate test.
  6. Analyse the response.
  7. Determine whether further investigation is necessary.

The agent provides the decision-making loop that connects the model to actions.

The harness

The harness provides the engineering infrastructure around that loop.

It determines how the agent receives context, accesses tools, stores information, executes tasks, handles errors, verifies results, and records its decisions.

A well-engineered harness can provide:

  • Controlled tool access.
  • Structured task definitions.
  • Persistent investigation state.
  • Execution limits and timeouts.
  • Failure recovery.
  • Validation and verification.
  • Model and tool evaluation.
  • Reproducible execution.
  • Audit logs and reporting.
  • Human approval for sensitive actions.
  • Etc..

The harness does not replace the model. It gives the model a structured environment in which to operate.

Consider a simple analogy.

A model is like an engineer who can reason about a security problem. An agent is that engineer performing an investigation. The harness is the laboratory, instrumentation, documentation, access control, testing procedure and quality assurance system surrounding the work.

Even a highly skilled engineer can produce unreliable results in a poorly organised laboratory.

The same principle applies to AI agents.

The model provides intelligence. The harness provides operational reliability.

This distinction is particularly important in offensive security because a false conclusion can waste investigation time, produce an invalid vulnerability report, or lead to an unsafe action against a target.

3. Why Offensive Security Needs Harness Engineering

Offensive security is not a single operation.

A realistic security assessment involves multiple stages, including reconnaissance, attack surface mapping, vulnerability analysis, validation, exploitation where authorised, impact assessment and reporting.

Each stage depends on information collected during earlier stages.

For example, discovering an API endpoint is only the beginning. An researcher might need to understand its authentication requirements, identify the relevant source code, inspect the data flow, examine the application's response and determine whether the behaviour creates a security impact.

If an agent loses context between these stages, the investigation becomes fragmented.

If it trusts an unverified assumption, the entire attack hypothesis may become invalid.

If it cannot distinguish a failed tool execution from a negative security finding, it may incorrectly conclude that the application is secure.

These are engineering problems, not simply language-generation problems.

Problem 1: Inconsistent execution

Problem 2: Repeated failures

Problem 3: Hallucinated findings

Problem 4: Lost investigation context

Problem 5: Uncontrolled actions

Problem x: Etc

4. The Difference Between Prompt Engineering and Harness Engineering

Prompt engineering remains useful, but it solves only part of the problem.

A prompt can tell an agent to inspect source code carefully, avoid unsupported conclusions, verify findings, and document its results.

However, instructions alone cannot guarantee that the agent will follow them.

Imagine giving an agent this instruction:

"Do not report a vulnerability unless it has been verified."

The instruction is sensible, but what happens if the model misunderstands the evidence? What happens if the agent forgets the instruction after a long investigation?

A prompt cannot independently enforce all the necessary controls.

Harness engineering turns important instructions into executable system behaviour.

Prompt engineering helps communicate the task. Harness engineering builds the mechanisms that make successful execution more consistent.

The same applies to fine-tuning.

A model trained to recognise common vulnerability patterns may produce better initial hypotheses. But it still needs reliable tools, structured state, and a way to validate those hypotheses.

The goal is not to replace prompting with engineering. It is to move critical requirements from instructions into enforceable system behaviour.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

5. A Reference Architecture for an Offensive Security Harness

Before discussing individual components, it helps to visualise the complete system.

A practical architecture can be organised into the following layers.

USER / SECURITY OBJECTIVE
                               |
                               v
                    1. TASK AND SCOPE LAYER
                               |
                               v
                     2. ORCHESTRATION LAYER
                               |
                               v
                       3. REASONING LAYER
                         LLM / AGENT
                               |
                               v
                       4. TOOL GATEWAY
                               |
                 +-------------+-------------+
                 |             |             |
                 v             v             v
              Source         Browser       Network
              Analysis       Testing       Analysis
                 |             |             |
                 +-------------+-------------+
                               |
                               v
                       5. RESULT NORMALISER
                               |
                               v
                        6. EVIDENCE STORE
                               |
                               v
                    7. VERIFICATION ENGINE
                               |
                               v
                      8. VERDICT ENGINE
                               |
                               v
                     9. REPORTING PIPELINE
       Supporting every layer:
       - Durable state and memory
       - Failure and recovery management
       - Policy enforcement
       - Evaluation and observability
       - Resource and execution budgets
       - Human approval where required

This is a reference architecture, not a mandatory design.

Some implementations may combine several layers. Others may use multiple specialised agents or independent verification services.

The important point is that each responsibility should be explicit.

The reasoning model should not be responsible for remembering everything, enforcing every security policy, managing every tool failure, and deciding whether its own claims are correct.

Those responsibilities should be distributed across components that can be tested independently.

Let's examine each layer.

6. Layer One: Task Definition and Scope Management

Every security investigation should begin with a clearly defined objective.

Consider these two instructions:

"Find vulnerabilities in this application."

"Review the authorised staging application and its source repository for authentication and authorisation weaknesses. Use approved test accounts, follow the defined request limits, record reproducible evidence, and stop before destructive actions."

The second instruction gives the agent a much stronger foundation.

A harness should convert the user's objective into a structured task specification.

A useful specification includes:

  • Target identifiers.
  • Authorised domains, IP addresses, applications, or repositories.
  • Permitted testing methods.
  • Prohibited actions.
  • Available credentials and their permitted use.
  • Resource and time budgets.
  • Required evidence.
  • Expected deliverables.
  • Conditions requiring human approval.
  • Explicit stopping conditions.

This specification should be machine-readable wherever possible.

For example, the orchestration layer should be able to determine whether a requested hostname is in scope before allowing a network tool to access it.

It should not depend solely on the model remembering which targets are permitted.

This distinction matters because an AI model can generate a reasonable plan that still includes an unauthorised action.

The harness must evaluate the action against the approved scope independently.

Scope should be enforced at the execution boundary

A prompt saying "only test these hosts" is not sufficient protection.

A stronger design checks the target whenever a tool call is made.

If the destination is outside the authorised scope, the gateway rejects the action and records the reason.

The same principle applies to sensitive operations.

For example, a harness might permit read-only source inspection automatically but require approval before a test could modify application data.

The exact controls depend on the assessment rules.

The model may propose an action, but the harness determines whether that action is permitted.

7. Layer Two: Orchestration and Workflow Management

Orchestration determines how the investigation progresses.

A simple agent may operate in a loop:

  1. Receive a task.
  2. Reason about the next action.
  3. Call a tool.
  4. Inspect the result.
  5. Decide what to do next.

This pattern is useful, but unrestricted loops can become inefficient.

An agent might repeatedly inspect the same file, retry an unsuccessful operation, or continue investigating an irrelevant hypothesis.

A harness should provide a workflow that defines the important stages of an investigation without forcing every decision into a rigid script.

For example, an application security workflow might include:

Stage A: Discovery

Identify the available source files, application endpoints, frameworks, and relevant components.

Stage B: Analysis

Develop hypotheses about potential weaknesses and identify the evidence needed to evaluate them.

Stage C: Investigation

Use approved tools to inspect the relevant code paths or application behaviour.

Stage D: Verification

Evaluate whether the observed behaviour satisfies the requirements for a vulnerability finding.

Stage E: Reporting

Produce a structured finding with evidence, impact, confidence, and remediation guidance.

Each stage should have entry conditions and exit conditions.

For example, a finding should not move to the reporting stage as a confirmed vulnerability merely because the analysis stage produced a plausible explanation.

It should satisfy the verification requirements defined for that vulnerability class.

Why not use a completely fixed workflow?

Because security testing is exploratory.

A useful discovery may reveal a new attack surface that requires additional investigation. A failed test may suggest a different hypothesis. A source code finding may justify a more targeted dynamic test.

The harness should therefore support both structured workflows and adaptive reasoning.

A good approach is to use bounded autonomy.

The harness defines the permitted actions, resource limits, important checkpoints, and completion requirements. The model chooses among the available options within those boundaries.

This provides flexibility without allowing the investigation to become uncontrolled.

8. Layer Three: Model Selection, Fine-Tuning, and Specialisation

One of the most common mistakes in agent development is assuming that the model must be improved before the surrounding system can be improved.

Model quality matters, but it is only one part of the system.

A security harness may use different models for different tasks.

For example:

  • A fast model for classifying tool output.
  • A code-capable model for source analysis.
  • A stronger reasoning model for complex vulnerability hypotheses.
  • A separate model for report drafting.
  • Deterministic tools for parsing, validation, and evidence comparison.

The best configuration depends on cost, latency, accuracy, context requirements, and task complexity.

Where fine-tuning helps

Fine-tuning can improve a model's performance on specific, well-defined tasks.

For example, a model may be trained on high-quality examples of:

  • Security-relevant data-flow analysis.
  • Vulnerability classification.
  • Framework-specific source code interpretation.
  • Structured tool-call generation.
  • Distinguishing confirmed findings from unverified hypotheses.
  • Security report formatting.
  • Mapping evidence to established vulnerability categories.

The quality of the training data matters more than simply increasing the number of examples.

If the dataset contains weak vulnerability claims, incorrect classifications, or misleading evidence, fine-tuning may teach the model to reproduce those mistakes more consistently.

A strong dataset should include both positive and negative examples.

It should also include ambiguous cases where the correct answer is "insufficient evidence."

Fine-tuning is not the same as reducing hallucination

Suppose a model learns to recognise common authentication bypass patterns.

It may become better at identifying suspicious code. But it could still incorrectly claim that an authentication bypass is exploitable when the relevant route is protected by middleware or an upstream authorisation check.

The missing information may not be a training problem. It may be a context problem.

The agent needs to inspect the relevant middleware, understand the execution path, and evaluate the applicable conditions.

In that situation, better retrieval and tool use may be more effective than fine-tuning.

Similarly, a model cannot reliably verify a remote vulnerability if it has no appropriate access to the target or no way to observe the necessary behaviour.

No amount of fine-tuning can replace missing evidence.

A practical improvement strategy

Before fine-tuning a model, determine where the errors originate.

Ask:

  1. Did the model misunderstand the task?
  2. Was relevant context missing?
  3. Did it select the wrong tool?
  4. Did the tool return an ambiguous result?
  5. Did the agent misinterpret the result?
  6. Was the verification procedure incomplete?
  7. Was the model genuinely unable to reason correctly from sufficient information?

Only the final category strongly suggests that a model capability improvement may be necessary.

The others may be better addressed through harness engineering.

9. Layer Four: Tool Gateway Engineering

A tool gateway is one of the most important components of a reliable security agent.

It acts as the controlled interface between the agent and the external tools it uses.

Without a gateway, an agent may interact with tools through inconsistent interfaces, poorly validated arguments, and unstructured output.

This makes failures difficult to diagnose and behaviour difficult to reproduce.

A tool gateway creates a common execution boundary.

A powerful model can generate ideas, analyse code, and suggest attack paths. But building a reliable offensive security agent requires much more than intelligence.

The real challenge is engineering the system around the model.

In Part 2, we will explore the technical architecture of harness engineering, covering tool gateways, durable memory, evidence management, verdict engineering, failure recovery and reusable harness specifications.

Stay tuned for Part 2: Under the Hood — Engineering a Reliable Offensive Security Harness.

I'll catch up with you later with another interesting topic.

Stay tuned, and feel free to connect with me on LinkedIn: VETTRIVEL U.

Thank you for reading! : )