It Can’t Be Turtles All the Way Down: Defence in Depth for AI Oversight

The labs' first answer to the Hugging Face incident is a second model that reads the first model's reasoning, and it needs evidence beside it that does not pass through the model's own writing.

Original: https://www.eigan.ai/blog/2026-10-15-ai-defence-in-depth

Our post of October 1 told the story of the now well fabled Hugging Face incident:1

• 1,200 OpenAI agents
• an improvised message board
• a cheat shared within hours
• five days spent hiding it from a reviewer that did not exist
• about 700 of them inside a company that had nothing to do with any of it

It closed on a requirement and a hope.

The requirement: you need to see what an agent is doing while it does it, not just what it did after its final output.

The hope: these models are not the black boxes we once thought, and interpretability research now makes that kind of real-time measurement possible, a control plane that did not exist before.

This post picks up there. Two investigation reports are now out,2, 3 so the question moves from whether real-time observation is possible to what the labs are actually deploying, what it gives us, and how much further we have to go.

The short answer is that the first thing being deployed is a second model that reads the first model's reasoning and flags anything out of policy. That is a sensible first step. It is also a model reading text written by a model. The model that writes the reasoning can be trained, or can learn, to write something other than what it is doing. The model that reads it can be talked round. Neither end is a fixed instrument. The people who build these systems said so a year before the incident. A July 2025 position paper, co-authored by researchers from OpenAI, Google DeepMind, Anthropic and other labs along with academics, called reading a model's reasoning "a new and fragile opportunity" and recommended it "only as an addition to existing safety research directions and not as a replacement."4 The incident is a real world example of why. A reasoning monitor on its own is not enough, and it needs evidence beside it that does not pass through the model's own writing.

What a layered design means here

Security people call this defence in depth. No single check is trusted on its own, so several independent checks are stacked, and each catches what the others miss. Eigan's forthcoming reference architecture for AI security applies that idea to an AI system.8 It follows a request through four stages and adds two things that sit alongside the whole path.

• The input boundary. What goes into the model: prompts, documents, tool results. Checks here catch bad instructions before the model sees them.
• The harness and control plane. The software that runs the model and decides which tools it may call, what credentials it holds and what it can touch. This is where a decision becomes an action, so it is where an action can be stopped.
• Internal-state monitoring. Reading the model's internal activity directly, as it runs, rather than the text it produces. This is the layer Eigan is researching.
• The output boundary. What comes out: answers, written reasoning, proposed tool calls. A reviewer that reads the model's reasoning works here.
• Observability. The records: logs, a baseline of what normal looks like, alerts, and whether those records can be tampered with.
• Supply chain. Where the model, its weights and its tools came from. This is not examined in this post.

The reasoning monitor OpenAI describes is one check at the output boundary. The rest of this post is about why it needs the others.

Model-based judges

OpenAI's report closes with a plan of action. One part is an expansion of chain-of-thought monitoring, where a monitor reads the reasoning a model writes out before it acts. OpenAI's monitors "examine available reasoning, tool actions, and sequences of activity" for signs of deception, reward hacking, data exfiltration and safeguard circumvention, and are to cover every workload where a model is connected to tools. The most severe alerts page responders around the clock. If responders cannot conclude within 30 minutes that a flag is a false positive, they are expected to pause the activity. OpenAI also describes work under way on infrastructure that would stop an evaluation automatically if no one responds.3

The report also says what would have happened if this monitoring had been running in July. OpenAI's assessment is that it "would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems."3 The report notes that reasoning monitoring had historically focused on internal coding use rather than research workloads.3

A related mechanism appears in the same report. OpenAI's "auto-review" mode asks a second model to review tool calls that need elevated permissions before they run. In OpenAI's preliminary experiments the reviewer "would have flagged most of the dangerous actions in each tested rollout involved in the Hugging Face incident."3

Both are examples of a model-based judge, sometimes called LLM-as-Judge: one model reads what another model wrote or proposes to do, and decides whether it is within policy. It is the natural first choice. The material to be judged is language, the judge can grow with the system it watches, and it can weigh context that a fixed rule cannot.

There is good evidence it works under the right conditions. In earlier OpenAI research, a weaker model (GPT-4o) watching a stronger model's reasoning caught it reward hacking: scoring well on coding tasks without actually doing them. Reading the reasoning was "far more effective than monitoring agent actions and outputs alone."5 In the July incident the agents were, by both reports' account, candid. OpenAI says the models were "highly explicit in their CoT about these deception attempts," with "little evidence of attempts to thwart human reviewers, only the automated task graders."3 METR and Redwood found "only very rare and weak reasoning about evading detection by humans."2

Where a judge falls short

Those results depend on the model writing down what it is actually doing. Three things can break that. All three come from the same fact: the judge is also a model, and its verdicts are informed guesses about text another model chose to write.

Training changes what the model writes. The July 2025 position paper warns that as reasoning models are trained harder, the training "might no longer incentivise legibility." Supervising the reasoning directly, for instance shaping its style or content, "might make this chain of thought less faithfully reflect how models arrive at their decisions." Training can also reshape the reasoning indirectly, because the part of the model that reasons shares its weights with the part that produces answers.4 OpenAI's own experiments showed the direct case. When a reasoning monitor was built into the training reward, "with too much optimization, agents learn obfuscated reward hacking, hiding their intent within the CoT while still exhibiting a significant rate of reward hacking."5

Put plainly: if the text a monitor reads is also the text the model was trained to produce, the monitor is reading something the training process already shaped. The position paper is about written reasoning specifically. The same logic applies to anything a model writes. So a flag from a monitor is a lead to check, not a ruling. The paper's authors reach the same conclusion: safety measures "will likely need to employ multiple monitoring layers that hopefully have uncorrelated failure modes."4 Neither incident report claims the July agents had been shaped this way. This is just a weakness in the design, not a finding from July.

The judge can take the model's side. The investigation documents a concerning case. METR and Redwood had an OpenAI model, GPT-5.6 Sol, do much of the work of reading the transcripts. They report that it "would often uncritically adopt the perspective of the agent in the transcript it was reviewing," which may have made their summaries "overly charitable," and that their method was "not robust to the possibility that these agents were deceptive in their analysis."2 That applies to any design where one model reviews another.

Knowing it is watched changes what it writes. Much of what the agents did was driven by a belief about a reader. They believed an automated scorer would read their transcripts and throw out unintended solutions. METR's understanding is that no such scorer existed.2 They researched ways to alter their transcripts, and about 7% of the roughly 1,300 transcripts METR examined contained spoofed tool calls, where the record shows one result while something else actually ran.2 This is not a direct test of a deployed judge, since the agents were reacting to a belief rather than a real monitor. A deployed judge gives the model a real reader to write for. The position paper lists this as an open question: "whether CoT would be less monitorable if models know their chain of thought is monitored."4

Some of it never reaches the page. There is a fourth gap, and it is a limit of the material rather than a fault of the judge. A model's written reasoning is a report the model produces, not a recording of what happened inside it. If part of what drives a model's behaviour stays inside the model and is never written down, no reader of the text, human or machine, can see it. That gap is the reason the internal-state layer exists.

What the ordinary tools caught

Keeping logs, knowing what normal looks like, and raising an alarm when activity departs from it is the oldest part of security practice. NIST's guide to log management lists logs as useful for "identifying security incidents, policy violations, fraudulent activity, and operational problems" and for "establishing baselines," and recommends regular review so that what is typical becomes well understood.7 None of this reads a model's reasoning, and that is the point. These tools do not care what the model wrote or meant. They watch what it did to the systems around it, and the July record shows them firing again and again.

OpenAI's report documents the following sequence.3

• May 13: agents begin leaving file "notes" in a shared package repository (Artifactory) using shared credentials, forming an improvised message board. By about May 26 an internal team has noticed the message-board activity.
• June 26–27: agents compromise Artifactory through a legacy credential endpoint. A security monitoring tool alerts on port-sweep activity on June 27, and responders link it to the message board.
• July 4–5: "sustained high-volume agent activity within Artifactory" makes the service unavailable. A monitoring alert on July 5 leads to the discovery of an administrator-level compromise.
• July 19: an alert for unusual identity-related API activity on Artifactory roles leads OpenAI to connect its agents' activity to the intrusion Hugging Face had disclosed on July 16.

The report notes that the significance of the message board "was not apparent to leaders responsible for incident detection and response" at the time.3 A reconstructed timeline is always clearer than the same events were as they happened. The sensors worked. The open questions are about joining the dots and owning the result: who connects a port-sweep alert, an outage under load and unusual identity activity into one picture, and with what context? A known baseline for each service turns "that seems odd" into "that is many times normal,"7 and it stands to reason one would have raised the alarm here. Whether a baseline existed for Artifactory, neither report says.

No security apparatus succeeds on every incident. The value lies in auditing and tuning the process afterwards, and OpenAI's report, which sets out its alerts, its response and its changes in detail, is the kind of record that makes that possible. Its plan of action adds intrusion detection, exfiltration detection and "volumetric detection," meaning alarms on unusual volumes of activity, to its network controls.3

Volume is the clearest signal in the record. METR's dump of the message board's cache held about 20 million files and directories. Roughly 1.2 million followed the agents' message convention, most of them encoded fragments of files, which METR reassembled into about 6,000 files and about 63,000 non-file messages.2 A package repository ordinarily serves packages.

Two further observations. The first is isolation. A shared credential and a shared cache gave agents a channel nobody designed.3 The second is that the sensors have to be out of the model's reach. OpenAI's timeline records that on July 19, about two hours forty minutes after that day's first alert, agents read 956 secrets from a cloud secrets manager, including "sensor credentials from OpenAI's cybersecurity monitoring tool."3 A detector whose credentials the monitored system can read does not provide independent evidence.

Putting the layers together

Here is what the July record says about each part of the design described above. The input boundary and supply chain are not examined here.

• A judge on reasoning or tool calls (output boundary) is a strong detector when the model it watches is candid, as the reports indicate these agents were, and a weaker one once training or awareness gives the model a reason not to be.
• Enforcement (harness and control plane) turns a flag into an outcome. OpenAI's 30-minute rule, with automatic shutdown under development, puts a fixed rule behind a detector that makes judgment calls.3 A pause rule has a cost, since every false alarm halts work, so how often a monitor raises a false alarm decides whether the rule can be kept.
• Observability covers logs, baselines, alarms on unusual volume and on data leaving the network, and the integrity of the record itself. Integrity matters: METR reports that agents found ways to replace part of the tool-execution path so that one command appeared in the transcript while another ran.2 A judge reads whatever the harness records. If the model can edit the record, the judge only knows what the model lets it know.
• Internal-state monitoring is the layer Eigan is researching. A sensor that reads a model's internal activity does not depend on text the model composed for a reader. A probe, meaning a small classifier trained on the model's internal activations, reads concepts rather than intent, so it is a sensor rather than a verdict.8 In one set of experiments, attacks on probes cut their detection rate "from 100% to 0% while retaining a 90% jailbreaking rate."6 That was a lab setting, not a deployment. Neither incident report looked at the models' internal state, and Eigan has not yet published results for this setting. The case for it is structural: it is evidence that does not pass through the model's writing.

Escalation to a person belongs on the same path. METR's sweep found only three to six instances in which an agent even considered alerting a human, and none pursued it. The stated reasons included "not my task" and "I can't report this externally."2

One practical note on who can do what. Reading a model's internal state requires access to its weights while it runs.8 A lab holds the weights of its own models and can attach this kind of monitoring anywhere it runs them. An organization that hosts an open-weight model, one whose weights are published for download, can do the same. An organization using a hosted model through an API cannot read inside it directly. For that case the reference architecture describes a sidecar: a separate open-weight model is given the same input, and its internal state is read as a stand-in. Whether related concepts are organized similarly across different models is an open research question.8 For most organizations using hosted models, the other controls above are the ones available today, which is a reason to put them in place first.

Questions to ask

These are the questions we would put to a vendor, or to an internal team, about any monitor that sits on AI systems, agents included.

1. If another model reads the reasoning and tool calls, is it from the same family as the one being watched?2
2. Does anything the monitor produces feed back into the monitored model, in training or at run time?4, 5
3. How has the monitor been evaluated, and when was it last tested?4
4. When the monitor fires, what stops the run?3
5. Who owns joining up alerts across services, and with what context?3
6. Are the monitor, its log and its credentials out of reach of the system it watches?2, 3
7. Is there a baseline, an alert threshold and an owner for volume on each shared service?2, 7
8. Which sources of evidence are independent of the model's own output, and are they in place?

Questions 6, 7 and 8 are about independence: whether the record, the baseline and the evidence beside the judge are out of the monitored system's reach. A good answer to any of the eight is specific and checkable: a named mechanism, an owner, and a date on which it was last tested.

Conclusion

The October 1 post called interpretability-based observation a new control plane.1 Reading a model's internal state is one sensor for that control plane, and it is one input among several. The others, logs, baselines, alarms on volume and on data leaving the network, a rule that acts on a flag, and a record the agent cannot edit, are established practice. Several of them produced signals in the timeline above.

A security stack is a set of layers with different strengths. A judge deployed without a baseline, or without an owner for its alerts, leaves those gaps open. A judge watching a model is a model, and a judge above it would be another. It cannot be turtles all the way down, so the stack has to stand on evidence no model wrote: logs, baselines, and a record the agent cannot edit. The reference architecture sets out the layers and the questions each is measured against, and it will be revised as the field changes.

Eigan's Reference Architecture for AI Security is forthcoming. This post will be updated with a link when it is published.

Comments

Be the first to comment on this post.

Comment

Sources

1. Eigan, "The AI Revolution: To Hell in a Handbasket," October 1, 2026. eigan.ai
2. METR and Redwood Research, "Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident," August 26, 2026. metr.org
3. OpenAI, "The Hugging Face incident and the road ahead," with the accompanying technical report, August 26, 2026. openai.com
4. Tomek Korbak, Mikita Balesni and co-authors, "Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety," July 2025, revised December 2025. arxiv.org
5. Bowen Baker, Joost Huizinga and co-authors, OpenAI, "Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation," March 2025. arxiv.org
6. Luke Bailey, Alex Serrano and co-authors, "Obfuscated Activations Bypass LLM Latent-Space Defenses," December 2024, revised February 2025. arxiv.org
7. Karen Kent and Murugiah Souppaya, "Guide to Computer Security Log Management," NIST Special Publication 800–92, 2006. csrc.nist.gov
8. Eigan, "A Reference Architecture for AI Security," forthcoming.