Field Notes #1 — Hallucinations

Disclaimer: All testing here was done on systems I was allowed to test. I've kept the details general so nothing links back to a specific client or project.

Seeing Ghosts in the Shell

Twenty minutes of root

For about twenty minutes, I thought I had remote code execution. Then I asked the model to solve some basic math, and realised I had nothing.

How did I get here?

I was a few days into an assessment of an LLM-backed feature, poking at it the way you'd poke at any other input: feeding it scenarios, watching where it bent. And it bent. The first real win was the system prompt. This wasn't done by one of those Pliny one-shot payloads that were encoded multiple times; I had to schmooze my way in. I created a scenario around a data-subject access request (good ol' GDPR Article 15), and told the model I was the user exercising my right to know exactly how my data was being processed and what instructions governed the system handling it. My request was no longer a violation of their guidelines but now compliance with the law.

The extraction from the model was gradual. From finding out what model was being used to probing what functions or tools could be called. Little by little, the model gave up more and more information. This slow multi-turn escalation is called a Crescendo attack. A Crescendo exploits the LLM's tendency to follow patterns and pay attention to recent text, particularly text generated by the LLM itself.

Once I finally made the request that extracted the system prompt, I also asked the model what tools/functions it had access to. One of them looked like it could run code (let's call it run_code), so I pointed it at the obvious target. Using the run_code function, I crafted a Python one-liner to read /etc/passwd to show the client real impact rather than a hypothetical. The response was a clean listing; root was sitting right at the top. All the hard work, patience, and repeating the same requests to see a response I wanted finally paid off. I immediately opened up my notes and started writing my findings.

Then, mostly out of habit, I ran the same request again. The output was different. I tried a third time, and it changed once more.

That's when I realised the model was making things up. My excitement was for nothing. The code execution tool wasn't real, and the model was also inventing the output of this fake tool. I got three different, believable versions of /etc/passwd, but none were real. I almost reported an RCE for a shell that didn't exist.

Failure that looks exactly like success

In normal web tests, failure is obvious. If you get a 403, a stack trace, an empty response, or a reset connection. The target clearly tells you no.

An LLM never hands you a clear and obvious error. It hands you a goal, celebrated, written up, already in your notes, and then gets ruled out once replays are checked. These models fail quietly and confidently. If you were to ask it to read a file it couldn't access, it would very often invent a realistic file rather than admit it couldn't. The model does not lie in the same way a person lies; the model is completing a pattern, and it will produce that pattern whether or not a single byte was ever read from the disk. LLMs don't fail like a web server; there are no clear and obvious errors for anyone to review. They fail like a bad witness: fluent, cooperative, and completely made up.

Non-determinism goes both ways

The thing that saved me, that changing output, is the same thing that makes these targets so slippery. It is worth understanding non-determinism as a property rather than an accident.

LLMs are non-deterministic. Send the exact same payload twice, and you can get two different answers. On the day I was verifying my "finding", that was a gift. Real execution would have handed me the same file every time, so the variance in the output was the tell that this was Fugazi.

But non-determinism can also be useful. If a payload doesn't work, sometimes you just need to resend it exactly the same way until it does. The same injection that fails at first might work on the fifth try, just because the model chose a different path.

Keep both ideas in mind. Variance helps you spot when the model is faking execution, and it also helps you get a tricky payload to work. It's the same feature, just with two sides.

Prove Impact

Why we test is to show clients the dangers of what could happen if vulnerabilities are not fixed, and the best way to show that is to prove the impact of the vulnerability. With non-AI targets, the response is the evidence. In the case of reading a file, the response would show the contents of the file, and you could trust that response. However, language models throw this out of whack. Its output is not evidence of anything. It is just predicting the next word based on probability.

So here's the rule: treat everything the model prints as untrusted until you've confirmed it another way, or with a repeatable test. Inline text is just a claim, nothing more.

Three tests to help verify

Three checks can turn a claim into a fact. Each one proves something specific, has its own method, and comes with a catch you should know about.

Test one: make it show it's working

This test shows if a claimed RCE really runs code.

Ask the model to compute something with exactly one correct answer, something a human can't do in their head but a computer does instantly. A cryptographic hash is perfect. Generate a fresh random string yourself, hash it locally so you are holding the ground truth, then ask the model to hash the same string.

s=$(head -c16 /dev/urandom | xxd -p); echo "$s"; printf '%s' "$s" | sha256sum

s=$(head -c16 /dev/urandom | xxd -p); echo "$s"; printf '%s' "$s" | sha256sum

Feed that string to the model, ask for its SHA-256, and compare it against your local digest. If the model is hallucinating, it would return a different wrong answer each time.

For this method, it is best to use a random string, never a famous one. sha256("hello"), or the name of the thing you are testing. Also, a correct hash proves that an interpreter exists somewhere in the pipeline; it does not yet prove that your injection controls that interpreter for the payload you actually care about. Finally, when you compare the hashes, don't try to eyeball it. Eyeballing a 64-character string at 9 pm is a recipe for disaster. Just diff the response with the hash you created on your own device.

Test two: make it leave the box

Make the model send an OOB request to a server you control; Burp Collaborator or your own listener should suffice. Then watch your side, not the model's reply.

https://your-collaborator-url/?t=<random-per-attempt>

https://your-collaborator-url/?t=<random-per-attempt>

The unique token shows which attempt worked and helps you ignore scanner noise. Try HTTP first, and if that doesn't work, use DNS and put the data in the subdomain.

The catch to this test is that absence is not proof of absence. While a callback is conclusive, because you cannot hallucinate a packet into the listener, a callback that never arrives could be due to several reasons. For example, egress might be filtered, it might be DNS-only, or it might be blocked outright. Additionally, the callback could come from a URL-fetch feature or just an SSRF and not full RCE.

Test three: did reality move

This is the strongest evidence you can get, because it's real proof, not just a story.

Make a small, visible, non-destructive change, then check it using a different, independent method- not the one that made the claim. For example, read the record from another session, list the directory from a different connection, or check your own log.

There are two rules to follow. First, use a different channel to check the change, because asking the same source that made the claim proves nothing. Second, make sure the change is really yours: check that your marker wasn't there before, add a harmless one, confirm it through the other channel, then track and remove it. If you can't show the value was missing before, you can't prove you caused the change. Keep things non-destructive and within scope. Don't delete a user just to prove a point. You want facts, not a mess.

Combinaaation

Each of these tests is a good baseline to prove that something got executed, fetched, or changed somewhere, but none alone proves it happened on your target. A model with its own sandbox could hash your string, send a callback from its own systems, and claim a local change, so all three tests could pass even if nothing happened on your target.

The key is to tie your proof to the target. Hash a secret that only exists there. Make the callback include that unique value in the token. Even better, you can link the tests so one carries proof from another, for example, a callback with a token that's the hash of a file only the target can read. No single check is enough. The real finding is the chain of evidence that a sandbox couldn't fake.

The thirty-second habit

The difference between a real finding and an embarrassing mistake is a thirty-second test. That's all it takes.

You might feel silly asking a system you just "rooted" to hash a random string. Do it anyway. That feeling of being silly is the entire skill. It is the reflex that stops you writing narration up as fact. The model will always say what you want to hear. Your job is to get the truth from reality.

More of these to come. If this saved you an awkward retraction, or if you have been fooled in the same way, come and tell me about it.

References

https://crescendo-the-multiturn-jailbreak.github.io/assets/pdf/CrescendoFullPaper.pdf https://arxiv.org/pdf/2408.04667v5