September 4, 2026

I built a security AI that cannot hallucinate, for a competition that never wrote back.

The limits of my language mean the limits of my world.

By CypherNova1337

9 min read

I built a security AI that cannot hallucinate, for a competition that never wrote back. The competition was the smaller half of what I got out of it.

The limits of my language mean the limits of my world.

— Ludwig Wittgenstein, Tractatus 5.6

A competition with an unusually good brief

Earlier this year I came across the Wittgenstein AI Tournament, run by Hackers-Arise. Fifteen thousand dollars for a grand prize, and a set of requirements that read like someone had actually thought about them rather than pattern-matched on the word AI.

The entry had to be built on an open-source model. It had to run locally, on hardware a person owns, not on somebody's API. It had to be light on energy and water. It had to keep humans in the loop rather than acting on its own. It had to run on ARM. And — the line that made me want to enter — it had to accept that neural networks are but one way of doing this, not the only one.

That last principle is the whole reason the project turned out to be worth building. A brief that permits you to not use a language model where a language model is the wrong tool is a brief that lets you build something that actually works.

The problem with security AI

Most security AI is a language model holding a log file, asked politely not to make things up.

It makes things up anyway, because nothing in the architecture prevents it. And in this domain a confident fabrication is worse than silence. An invented IP address in an incident report reads exactly like a real one. An analyst who catches one invention correctly stops trusting the other ninety-nine findings.

You cannot fix that with a better prompt. Please don't hallucinate is not an architecture. So I took Wittgenstein at his word and made it structural.

The Lexicon

At the center of VoidAI is a closed, typed vocabulary of the propositions the system is permitted to assert. Eighteen predicates. Each one declares which kinds of noun it will accept as a subject and which as an object.

A host can beacons_to an address or a domain. It cannot beacons_to a port, because the grammar rejects it at construction. There is no severity flag for that, no confidence score, no warning in a log somewhere. The sentence simply has no representation. It is unsayable.

Every proposition that is sayable has to carry a chain of custody down to a byte range in a specific file:

Artifact  →  Evidence  →  Finding  →  Incident  →  Claim
(a line in  (a measured  (a grounded (correlated  (language-layer
 a file)     observation) assertion)  cluster)     commentary)

Nothing may skip a link. A finding with no evidence is rejected when you try to build it. A claim citing a finding that was never in the brief gets struck by a verifier before an analyst ever sees it. Both are enforced by the type system rather than by convention, because a provenance rule that lives in a style guide is a provenance rule that is already broken.

The language model, in all of this, never sees a raw log. Detection is done by deterministic statistics — four analyzers, no neural network in any of them. The model receives a few hundred tokens of already-grounded findings and does the three things it is genuinely good at: narrating a correlation, ranking competing hypotheses, and suggesting what to look at next.

Which means detection quality is identical with the model switched off. You lose the prose. You do not lose a single finding.

What real data did to my assumptions

I validated against CTU-13 — thirteen captures of real botnet traffic on a university network, with per-flow ground truth, published by Stratosphere IPS.

This is the part I would tell anyone starting a detection project. Synthetic data does not validate your detector. It validates your assumptions, because you built both of them out of the same beliefs. Real traffic found four bugs that my generator was structurally incapable of finding.

A signal that was measuring my own generator

One of my beaconing signals scored a near-perfect 1.0 on every synthetic implant I threw at it. I was delighted, briefly. It scored perfectly because my generator applied symmetric jitter by construction, and the signal measured symmetry. Real command-and-control is heavily right-skewed — its tail is missed check-ins at two, three and five times the base period. The measure was penalising its own best evidence. I replaced it with a hard-floor test: a timer cannot fire early, a person has no such constraint.

One check-in is not one record

NetFlow splits connections, so a 33-second beacon arrived as 384 records with a median interval of 0.15 seconds. Every statistic I had was faithfully measuring record framing rather than the beacon. It now detects that as bimodality in log-space using Otsu's method and coalesces the bursts before measuring anything.

The textbook rule that deleted the real C2

The server has the lower port number is a rule you will find in any traffic-analysis tutorial. In one capture the bot sources from ports 1027–4985 to a controller on 5678. Under that rule the genuine C2 channel is classified as a reply and discarded — taking the capture's only true positive with it. My findings went from 618 to 309 and the detection rate went from yes to no, silently, because a textbook rule is exactly the kind of thing you don't re-examine.

A negation matched by the word it negates

Suricata's category Not Suspicious Traffic scored 0.55 on my suspicion scale, because the substring suspicious matched first. Suricata's noisiest category was reading as moderately suspicious and flooding the queue by itself.

Detected and invisible are the same thing

The single most useful lesson had nothing to do with detection at all.

My beaconing analyzer found the C2 channel in a real capture and buried it at rank 358 of 395. It worked perfectly. It was also completely useless, because no analyst working a queue is reaching item 358.

The fix was not a better periodicity measure. I looked at what was outranking it, and those findings were genuinely beacon-like: monitoring agents, backup jobs, update checks. They deserved their scores. What separates a compromised host is not that it does one suspicious thing very suspiciously — it is that it does several unrelated suspicious things at once.

So findings are grouped by subject and ranked by noisy-OR over the strongest finding per predicate, times a corroboration bonus. Per-predicate deliberately: twenty beaconing findings on one host are twenty views of one behaviour, not twenty independent reasons to believe it.

CTU-13 — real botnet captures with ground truth
                        Scenario 3        Scenario 6
                        Rbot, 66.8 h      Menti, 2.15 h
  ------------------------------------------------------
  Flows analysed        12,689,947        1,916,655
  Findings → incidents  1328 → 214        397 → 133
  Infected host, rank   2 of 214          1 of 133
  Peak memory           2.6 GB            0.6 GB

Scenario 6 went from rank 358 to rank 1. Not because detection improved — because ranking finally answered the right question. A finding answers how beacon-like is this traffic? An incident answers how much should an analyst care about this host? Conflating those two is what buried the true positive.

Believed, measured, and the difference

For most of this project's life, the honest answer to does it run on a Raspberry Pi? was we believe so. My documentation said the 66-hour capture needed more than a 4 GB board could give.

I hadn't measured that. I had inferred it from a peak-memory figure, which is a different thing wearing a similar hat.

So I wrote a harness that runs the pipeline inside a cgroup with a hard memory limit and swap pinned to the same value — the same mechanism a board with no swap enforces. Then I bisected.

12.7M flows against a hard memory ceiling
  Ceiling                             Outcome
  --------------------------------------------------------------
  2,000 – 2,400 MB                    OOM-killed, every run
  2,500 MB                            Flaky — killed on one run,
                                      completed on the next
  2,600 MB                            Completes
  3,000 MB                            Completes, peak 2,545 MB
  3,696 MB (4 GB board, less the OS)  Completes

The documentation was wrong. It fits a 4 GB Pi with about 1.1 GB to spare. My estimate had been pessimistic by an entire board class. I had written it down, repeated it in four separate documents, and never once tested it.

The flaky row is the one I find most instructive. At exactly 2,500 MB it completes about half the time. That is what running at the wall looks like, and no single test run would have told me it was there.

The result I actually care about came next. The same 12.7 million flows, held to 3 GB and a single core, finish in 166 seconds and still return the infected host at rank 2 of 214 — identical to the unconstrained run. Constraining the hardware changes how long the answer takes. It does not change the answer.

The whole pipeline including a 1.5-billion-parameter model at 4-bit peaks at 2,072 MB. The demo runs in 512 MB.

I still haven't run it on actual ARM silicon, and I say so in the README. A cgroup reproduces a memory ceiling exactly; it cannot reproduce an instruction set. Anything that would only break on aarch64 passes on my x86 box. That gap is real and it stays labelled until a board arrives on my desk.

The part where nobody wrote back

I signed up for the tournament twice, through the registration form, and never heard anything back either time. No confirmation, no rejection, no follow-up.

I want to be careful about how I say this, because I don't think it's a scandal and I'm not interested in making it one. Small organisations run events with small teams. Forms break. Mail lands in spam folders in both directions. I have no idea what happened on the other end and I'm not going to invent a story about it.

What it means practically is that I don't have a submission in a $15,000 competition. Which was, honestly, always the least likely outcome and the least interesting reason to have done the work.

None of this changes what I think of Hackers-Arise. I'd recommend OTW's courses and the Hackers-Arise material to anyone who wants to learn actual hacking rather than collect certifications — it is one of the few places teaching the real thing, taught by someone who obviously does it. The brief for this tournament is itself evidence of that: eleven principles about local, small, energy-honest, human-in-the-loop AI, written by someone who had clearly thought hard about where this is all going.

I built something good because of that brief. That part happened regardless of whether the form went through.

What I'm keeping

I'm continuing to maintain VoidAI, and the reason is straightforward: writing it taught me more about building AI-type programs than any amount of reading had.

The model is a bad detector and a good explainer

Nearly every design decision that worked came from that split. Statistics find the thing. The model describes what was found. The moment I stopped asking the language model to do detection, it got dramatically better at the job it was actually suited for — and the whole system got small enough to run on a board that costs less than a night out.

Constrain what a system can say, not what it should say

Asking a model to be truthful is a request. Giving it a vocabulary in which false things are structurally unrepresentable is an architecture. The verifier that strikes claims citing unresolvable evidence has caught real fabrications. The count of struck claims is printed on every run receipt, so the system reports how much it is confabulating rather than hiding it.

The measurement is the work

Four of the bugs above were invisible until I ran against something I hadn't made. My memory claim was wrong until I imposed a real ceiling. And in one case the test itself was the problem: I had a test named test_all_zero_values_are_not_treated_as_regular that asserted, in code, that all-zero values were treated as regular. I'd written the assertion to match the implementation instead of the intent, and it sat there passing, guarding nothing, for weeks.

Typed structure pays you back somewhere unexpected

Because findings are typed propositions rather than prose, I can turn one into a Sigma rule, a KQL query, an SPL search or a zeek-cut pipeline by templating — no model in the path. You cannot mechanically convert "host 10.0.1.14 looks like it might be beaconing" into a detection rule. You can convert a predicate with declared entity types and a measured payload into one, deterministically. That fell out of a decision I'd made for entirely different reasons months earlier.

$ voidai demo
 #  Severity  Prio  Subject        Behaviours               Findings
 1  CRITICAL  2.50  ip:10.0.1.14   beacons_to, scans,              5
                                   triggered_signature,
                                   tunnels_dns_over
 2  CRITICAL  0.89  ip:10.0.1.23   tunnels_dns_over                1
 3  HIGH      0.87  ip:10.0.1.17   beacons_to                      1
74,157 records · 0.35 s · no model · no GPU · no network
4 independent behaviours corroborate — nothing labels this host

Six runtime dependencies. No CUDA, no compiler toolchain, no wheel without an ARM build. It runs with the network interface down, and the test suite severs sockets and asserts that it still completes — because a claim about not phoning home is worth exactly as much as the test behind it.

The thing I'd say to someone starting

Build the thing that would embarrass you if it were wrong, then go looking for the ways it's wrong. The best days on this project were the ones where a real capture told me something I'd built was nonsense. Four bugs, one wrong memory claim, one test asserting the opposite of its own name — every one of those found by pointing the code at reality instead of at my own assumptions.

The competition would have been nice. The habit is better.

VoidAI is Apache-2.0 and it's on GitHub. It runs on a Raspberry Pi, cites every claim it makes down to a line in a file, and prints what it cost you in joules. If any of that sounds useful, take it.

github.com/CypherNova1337/VoidAI

Corpora: CTU-13, Stratosphere IPS (CC-BY). Model: Qwen2.5–1.5B-Instruct (Apache-2.0).

Hackers-Arise and OTW's courses: hackers-arise.com