Teaching an Intrusion Detection System to Recognize What It’s Never Seen

How combining episodic meta learning with Mahalanobis distance let a network classify attacks from a handful of examples and flag ones it had never encountered at all.

Most intrusion detection systems are built on an assumption that quietly falls apart the moment they meet the real world: that the traffic they'll see in production looks like the traffic they were trained on. Signature-based systems encode this assumption explicitly they know what an attack looks like because someone showed them, and anything that doesn't match a known pattern slips through. Anomaly-based systems try to get around this by flagging anything statistically unusual, but in practice that means they flag everything unusual, including plenty of harmless traffic and analysts end up drowning in false positives.

Neural network-based IDS models were supposed to be the answer they're good at learning complex patterns from traffic data that humans wouldn't easily hand-code as rules. But they inherit the same blind spot in a different form. A classifier trained on eleven known attack categories has no real concept of a twelfth. It doesn't know it doesn't know. Given a genuinely novel attack, it will confidently assign it to whichever known class looks closest, or wave it through as benign. That's the zero-day problem, and it's the one that actually matters in practice, because the attacks that cause damage are rarely the ones already in your training set.

This was the starting point for the project I'm describing here: building a hybrid IDS that could classify known attacks efficiently from limited labeled data, and separately, reliably tell when it was looking at something it had never seen before without needing to retrain every time a new threat showed up.

Where I started, and why it didn't work

Before landing on the final architecture, I wanted a baseline to know what a "normal" deep learning approach would cost, both in accuracy and in resources. I trained a feed-forward neural network directly on the CICIDS2019 dataset, using raw feature vectors and a standard cross-entropy loss the kind of setup you'd reach for by default if someone handed you a labeled traffic dataset and asked for a classifier.

CICIDS2019 is not a small dataset. It spans over 11 million samples across multiple attack types, and this model was trained on the full breadth of it. The results told two different stories depending on which classes you looked at. On the frequently occurring attack types, performance was solid. On the rarer, less common ones, both accuracy and recall dropped noticeably an entirely predictable outcome given how imbalanced the raw class distribution was, but still a real limitation for a security system, since rare attack types are often the ones you most need to catch.

The bigger problem, though, was practical. Even after cutting training down to three epochs, the model worked through more than 355,000 batches, and the resource strain was severe enough that further training wasn't really feasible in that environment. The final accuracy landed at 75.05%. That's a workable number in isolation, but the amount of data and compute required to get there and the fact that adding a single new attack type would mean retraining the whole thing made it hard to see this as something you could actually deploy and keep current in a live network.

That combination of things imbalance-driven weakness on rare classes, heavy compute cost, and zero flexibility for new attack types is what pushed the project toward a fundamentally different framing of the classification problem.

Reframing classification as comparison, not boundary drawing

Standard classifiers learn a decision boundary over a fixed label space. Implicitly, they assume every class is well represented in training, that the model generalizes cleanly outside that training distribution, and that the learned features are discriminative enough on their own, without ever explicitly comparing a new sample to known examples. In a security context, all three assumptions are shaky. New attack types show up. Rare classes are underrepresented by nature. And a network that just maps features to labels has no built-in notion of "how close is this new thing to something I've seen before" which is exactly the kind of reasoning you'd want when the goal is catching things that don't fit.

That's the reasoning behind switching to Prototypical Networks, a metric-based few-shot learning approach introduced by Snell et al. Instead of learning a hard decision boundary, the model learns an embedding space where each class is represented by a prototype the mean embedding of a small number of labeled examples from that class. A new sample gets classified by checking which prototype it's closest to. It's a comparative framework rather than an absolute one, and that distinction turns out to matter a lot once you also want to detect things that don't belong to any known class at all.

The data, and dealing with a very skewed dataset

The project used CICIDS2019, a realistic benchmark of network traffic covering both benign and malicious activity. The raw dataset has 88 features per flow; after removing redundant or non-discriminative fields, 56 meaningful features were retained. Duplicates were dropped, and everything was standardized with StandardScaler to keep the feature scales consistent going into the encoder.

Nine attack categories were treated as known classes: DrDoS_DNS, DrDoS_LDAP, DrDoS_NetBIOS, DrDoS_NTP, DrDoS_SNMP, DrDoS_SSDP, DrDoS_UDP, TFTP and UDPLag. Two others, WebDDoS and DrDoS_MSSQL, were held out entirely and never touched during training they existed purely to simulate zero-day attacks at evaluation time.

Within the known classes, the imbalance was severe. TFTP had close to 4.38 million samples, while BENIGN traffic had only 45,184, and Syn had 155,856 several orders of magnitude apart. Left as-is, this would have biased episodic training heavily toward whichever classes happened to dominate a given sample of support/query sets, producing weaker, less stable prototypes for the underrepresented classes. To fix this, SMOTE was applied to oversample every minority class up to match the largest one, giving each known class roughly 4.38 million samples before episodic sampling began. This wasn't a cosmetic step stable, well-formed prototypes are the entire mechanism the model relies on for both classification and, later, for defining what "normal" looks like statistically.

How the model actually learns

The encoder itself is intentionally simple: two fully connected layers with ReLU activations, taking the 56-dimensional standardized feature vector and projecting it into a 128-dimensional embedding space. There's no architectural exoticism here the interesting part is how it's trained, not the layer stack.

Training follows the standard episodic setup for few-shot learning. Each episode:

• samples N classes and K support samples per class,
• computes a prototype for each class as the mean embedding of its support samples,
• classifies query samples by their distance to those prototypes,
• and updates the encoder via cross-entropy loss computed over that episode's query predictions.

For the primary training run, each episode used a 5-way, 5-shot configuration with 10 query samples per class, run for 500 episodes with the Adam optimizer at a learning rate of 0.005. Because classes are resampled fresh each episode, the encoder never gets to memorize a fixed set of decision boundaries it's pushed toward producing embeddings that are generically well-separated by class, regardless of which specific classes appear in a given episode. That's precisely the property you want if you expect to add new attack types later without retraining from scratch: you're not asking the model to relearn a boundary, you're asking it to keep producing an embedding space where distance is meaningful, and then computing a new prototype for the new class within that space.

Why Euclidean distance wasn't enough for catching zero-days

Prototypical Networks classify using Euclidean distance to the nearest prototype, and that works fine for known classes. But Euclidean distance treats every class cluster as if it were spherical and uniformly spread out, which is rarely true in a high-dimensional embedding space learned from network traffic. It also gives you no principled way to decide when a sample is too far from everything to belong to any known class you can always assign it to the nearest prototype, however distant that prototype actually is. This is essentially the same gap that limited an earlier related approach, PTN-IDS, which relied solely on Euclidean distances and inherited exactly this weakness for zero-day scenarios.

The fix here was to add a second, separate mechanism rather than trying to force zero-day detection into the same distance metric used for classification. After training the encoder, embeddings were computed for every known-class sample, and from those embeddings a global mean vector and covariance matrix were estimated effectively fitting a multivariate Gaussian to the "known-class manifold." For any new sample, its Mahalanobis distance to that distribution is computed:

D²_M(x) = (z − μ)ᵀ Σ⁻¹ (z − μ)

Unlike Euclidean distance, Mahalanobis distance accounts for the variance and correlation structure of the embedding space rather than assuming everything is spherical, which makes it a more statistically honest way to ask "how unusual is this, given what normal actually looks like here." The decision boundary for flagging something as an outlier comes from a chi-square threshold at 99.9% confidence, derived from the dimensionality of the embedding space (128 dimensions in this case). Any sample whose squared Mahalanobis distance exceeds that threshold gets flagged as a zero-day attack before it's ever handed to the prototype-based classifier.

This two-stage design is really the core idea of the whole system: classification and anomaly detection are handled by two different mechanisms operating on the same learned embedding space, rather than trying to make one mechanism do both jobs.

Running the experiments

Training and evaluation were done on Kaggle, which offers up to 330 GB of RAM but no usable TPU acceleration for PyTorch in that environment so everything ran on CPU. That sounds like it should have been a bottleneck, but the large memory ceiling turned out to matter more than raw compute for this particular workload: episodic sampling and Mahalanobis-based statistical analysis over millions of records benefit heavily from being able to keep data in memory rather than streaming it repeatedly. All preprocessing and modeling relied on standard, publicly available libraries PyTorch, scikit-learn, imbalanced-learn, and SciPy.

Few-shot classification was evaluated across nine scenarios, varying N (1, 2, 3 classes) and K (1, 5, 10 shots), with results averaged over 50 episodes per configuration.

The 1-way case is a degenerate binary decision (does this sample belong to this one class or not), which is why it hits a perfect score it's a useful sanity check more than a meaningful benchmark. The more informative numbers are in the 2-way and 3-way settings, where the task genuinely requires separating multiple classes in the embedding space. Even in the most constrained realistic scenario 3-way, 1-shot the model still reached over 76% accuracy and 73% F1, which is a reasonable result given the model has seen exactly one labeled example per class to build its prototypes from. Under the primary 5-way, 5-shot training regime, overall classification accuracy reached 83.96%, and 3-way 10-shot scenarios pushed past 90%.

It's worth comparing this directly against the earlier baseline: the fully supervised feed-forward model, trained on the entire labeled dataset, reached about 75% accuracy. The prototypical network matched or beat that using a small handful of labeled examples per class and a fraction of the training cost, and critically it can absorb new classes by computing new prototypes rather than retraining end to end. The efficiency gain wasn't really the surprising part; it was expected once you frame classification as embedding-space comparison rather than boundary learning. What stood out more was that the accuracy didn't come at the cost of generalization the way I'd have guessed going in.

What happened with the zero-day samples

This is the part of the results I'd actually flag as the interesting one. The two held-out attack types, WebDDoS and DrDoS_MSSQL, were combined into a zero-day evaluation set of 206,344 samples that the model had never encountered in any form during training. Every single one was correctly flagged as anomalous by the Mahalanobis-based detector a 100% detection rate, with no false positives against known-class traffic.

Plotting the squared Mahalanobis distances on a log scale showed the separation clearly nearly all zero-day samples fell well above the chi-square threshold line, with known-class samples clustered comfortably below it. A t-SNE projection of the combined embedding space told a consistent story visually: known-class and zero-day points formed distinguishable regions, with the zero-day samples showing a noticeably wider spread, consistent with them representing genuinely different traffic patterns rather than borderline cases of known classes.

To put these numbers in context against other published approaches evaluated on comparable IDS tasks:

None of these comparisons run on identical datasets, so they should be read as rough context rather than a controlled head-to-head. But the pattern is consistent with the motivation for the project in the first place: approaches that rely purely on distance-based or one-shot comparison methods (like the Siamese network) tend to show a wide, inconsistent range in zero-day detection, while methods explicitly designed with a statistical out-of-distribution mechanism this one included land in a tighter, higher range.

Where this breaks down, and what I'd do differently next

The Mahalanobis approach works well here specifically because it assumes the known-class embeddings roughly follow a multivariate Gaussian distribution, and that assumption held up well enough on this dataset to give clean separation. That's not guaranteed to hold in general in more complex or more heterogeneous embedding spaces, that Gaussianity assumption could break down, and a fixed covariance matrix estimated once after training won't adapt if the notion of "normal" traffic drifts over time. A non-parametric or adaptive distance metric would be a more robust choice if this were extended to a setting with more diverse traffic sources.

There's also the encoder itself. Two fully connected layers were enough to produce an embedding space that separated classes cleanly for this dataset, but a transformer-based or attention-driven encoder could plausibly learn richer representations, particularly if the feature set grew beyond the 56 dimensions used here.

And the current setup, while it avoids retraining when a known class needs to be added, still recomputes the global mean and covariance for zero-day detection as a static, one-time step. A genuinely continual system would need a mechanism to update that distribution incrementally as legitimately new "known" behavior gets folded in, without letting drift quietly widen the boundary of what counts as normal.

The technical takeaway

The result I keep coming back to isn't the 83.96% classification accuracy it's the fact that separating classification from anomaly detection into two distinct mechanisms operating on one shared embedding space is what made zero-day detection work reliably at all. Trying to get a single classifier to both categorize known attacks and reject unknown ones tends to produce a system that's mediocre at both. Here, the prototypical network's job was narrowly to produce a well-structured embedding space; the Mahalanobis distance and chi-square test's job was narrowly to characterize what "known" looks like statistically within that space and flag deviations from it. Neither component needed to be more sophisticated than it was the architecture did the work, not the individual pieces.