Four NLP Terms a Security Analyst Should Understand

Natural language processing (NLP) can be used by analysts to automatically extract entities like IPs, domains, and threat actors from unstructured reports. That's NLP in action. The terminology, e.g., tokenization, part-of-speech tagging, lemmatization, named entity recognition, may make you feel like you're in a linguistics class instead of learning about security tooling. Here's what each term means using a sample sentence you'd see in a threat report: "The malware contacted 198.51.100.23 and downloaded an update from maliciousupdate.example."

Tokenization: breaking a sentence into manageable pieces

Before a computer can do anything with a sentence, it has to split it into individual chunks called tokens. This includes the words, numbers, and punctuation. This sounds like a trivial task, but it is vital to the process. A generic tokenizer might split 198.51.100.23 into several tokens because it treats periods as punctuation rather than recognizing the entire sequence as an IP address. This is an issue I ran into while building an IOC extraction tool. The tokenizer would break IP addresses and domains apart instead of recognizing them as a single value.

Please note, tokenizers also split on spaces, commas, and other punctuation. But periods are the specific culprit here.

Part-of-speech tagging: labeling the role of each word

Once a sentence is broken into tokens, part-of-speech (POS) tagging labels each one. "Malware" is a noun, "contacted" is a verb, "an" is an article. This is elementary, my dear Watson. POS tagging labels words according to their grammatical role, while later parsing steps can use those labels to determine relationships, such as which entity performed an action and which entity received it.

That distinction matters. A report could mention two IP addresses in the same sentence, and knowing whether one of them is sending traffic or receiving it changes an analyst's course of action. Do they block the source, monitor the destination, or something else altogether? POS tagging helps an automated tool make a distinction instead of just dumping every IP address it identifies into one undifferentiated list.

Lemmatization: reducing words to their dictionary form

Lemmatization strips a word down to its dictionary form. For instance, "contacted" becomes "contact," "downloaded" becomes "download." What is the reason? If you're searching thousands of threat reports for beaconing activity, you don't want separate searches for "beacon," "beacons," "beaconed," and "beaconing." Lemmatization reduces those variations to the same dictionary form so a single search can match them all.

Named entity recognition: pulling out the important information

This is the one most tied to the project I've been documenting. Named entity recognition (NER) scans text and labels spans of it belonging to a category. The category could be a person, an organization, or in the security context, an IP address, a domain, a CVE number, or a threat actor. Everything covered thus far (tokenizing, tagging, reducing words to their root form) is the foundation for NER. NER answers the question an analyst most cares about: what are the actionable items mentioned in this report?

Not every IOC extraction tool relies on machine learning. Some use regular expressions and other rule-based techniques for structured indicators like IP addresses and domains. NER becomes useful for identifying less structured entities such as threat actors, malware families, or organizations.

Why this is worth knowing

You don't need to implement any of the code involved here from scratch to benefit from its potential. CS0–003 mentions using tools and automation to process threat intelligence efficiently. Every tool that claims to automatically extract IOCs from text is using some version of this pipeline under the hood. Understanding the pieces that make this possible means you know what a tool is doing, where it's bound to make mistakes (i.e., edge cases), and why the output still needs to be checked by an analyst before anyone acts on it.

None of this requires writing code yourself. It requires understanding what each step does and why. That is the more useful skill anyway.

Sources: