Effective SOC Topology (part 1): Triage 101; Noise, beautiful noise

"Jobs' vision of Signal was the top 3 to 5 things you have to get done in the next 18 hours. Anything that stops you from doing that is the noise. His signal to noise ratio was 80:20." — Kevin O'Leary, The Diary of a CEO

While this quote is often disseminated in startup culture like a passage from scripture, it holds much truth, especially in blue teaming, but the "why" will be answered later.

As the title suggests, this article series is dedicated to documenting progress on solo building a SOC topology that is intended to deliver promising blue team operation result statistics and also have it withstand the change that the tech landscape is accustomed to — And perhaps to let you, as a viewer reflect on what your setup & tech stack is better or worse at than my build.

Since I already went past the phase of planning and implementing the software necessary, here is an "Alpha Release" graph:

The tech stack for this is:

Linux host — Wazuh Manager + linux agent endpoint — OSTicket — Suricata — Shuffle SOAR

While the topology will see many alterations in the near future, I will continue to use a cloud solution for SOAR since it is by far the most resource-exhausting of all SOC components. Most of the remaining system memory will be allocated to a local python AI server (You will find out how AI comes into all thi next episode! Or part)

To return to the question of why Steve Jobs was also a visionary in Security Operations (or at least his philosophy), we must take a look at how Wazuh SIEM reacts on a post-vanilla install login with all those newly integrated technologies.

Usually, on fresh login with a Wazuh-only defense setup the total alerts would not have ranged past ~50, but security solutions are paranoid by default: When tinkering SOC labs, this is the part where you have to have to engineer optimal operation conditions: Customizing rules, removing recurring false positive triggers, automation and more automation.

Yet this article aims to tackle the part of alert Triage, not alert Engineering: It will be demonstrated how, in spite of harsh conditions, good triage practice can still overcome such setbacks and correctly asses the _true positives, b_y triggering simulated malicious activity on the same endpoint.

• Looking back at the quote from the beginning, alerts are triaged in order based on severity regardless of rule configuration as a disciplinary means. The workflow should be: investigate critical alerts -> correlate to lower-severity ones by matching logical patterns (e.g. same targeted endpoint/ matches hypothesis) -> Triage writeup; Signal-to-noise ratio stays 80:20.

And finally, to test Triage operations under this specific setup I will simulate 3 common attack types:

• DoS
• Phishing
• Backdoor

After which, I will try to establish an entire story based only on the confirmed logs; So here is the simulation plan —

1. Pre-operation setup (simulating threats, creating tickets)

2. SIEM operator opens tickets from server

3. Map each issue inquiry to specific logs

To make the scenario more immersive, rudimentary tickets were created on the self-hosted OSTicket platform to hint at the existence of every perpetrated offensive technique, among which:

• A "Phishing campaign" ticket: Notifies that network systems are targeted by phishing attacks which are believed to
• The "Platform memory allocation issue" ticket: A high-priority query, this is the part that also proposes specific objectives to the SOC;

• Determine the timespan of the attack;

• Investigate IoC (indicators of compromise);

• Provide a hypothesis and solution in due time;

Now after exploit and ticketing phases, it is time for secOps; This is also a good time to test the following experimental initial-test Triage framework:

I. Severity assessment II. Threat intel documentation III A. Initial context determination III B. Deescalate and disregard III Phase 2: Event hypothesis IV A. Active threat Hunting IV B. Passive threat hunting IV C. Threat escalation

This framework is developed side-by-side while studying the Cyber Kill Chain rather than basing it on a preexisting defensive one. It is also shaped like a tree graph instead of a linear vector. Every roman numeral represents the next step in the framework, and every alphabetic character represents a user or bot-led decision. Again, this is not a final release and it is intended to be gradually adjusted for practicality and automation needs.

To put it to the test, it will be used as basis for every log investigation;

But before the investigation itself, it is worth noting that Wazuh is a pain to configure with extensive security solutions. For instance, it detects up to 50 false positives of "trojaned binaries" in my case. Researching the issue before starting with step I led me to an active discussion, which may be referenced at https://github.com/wazuh/wazuh/issues/20363, the bug still being unaddressed after 3 years as of time of writing: This is the downside of Free & Open Source software, where you are part of the development process by implementing patches. My fix was to disable <check_trojans> in /var/ossec/etc/ossec.conf.

I. Severity Assessment

In triage, highest severity is assessed first; There are 130 medium severity hits and 1 high severity hit after filtering out known false positives in my case; Here is the DQL query:

not full_log: "Trojaned version of" AND not rule.mitre.technique: "Sudo and Sudo Caching"

Some common false positives were left out of the exclusion query, such as "Defense Evasion" as it will be elaborated on in the next stages for every need.

II. Threat Intel Documentation

Looking at the issues reported on the ticket server, we already know that there was a phishing campaign targeting network endpoint users that was is thought to have evolved into a malware threat based.

Mapping the hypothetical incident to MITRE ATT&CK, it is considered that the technique used for initial access was phishing, and it is up to the SOC to determine what steps it took to install malware, and most importantly to locate and contain it.

The log most relevant to tracing the malware is the high severity one, correctly alerting an "Endpoint Denial of Service" targeting resource exhaustion.

Thankfully for the defending party, the log was very verbose: "Out of memory: Killed process 3032 (python3) total-vm:3363008kB, anon-rss:1236016kB, file-rss:2460kB, shmem-rss:0kB, UID:1000 pgtables:6276kB oom_score_adj:0"; This signifies the execution of a malicious python program that had the potential to cause service disruption, and that it also was killed by the system before causing availability impact.

We now know that the malware most likely failed in only one aspect of the CIA triad, so the next step is to investigate origin and other possible risks, with a mean timestamp of Sep 12, 2026 @ 14:41:17.84.

III A. Initial context determination

True positives may not be disregarded. To better determine the timespan and origin of attack, filter with DQL based on time. Now with an optimized filter, 33 relevant hits are shown; Here are a few takeaways that could build a hypothesis:

• Netstat — Port 22 open and listening, could indicate based on the ATT&CK framework as a possible Execution vector. Paired with the known phishing campaign, a user could have installed a "phone-home" reverse shell.
• rule.mitre.tactic:Defense Evasion, Persistence, Privilege Escalation, Initial Access — This value is what I suspect that the server logged the malware installation as. However, it becomes difficult to determine the exact hit as the one where sudo was used to install it, but the most likely culprit was the one at 14:20:43.184, as it was the last before malware fired and aligns with what the OSTicket platform report notes; It will be used to answer "Determine the timespan of the attack".

Since all relevant logs before the High One were reviewed, now it is time to see how far the adversary has moved in the ATT&CK framework by filtering for after 14:41:17.84, with the first log being the Denial of Service hit; The very second one after that is another suspicious entry, specifically "PAM unable to dlopen(pam_lastlog.so)". I am unfamiliar with it, so it is necessary to use online threat intelligence to match any known attack patterns.

As a security analyst, never omit these questions when triaging possible offensive techniques:

• What negative impact could this bring?
• What attack pattern is it related to?
• How could it vary while escalating privileges?

Coming back from researching possible pam_*.so attack patterns, it is confirmed that a backdoor was successfully established on the endpoint, and logs indicate that on every backdoor connection pam is open, while on disconnection it is closed. further looking at every session, it seems that after initial backdoor connection, multiple ones were reopened in the course of ~10 minutes, after which there are no more such log patterns based on pam: As no other sign of attack patterns are noted on the date of September 13, it can be confidently noted that the attack timespan was between Sep 12, 2026 @ 14:41:17.84 and < Sep 13.

III Phase 2: Event hypothesis

We know that the OSTicket server admin most likely installed malware propagated by a phishing campaign. It is also noted that backdoor activity patterns are given away by activity related to the pam file and that there are no clear IoCs (indicators of compromise).

The hypothesis is as follows: Adversary established reverse TCP connection (ATT&CK Execution) through malware installed by admin from a phishing mail (ATT&CK Initial Access). The first action done by the attacker was to attempt Endpoint DoS (ATT&CK Impact) but with limited impact, and afterwards reconnected by port 22 repeatedly with multiple failed attempts at PrivEsc, after which the threat remained dormant. Backdoor malware is not contained at the time of triaging.

How accurate it is: In the exploit simulation phase, the "Suricata OSTicket" Wazuh Endpoint downloaded a python malware file from my lab exploit server which contained an SSH backdoor. As the Client (Ubuntu server, in the case of the reverse shell simulation) connected to the Server (host), it exploited memory allocation by repeating a Python function until the Client killed the script. Afterwards, attempting to find edge cases where data compromise could be possible were tried but some shell outputs(such as the yes/no selections) were only viewable on the VM itself, leading to many reconnections that are viewable in logs as pam file activity. Overall, triage proved to be efficient and accurate for this scenario.

Note: Since this is a triage-only scenario, threat hunting will be omitted, but logical next steps would be to contain the malware while minimising downtime: Applying managerial (notifying stakeholders of phishing campaigns) and technical controls are necessary in such a case. Threat actor is regarded as cybercriminal adversary rather than APT for sticking to a stingle technique rather than applying lateral movement or attack vector variations.

Summary

A backdoor was succesfully established, and the only impact it was successfully used for based on CIA triad was Availability. Triage was only efficient because best practices were applied:

• Maintain an 80:20 signal-to-noise ratio where signal is the investigation leads that are logically related to the incident's context, while noise is likely to be false positives (based on recurring patterns such as sudo caching while investigating pre-escalation movement). This speeds up the time to hypothesis and avoid alert fatigue.
• Possible known attack pattern? Research online intel sources as was applied during phase III, but it would have been better to perform research at phase II. It was still of critical importance to determine the threat.
• Create timestamp points of reference which are written in triage notes and later repurposed for writing a hypothesis.

Lessons Learned

• A baseline for alerts is necessary. Determine what are usual smoking guns for repeated true positives and also note what would unusual mean in the logs.
• Phase II should always include online intel research when not confident with identifying the threat patterns.
• Solo triaging doesn't hold up with larger scales than simulated in the scenario. Try creating automation pipelines in the future.

Building and simulating this scenario was fun and revealed & remediated critical knowledge gaps. I hope you learned something about your own SOC infrastructure and workflow while reading, and consider reading my other posts if you like this kind of article format.