Think Like a SOC Analyst: Investigating a Security Incident

In a previous article, we analized a machine from a technical perspective, focusing on identifying the artifacts left on the system and answering each of the investigation objectives.

If you'd like to follow the complete technical analysis first, you can read it here:

Shcout Hacviser Write-up — Forensic In this write-up, I share the analysis and resolution process for a practical cybersecurity scenario, focusing on Linux…

In this article, we'll revisit the exact same forensic scenario, but with a different objective.

Rather than focusing on the technical steps taken to answer each question, we'll approach the investigation through the mindset of a SOC Tier 1 analyst. The emphasis is not on the commands executed, but on the reasoning behind the investigation: what questions should be asked first, which evidence should be prioritized, and how individual findings are correlated to understand the incident.

Now, let's investigate the incident as a SOC analyst would.

Alert Triage

Although this investigation did not originate from a real-time SOC alert, the same analytical methodology can be applied during a forensic investigation.

At the start of the analysis, the only assumption was that the Linux server might have been compromised. No information was available regarding the attack vector, the affected accounts, or whether the compromise had even been successful.

The first objective was therefore to validate whether there was evidence of malicious activity or whether the observed behaviour could be explained by legitimate administrative actions.

Authentication logs provide one of the best starting points for answering that question.

By reviewing logs from authentication records, it became clear that a single external IP address generated a large number of failed login attempts against the same user account within a short period of time.

The consistency of the source IP, the frequency of the attempts and the targeted nature of the activity strongly indicated a brute-force attack rather than normal user behaviour.

At this stage, the investigation already allows us to identify:

• The attacker's IP address.
• The targeted service.
• The targeted user account.

These findings establish the first confirmed evidence that the server was under attack and provide the starting point for reconstructing the compromise.

Scoping the Incident

Confirming a brute-force attack is only the beginning. The next question every analyst must answer is:

Did the attack actually succeed?

An unsuccessful brute-force attack has very different implications from one that results in a compromised server. Determining the scope of the incident therefore becomes the immediate priority.

Authentication records revealed that the same source IP responsible for the failed attempts eventually authenticated successfully. Even more importantly, additional successful logins appeared shortly afterwards using the root account.

This immediately changes the severity of the incident.

Instead of dealing with a compromised user account, the investigation now involves a host where an attacker achieved full administrative privileges.

At this point, the investigation expands to answer questions such as:

• Was only one account compromised?
• Did the attacker obtain root privileges?
• How long did the attacker remain active?
• What actions were performed after gaining access?

The authentication timeline demonstrated that the attacker maintained several interactive sessions over approximately forty minutes, indicating hands-on-keyboard activity rather than a fully automated attack.

Root Cause Analysis

Knowing that root access was obtained is not enough.

One of the primary objectives during incident response is understanding how the attacker managed to elevate privileges. Without identifying the root cause, the same compromise could happen again even after the attacker has been removed.

Rather than immediately searching for malware or persistence mechanisms, the investigation first focused on reconstructing the attacker's actions after the initial compromise.

Reviewing the user's shell history quickly revealed the privilege escalation technique.

Instead of exploiting a kernel vulnerability or a publicly known local exploit, the attacker abused a misconfigured SUID binary.

A system-wide review of privileged executables confirmed that OpenVPN had been incorrectly configured with the SUID bit enabled. This misconfiguration allowed the attacker to execute OpenVPN's scripting functionality to spawn a privileged shell while preserving root permissions.

From a defensive perspective, this is an important finding because the root cause is not the OpenVPN application itself, but an insecure system configuration that transformed a legitimate binary into a privilege escalation vector.

At this point, the attack path becomes much clearer:

• Initial Access → Brute-force attack
• Privilege Escalation → Misconfigured SUID OpenVPN binary
• Result → Full root compromise

Scoping Persistence

Once an attacker obtains root access, analysts must assume that persistence mechanisms may have been deployed.

The next objective is therefore to answer another fundamental question:

If the attacker disconnects today, can they come back tomorrow without repeating the attack?

The investigation initially focused on services capable of automatically launching processes during system startup.

A suspicious systemd service responsible for launching a VNC process immediately attracted attention. Although it appeared to be the persistence mechanism, further investigation suggested it was only responsible for starting another payload.

Expanding the investigation to files created and modified during the compromise window revealed a far more suspicious artifact.

A hidden executable located under /etc/* exhibited several indicators of compromise:

• Created during the attack timeline.
• Owned by root.
• Executable permissions.
• Unusual ownership relationship.
• Hidden inside a system directory where executables are uncommon.

These characteristics strongly suggested that the executable represented the actual root-level backdoor, while the VNC service simply ensured its execution after every reboot.

Distinguishing the launcher from the payload provides a much more accurate understanding of how persistence was implemented on the compromised host.

Building the Incident Timeline

By this stage, individual pieces of evidence have been collected from multiple forensic sources.

However, isolated artifacts rarely tell the complete story.

One of the final tasks during incident response is correlating every finding into a chronological sequence that explains exactly how the compromise unfolded from initial access to persistence.

By combining authentication logs, shell history, privilege escalation evidence and persistence artifacts, the complete attack path can be reconstructed.

Incident Timeline

1. Initial Access

The attacker successfully brute-forced credentials for a valid user account.

2. Credential Access

The compromised credentials were used to establish authenticated a vulnerable service sessions.

3. Execution

The attacker interacted with the system through vulnerable service sessions.

4. Privilege Escalation

A misconfigured SUID OpenVPN binary was abused to obtain root privileges.

5. Persistence

A hidden root-owned executable was deployed and configured to execute through a VNC-related service.

6. Objective Achieved

The attacker established persistent root-level access to the compromised host.

Rather than viewing these artifacts independently, correlating them into a chronological timeline provides a complete understanding of the intrusion, allowing responders to determine not only what happened, but also how the attacker progressed through each phase of the compromise.

This structured methodology — triage, scoping, root cause analysis, persistence analysis and timeline reconstruction — is the same investigative workflow followed by SOC and DFIR teams during real-world incident response engagements.

Containment, Eradication & Recovery

Once the investigation has identified the attack path, privilege escalation method, and persistence mechanisms, the focus shifts from analysis to response.

The compromised host should be isolated to prevent further attacker activity while preserving forensic evidence. After containment, all persistence mechanisms and malicious artifacts must be removed, compromised credentials rotated, and the misconfigured SUID OpenVPN binary corrected to eliminate the privilege escalation path. Finally, the system can be safely returned to operation, with enhanced monitoring to detect any signs of reinfection or continued attacker activity.

Post-Incident Activities

The investigation doesn't end when the system is back online.

All findings should be documented in a formal incident report, including the attack timeline, root cause, affected assets, and remediation actions. The lessons learned from the incident should then be translated into security improvements — for example, enforcing stronger authentication policies, hardening privileged binaries, reviewing SSH access controls, and improving monitoring to detect similar activity earlier in future investigations.

I hope this article has helped you see how a SOC analyst approaches a forensic investigation and has given you a different perspective on analyzing compromised systems.

If you enjoyed this article, found it useful, or have a different way of approaching the investigation, I'd love to hear your thoughts in the comments. Your feedback always helps improve future content.

And if you're interested in more hands-on SOC, DFIR, and cybersecurity content, don't forget to follow me on Medium so you don't miss the next article.

Thanks for reading, and see you in the next investigation. 🔍