Post cover image

October 17, 2026

Inside Splunk SIEM: Building Detections, Investigating Attacks, and Exposing Visibility Gaps

A hands-on SOC lab exploring SSH brute force, privilege escalation, SPL detection engineering, and simulated data exfiltration.

By Baraa Mohamed

9 min read

A hands-on SOC lab exploring SSH brute force, privilege escalation, SPL detection engineering, and simulated data exfiltration.

Three attack stages. A centralized SIEM. One critical question: how much can your logs actually tell you?

A security alert may indicate suspicious activity, but it does not automatically explain what happened, how far an attacker progressed, or whether sensitive information was successfully transferred.

For a SOC analyst, the challenge is not simply detecting unusual events. It is transforming raw telemetry into actionable detections, investigating the evidence behind each alert, and recognizing when the available data is insufficient to support a conclusion.

In this lab, I built an isolated environment using Kali Linux, Ubuntu Server, and Splunk Enterprise to investigate three attack scenarios:

  • SSH brute force to simulate initial access.
  • Privilege escalation through a misconfigured sudo permission.
  • Simulated data exfiltration over SSH using SCP.

I configured log forwarding, developed SPL searches, tested alert behavior, and examined the limitations of authentication-based monitoring.

The most interesting finding emerged during the final scenario: the simulated file transfer succeeded, but the available authentication logs did not directly establish which file had been transferred.

This investigation explores both sides of detection engineering: building detections that identify suspicious activity and understanding what those detections cannot prove.

The Lab Environment

Before simulating the attacks, I established an isolated virtual network to separate the attacker, target, and SIEM.

The environment consisted of three virtual machines connected through VMware VMnet2 in Host-only mode.

  • Kali Linux — Attacker: 192.168.50.130
  • Ubuntu Server 24.04 LTS — Target: 192.168.50.129
  • Ubuntu Server 24.04 LTS — Splunk SIEM: 192.168.50.131

All three machines operated within the 192.168.50.0/24 subnet.

None

This architecture separated attack generation, security-event collection, and centralized investigation. The target generated the authentication telemetry, while Splunk provided the platform for searching and analyzing those events.

With the environment established, the next step was to verify that the SIEM could receive the evidence required for detection.

Building the Splunk Telemetry Pipeline

A detection rule is only as reliable as the telemetry supporting it.

I configured the Splunk Universal Forwarder on the Ubuntu target to monitor:

/var/log/auth.log

The forwarder sent the events to Splunk Enterprise over TCP port 9997.

The events were indexed in main with the sourcetype linux_secure.

To validate ingestion, I ran the following SPL search:

index=main sourcetype=linux_secure
None

The search returned 126 events, confirming that authentication telemetry was reaching Splunk at that point in the lab.

This initial validation was essential. Before troubleshooting a detection, an analyst should confirm that the relevant events are being collected, forwarded, parsed, and indexed correctly.

With the telemetry pipeline operational, I began testing the detection use cases.

Detecting SSH Brute Force with SPL

The first scenario simulated password guessing against the target's SSH service.

From Kali Linux, I used Hydra with a wordlist containing 155 common passwords. According to the lab results, the correct password was discovered after 149 failed attempts.

None

From a SOC perspective, the primary signal was the concentration of failed authentication events associated with a single source IP address.

A single failed login may be normal. Repeated failures within a short period, particularly when followed by a successful authentication, provide a stronger reason to investigate.

Engineering the detection

I developed the following SPL query to extract the source IP address and count failed-password events:

index=main sourcetype=linux_secure host=target "Failed password"
| rex field=_raw "from (?<src_ip>\d+\.\d+\.\d+\.\d+)"
| stats count as failed_attempts by src_ip
| where failed_attempts > 10

The search filters authentication events, extracts the source address using a regular expression, aggregates the matching events by source, and returns addresses exceeding ten failures.

None

The search identified 176 matching events associated with the attacker IP, 192.168.50.130.

The difference between Hydra's reported 149 failed attempts and Splunk's 176 events is worth investigating. These figures represent different measurements and may be affected by retries, event-generation behavior, or search-window boundaries. A production investigation would reconcile the timestamps and individual events before treating the counts as equivalent.

Configuring the alert

I configured a real-time alert named SSH Brute Force Detected.

The recorded configuration included a trigger condition based on matching results and a ten-minute throttle per source IP.

None

The throttle was intended to reduce repeated notifications from the same source during the configured interval.

This is an important operational consideration: a detection should generate useful investigative signals without flooding analysts with unnecessary duplicate notifications.

Finding: The lab identified concentrated SSH authentication failures and triggered the configured alert.

Detection limitation: A threshold-based search is a starting point rather than a complete brute-force detection strategy. Production deployment would require validating the search window, successful-login correlation, source distribution, and expected benign authentication patterns.

The first scenario demonstrated how raw authentication events can become actionable security alerts. The next scenario examined what happens when an attacker obtains elevated privileges.

Investigating Privilege Escalation Through Sudo Misconfiguration

After obtaining SSH access through the regular ubuntu account, I exploited a deliberately misconfigured sudo permission that allowed the find utility to execute as root without a password.

The command used in the lab was:

sudo find . -exec /bin/bash \; -quit

The weakness was not an inherent vulnerability in find. It was the excessive privilege granted to a utility capable of launching another process.

None

The resulting root shell confirmed that the simulated privilege escalation succeeded.

For a SOC analyst, the investigation now shifts from authentication alone to privileged execution: which command ran, which account initiated it, and whether the resulting privileges were authorized.

Detecting suspicious privileged commands

I used the following SPL search to inspect sudo events associated with root execution:

index=main sourcetype=linux_secure host=target "sudo:" "USER=root"
| rex field=_raw "COMMAND=(?<command>.+)"
| where match(command, "(find|vim|vi|less|python|perl|nmap|cp|mv)\s")

The query extracts the command recorded in the raw event and filters for selected utilities that may warrant additional investigation when executed with elevated privileges.

None

I then tested the alert named Privilege Escalation via Sudo Misconfiguration during repeat exploitation.

None

Interpreting the evidence

The lab captured the suspicious find execution in the root-execution context and demonstrated that the configured alert could identify the tested behavior.

However, command-name matching does not automatically establish malicious intent. Administrators may legitimately use find, python, and other utilities in privileged workflows.

The query also depends on the actual format of the sudo events. Before production use, it would need validation against representative logs, command-line variations, and the host's sudo policy.

A more mature detection could correlate sudo execution with endpoint process telemetry, parent-child process relationships, and changes to privileged accounts or security configurations.

Finding: The lab detected the tested root-privileged command associated with the deliberate sudo misconfiguration.

Detection limitation: The rule covers selected command patterns. It does not identify every possible privilege-escalation technique or establish malicious intent by itself.

With initial access and privileged execution demonstrated, I moved to the final scenario: investigating a simulated transfer of sensitive data.

Investigating Simulated Data Exfiltration

The final scenario involved transferring a simulated sensitive-data file from the Ubuntu target to the Kali machine using SCP over SSH.

The command used was:

scp ubuntu@192.168.50.129:/home/ubuntu/customer_data.csv /home/kali/stolen_data.csv
None

The file represented customer or credit-card data for demonstration purposes. It was a simulated lab artifact, not evidence of an actual customer-data compromise.

The successful transfer demonstrated that the file reached the attacker-controlled machine. The remaining question was whether the monitored authentication logs could independently identify the file-transfer operation.

The telemetry gap

The monitored /var/log/auth.log events did not expose the transferred filename or directly identify the SCP file-transfer operation in the events used for this investigation.

Instead, the available telemetry provided authentication and session-related activity.

This distinction matters because different observations support different conclusions:

  • A successful SSH authentication establishes that authentication succeeded.
  • A short-lived session establishes a brief observed session, provided the relevant events support that interpretation.
  • Neither observation alone establishes that a particular file was transferred.

Authentication telemetry is not equivalent to file-transfer telemetry.

Rather than treating the absence of a direct file-transfer event as proof that no transfer occurred, I explored whether short-lived SSH sessions could provide a behavioral signal for further investigation.

Testing a behavioral detection

I developed the following SPL search to correlate SSH authentication and session-close events:

index=main sourcetype=linux_secure host=target
("Accepted password" OR "session closed")
| rex field=_raw "sshd\[(?<session_pid>\d+)\]"
| transaction session_pid maxspan=15s
| where duration < 3 AND eventcount=2

The search attempts to identify sessions lasting less than three seconds that contain two correlated events within a maximum transaction span of 15 seconds.

None

The search identified short-lived SSH session activity, including a session reported as lasting 1.19 seconds.

I configured a scheduled alert named Suspicious Short-Lived SSH Session, running every five minutes.

None

What the Investigation Revealed

The three scenarios demonstrated different aspects of Splunk-based detection engineering.

SSH brute force: Repeated failed-password events provided a measurable signal that could be transformed into a threshold-based alert.

Privilege escalation: Sudo telemetry exposed the tested root-privileged command associated with a deliberately insecure permission.

Simulated data exfiltration: Short-lived SSH session behavior generated an investigative lead, but the available authentication events did not directly establish which file had been transferred.

These results describe the tested lab scenarios. They do not establish universal detection coverage or demonstrate that the same rules would identify every variant of these techniques in production.

Broader validation would require additional attack patterns, benign test cases, event-format variations, and systematic evaluation of false positives and false negatives.

Lessons for SOC Analysts

1.Telemetry validation comes first

The forwarding pipeline had to be verified before the detection rules could be evaluated. Missing events, incorrect sourcetypes, or inconsistent field extraction can undermine an otherwise well-designed query.

2.Alert quality matters as much as detection logic

Thresholds and throttling affect how useful an alert is in a real SOC workflow. The goal is to provide meaningful investigative signals without unnecessary duplication.

3.Configuration weaknesses can enable privilege escalation

The sudo scenario demonstrated how excessive permissions can turn a legitimate system utility into a path toward root access. Reviewing sudoers policies and monitoring privileged execution are complementary defensive measures.

4.Log visibility defines investigative limits

Authentication logs may reveal connection-related activity without identifying files transferred over an encrypted connection. Additional endpoint, file-access, or network telemetry may be necessary to establish what happened.

5.Alerts require corroboration

A rule identifies activity matching a defined condition. Determining whether that activity is malicious requires context, supporting evidence, and validation of the assumptions behind the detection.

Defensive Recommendations

Based on the scenarios explored in this lab, several improvements would strengthen a comparable monitoring environment.

  • Harden SSH access: Use appropriate strong authentication, restrict access to authorized users and network sources, and consider MFA where supported.
  • Apply least privilege: Audit sudoers entries and remove unnecessary permissions that allow powerful utilities to execute with elevated privileges.
  • Expand endpoint visibility: Collect suitable process-creation and command-line telemetry to reconstruct activity beyond authentication events.
  • Improve file-transfer monitoring: Combine endpoint evidence, file-access auditing, network flow analysis, and DLP controls where appropriate.
  • Validate detections continuously: Test expected matches, benign cases, command variations, and telemetry changes before relying on a rule operationally.
  • Document detection limitations: Record the assumptions, known blind spots, confidence level, and additional evidence required to confirm an alert.

These measures work together. No single log source or detection rule can provide complete visibility into every stage of an attack.

Conclusion

This lab started with a straightforward objective: build a Splunk-based detection pipeline and use it to investigate three simulated attack stages.

The brute-force scenario demonstrated how repeated authentication failures could be transformed into an actionable alert. The privilege-escalation scenario showed how monitoring privileged command execution could expose behavior associated with an insecure sudo configuration.

The final scenario presented a more difficult challenge. The simulated file transfer succeeded, but the authentication-log search did not directly establish which file had moved. Instead, it identified short-lived SSH session behavior that required further investigation.

That limitation was an important part of the result.

Effective SOC work is not simply about making detections trigger. It is about understanding the evidence behind each alert, validating the logic against real event structures, and identifying the additional telemetry needed to support a defensible conclusion.

A SIEM alert can tell you where to look. Evidence determines what you can conclude.

Lab environment: Kali Linux, Ubuntu Server 24.04 LTS, Splunk Enterprise, Splunk Universal Forwarder, Hydra, and SCP.

Scope: Authorized simulation within an isolated VMware Host-only network. The results describe the tested scenarios and do not establish universal detection coverage.

References