Threat Hunting Walkthrough: Operation Dark Door

When system administrators report unusual latency on a core mail server, SOC analysts need to move quickly. In this walkthrough, I will take you through Operation Dark Door — a threat hunting exercise where I used Splunk Enterprise to detect an SSH brute-force attack, verify if a breach occurred, and write a detection alert rule.

The Scenario & Hypothesis

• Scenario: On September 8, 2022, IT noticed performance drops on the core mail server (mailsv1). Security operations suspected an external actor was brute-forcing the server via Secure Shell (SSH).
• Hypothesis: An external attacker is running automated scripts to guess SSH login credentials against Linux user accounts.

Step 1: Initial Discovery & Finding the Attacker

To verify the hypothesis, I searched Splunk for all failed password events on the mail server logs:

• Total Failed Events: 8,154 total failed password log entries were identified.
• Attacker IP Identification: By extracting the source IP address from the log entries using regex parsing, I isolated the primary attacker's IP: 194.8.74.23.

Step 2: Target Analysis

Next, I needed to see which user accounts were being targeted and whether the attacker was targeting existing system accounts or guessing random names.

Using the following Splunk Search Processing Language (SPL) query, I extracted the targeted usernames and generated a summary table:

Key Findings:

• Account Types: The attacker targeted both valid existing users and invalid/non-existent usernames (such as appserver and testuser).
• Top Targets: Accounts like root, appserver, and testuser saw hundreds of automated login attempts within a brief window.

Step 3: Breach Verification ("Did They Get In?")

The most critical question in any brute-force investigation is: Did the attacker successfully compromise an account?

To check for successful authentications, I queried the index for accepted password events:

Result: While Accepted password events existed for valid internal network IP addresses (e.g., 10.2.10.163 and 10.3.10.46), there were zero successful logins from the external attacker IP (194.8.74.23). The brute-force attack was unsuccessful.

Step 4: Detection Engineering

To ensure future brute-force attempts are caught automatically without manual hunting, I built an SPL detection query and converted it into a Splunk Alert.

SPL Query:

index="secure_logs" sourcetype="MailServer" "Failed password"
| rex "from\s+(?<attacker_ip>\d+\.\d+\.\d+\.\d+)"
| stats count by attacker_ip
| where count > 10

index="secure_logs" sourcetype="MailServer" "Failed password"
| rex "from\s+(?<attacker_ip>\d+\.\d+\.\d+\.\d+)"
| stats count by attacker_ip
| where count > 10

Alert Configuration:

• Title: SSH Brute Force Attack Detected
• Condition: Triggered whenever a single source IP records more than 10 failed login attempts within a 5-minute window.
• Action: Save as an active alert to notify SOC analysts for immediate IP blocking/isolation.

Key Takeaway

By leveraging SPL commands like rex, stats, and top, SOC analysts can rapidly pivot from raw log data to actionable threat intelligence. Automating these findings into active alerts turns a manual investigation into proactive defense.