Post cover image
Splunk analytical dashboard processing raw kernel telemetry into visual, actionable security metrics.

June 11, 2026

Insider-Threat-I: Simulating an Advanced Malicious Actor and Engineering the Splunk Telemetry…

Every security analyst remembers the first time they realized that the systems we trust to protect enterprise infrastructure are, by…

redacted

7 min read

Every security analyst remembers the first time they realized that the systems we trust to protect enterprise infrastructure are, by default, lying to us by omission.

I was diving deep into my TryHackMe Security Operations Center (SOC) L2 training pathway, specifically looking at advanced purple team simulation rooms like Tempest and Follina. The foundational rule of those environments is simple: if you want to know how your defensive perimeter actually holds up under fire, you have to break things yourself, analyze the footprint left in the logs, and actively measure the boundaries of your visibility.

Fueled by that exact mindset, I stepped back from the textbook modules and built a dedicated, 3-VM virtual threat playground (SOC Lab). I mapped the environment out on an isolated, private VirtualBox internal network switch (soc-lab) using a static 10.10.10.0/24 subnet. The goal was straightforward: simulate a realistic insider threat compromise from initial reconnaissance to catastrophic asset sabotage, and watch how my centralized Splunk SIEM platform handled the telemetry stream.

What I discovered instead was a massive, silent endpoint blind spot. An attacker using standard user credentials could raid the server, exfiltrate data, and drop bombs on the file system while leaving the security team completely in the dark.

Here is exactly how I executed the attack lifecycle, exposed the architecture of a default Linux visibility gap, and engineered a custom kernel-level auditing policy to restore total visibility.

Part 1: The Anatomy of the Attack (Red Team Emulation)

To understand the detection engineering fix, we first have to look at the exact digital footprint left by the adversary. For this exercise, I simulated an insider threat scenario where a standard, unprivileged corporate profile named richard was systematically targeted and compromised from an external deployment box.

1. Active Infrastructure Mapping

Operating from a Kali Linux attack node (10.10.10.30), the offensive lifecycle initiated with an active network sweep against the targeted Ubuntu web server host (10.10.10.40) to map the exposed attack surface and discover accessible service banners.

nmap -sS 10.10.10.40

The initial automated port sweep capturing service versions running on the target.

On the wire, this massive volume of connections targeting Port 22 created immediate noise. Capturing the raw packet flow locally via _tcpdum_p — a network inspection tool, instantly validated the existence of a high-velocity reconnaissance sweep.

Packet analysis identifying the rapid TCP handshake patterns characteristic of an active port scan.

2. Initial Access via Automated Credential Spraying

With Port 22 confirmed open, the attack shifted toward credential harvesting. Using Metasploit's automated SSH login auxiliary infrastructure, I fed targeted dictionaries against the endpoint to brute-force authentication access for the local profile richard.

msf> use auxiliary/scanner/ssh/ssh_login
msf auxiliary(ssh_login) > set rhosts 10.10.10.40
msf auxiliary(ssh_login) > set username richard
Setting up the target variables inside the Metasploit framework module.

Once executed, the framework hammered the server's authentication daemon.

The active brute-force process cycling through wordlists sequentially against the targeted user account.

The velocity of the attack ultimately paid off, isolating a valid credential match and triggering an active interactive terminal session into the server core. Initial Access was officially established.

Metasploit successfully establishing a backdoor shell session using compromised valid accounts.

3. Post-Exploitation, Exfiltration, and Sabotage

Raw reverse shells are inherently unstable and brittle. To ensure long-term stability and prevent the session from dropping unexpectedly during operational tasks, I systematically upgraded the interactive connection to an advanced Meterpreter implant and hardened by spawning a Python pseudo-terminal:

python3 -c 'import pty; pty.spawn("/bin/bash")'

Acting as richard, I began sweeping the system for corporate assets. It didn't take long to locate high-value target files sitting plainly within the user's home directory.

None

To pull the information out of the network without setting off traditional network intrusion flags, I deployed a classic administrative trick: spinning up an ad-hoc, unencrypted Python HTTP server right out of the target folder over port 8000.

python3 -m http.server 8000
Spawning a localized web server to stage data exfiltration paths.

By curling the open port from the attack machine, the proprietary corporate data was seamlessly drawn out of the server environment. Because the transfer occurred over a completely unencrypted stream, packet inspection tools captured the plaintext contents moving across the wire.

Raw network data stream verifying the cleartext capture of corporate trade secrets during exfiltration.

With the data stolen, the adversary completed the destructive phase of the insider threat lifecycle. Using the native Linux rm utility, the targeted directories and configuration schemas were permanently deleted from disk, leaving behind a custom text file mockingly alerting the organization to the compromise.

The final act of sabotage — wiping local configurations to degrade system integrity.

Part 2: Confronting the Blind Spot (The Blue Team Reality Check)

Once the red team simulation was complete, I pivoted over to my Splunk Enterprise console (localhost:8000, bridged safely out to my physical host browser via VirtualBox port forwarding) to review the incident timeline and build my alerts.

The initial logs looked fantastic. My SIEM had completely ingested the authentication telemetry. I could see the exact volume of failed sshd attempts mapping to richard right before the successful compromise hit the logs.

Splunk successfully isolating the massive wave of inbound authentication failures leading to the breach.

But then, the data trail went dead.

When I queried my index for the post-exploitation phase — searching for the exact commands that stole our trade secrets and erased our configurations, Splunk came up completely blank. The execution of _python3 -m http.serve_r was missing. The destructive rm system execution didn't exist. As far as my SIEM knew, Richard had logged in via SSH and simply sat there doing absolutely nothing.

Why Did This Happen?

This is the core technical realization that every engineer must face: Default Linux auditing configurations are completely blind to standard, unprivileged user actions.

When you install a modern Linux enterprise distribution and turn on native logging daemons like auditd, the baseline ruleset is heavily optimized for compliance and system integrity. It monitors authentication endpoints (/var/log/auth.log), configuration modifications, and administrative privilege elevations via sudo.

However, if a threat actor logs in with valid, low-privileged credentials and runs standard binaries natively from their interactive path, the kernel assumes it's routine, non-threatening daily activity. It ignores the process creation completely to save processing power and log storage space.

In a real-world enterprise scenario, this default behavior means an insider threat can roam your server, copy intellectual property, and delete files completely under the radar of the SOC.

Part 3: Engineering the Solution (Kernel-Level Remediation)

Leaving a critical visibility gap open is an unacceptable defensive failure. To fix this without blinding our SIEM with millions of noisy background system log loops, I designed a targeted, role-based auditing control.

I needed to capture the human footprint on the operating system without monitoring the thousands of automated processes spawned by background daemons. To achieve this, I targeted the execve system call — the specific kernel mechanism responsible for executing new programs.

I opened up the permanent rule configuration file on the target server:

sudo nano /etc/audit/rules.d/audit.rules

And injected two specific, role-based policy mandates targeting the precise User IDs (UIDs) of our admin account (1000) and our suspect profile (1002 — Richard):

-a always,exit -F arch=b64 -F uid=1002 -S execve -k v_user_commands
-a always,exit -F arch=b64 -F uid=1000 -S execve -k v_user_commands

By assigning both rules the exact same custom tracking key (-k v_user_commands), I consolidated the tracking metric into a single, highly indexable keyword within my SIEM environment.

I then recycled the audit engine to enforce the rule space directly into the running kernel memory:

sudo service auditd restart

Part 4: The Payoff (The Insider Threat Monitoring Desk)

With the custom kernel rules deployed, I reran the entire attack emulation chain from the Kali machine. This time, when I stepped back into Splunk and queried my custom key, the telemetry lit up the dashboard.

index="main" "user_commands"

The system call data was beautiful. Because auditd operates deep within the system kernel space, it captured everything natively — even when commands were executed through an interactive, non-login reverse shell channel that bypasses standard text-based history files.

Raw system call events successfully tracking Richard's interactive process executions chronologically.

To turn this raw data stream into something an active SOC analyst could use to defend an enterprise network in real-time, I built a centralized visual portal: The Insider Threat Monitoring Desk.

The final analytical dashboard processing raw kernel telemetry into visual, actionable security metrics.

Dashboard Analytics Breakdown:

Top Executed Binaries by Suspect Users: This bar chart filters standard user behaviors. While normal utilities like bash and ls comprise the high-volume baseline, anomalous tools like /usr/bin/rm and /usr/bin/python3.10 stand out visually, pointing directly to a potential system compromise.

Session Activity Distribution: A donut chart providing an immediate high-level ratio of who is actively driving process creation on the asset. It cleanly segregates administrative tasks (redacted) from the suspect user tracking lane (richard).

Real-Time User Command Timeline: An immutable, structured ledger showing the exact microsecond a binary was called, the process ID, and the exact terminal interface (tty) used to execute it.

Engineering Reflections & The Next Frontier

Building this end-to-end lab completely reinforced why our industry is moving aggressively toward dedicated Endpoint Detection and Response (EDR) models.

Traditional operating system logging utilities like auditd are structurally fragmented. They split single events into separate log lines (type=SYSCALL and type=EXECVE) and natively output command-line text arguments as heavily encoded hexadecimal strings to prevent spacing errors. The sheer analytical processing power required to decode and piece those lines back together at scale is exactly why enterprise teams require unified EDR agents to normalize and stitch process trees together automatically.

As a final engineering step for this ongoing project, my next developmental phase involves exploring shell-level environment instrumentation using the Bash PROMPT_COMMAND variable. By injecting global logging hooks directly into the system shell, we can forcefully stream human-readable command strings straight to syslog the exact millisecond a user presses the Enter key:

export PROMPT_COMMAND='logger -p local6.notice -t bash_history "$(whoami) [$$]: $(history 1 | sed "s/^[ ]*[0–9]*[ ]*//")"'

This creates a dual-layered telemetry system where a malicious actor can run defensive evasion parameters like set +o history or redirect their HISTFILE to /dev/null, yet their actions are already captured by the shell environment hook and forwarded straight to Splunk before the process even finishes executing.

Cybersecurity isn't about deploying a tool and assuming you're safe. It's about hunting down the blind spots, understanding the architecture of what you cannot see, and engineering the custom pipelines required to bring the truth back into the light.

Thanks for walking through this simulation build with me! You can check out the full configuration rulesets, lab scripts, and technical architecture layouts on my GitHub repository here: Insider-Threat-I.

What are your favorite custom auditing tricks to run on corporate Linux endpoints? Let's connect and swap ideas on LinkedIn or follow my lab updates on X @thatboringbro!