SOC Analyst Day 8: Linux Investigation, Process & Network Triage, and User Management

SOC Analyst Day 8: Linux Investigation, Process & Network Triage, and User Management

My hands-on learning journey into Linux security monitoring and incident investigation.

Introduction

Linux is widely used across servers, cloud infrastructure, and security environments. For a Security Operations Center (SOC) analyst, understanding Linux is essential for investigating suspicious processes, reviewing login activity, analyzing network connections, and identifying potential security issues.

During Day 8 of my SOC learning journey, I focused on practical Linux commands that help answer important investigation questions:

Who logged into the system?

Which processes are running?

What network connections are active?

Who owns a file, and who has permission to modify it?

Are there suspicious services or scheduled tasks?

How can command-line tools help analyze logs efficiently?

This article summarizes the commands and investigation techniques I studied.

1. File Permissions and Ownership

File permissions are an important starting point when investigating Linux systems. They help analysts understand who owns a file and which users can read, write, or execute it.

Useful commands

ls -l id

ls -l displays file permissions, ownership, file size, and modification information.

id displays the current user's UID, primary GID, and group memberships.

For example:

ls -l /opt/lab/script.sh

When reviewing a file, consider:

Does the owner make sense?

Can an unexpected user modify it?

Is the file located in an unusual directory?

Why does a privileged account own it?

Could another user modify a script that runs with elevated privileges?

These questions help identify potentially risky configurations that deserve further investigation.

2. Users and Authentication

Authentication records help establish who accessed a system and when that access occurred.

Useful commands

who w last -ai | head -20 getent passwd getent group sudo

What these commands do:

who — displays currently logged-in users.

w — displays logged-in users and their activity.

last -ai | head -20 — shows recent login history, including remote host information where available.

getent passwd — retrieves account entries from the system's configured account database.

getent group sudo — retrieves information about the sudo group, if configured on the system.

Questions for a SOC analyst

During an investigation, ask:

Was the account expected to log in?

Was the login time normal?

Did a remote IP address appear?

Does the account have sudo or administrative privileges?

Does the login timeline match the reported alert?

A single unusual login is not proof of compromise. It should be correlated with other evidence, such as authentication logs, process activity, and network connections.

3. Process Investigation

Processes represent programs currently executing on a Linux system. Investigating them can help identify unexpected applications, suspicious command execution, and possible signs of compromise.

Important commands

ps -ef --forest pstree -ap pgrep -a sshd readlink -f /proc//exe tr '\0' ' ' < /proc//cmdline

Command explanations:

ps -ef --forest — displays processes in a tree-like parent-child format.

pstree -ap — displays the process hierarchy, including process IDs and arguments.

pgrep -a sshd — finds processes matching sshd and displays their command lines.

readlink -f /proc//exe — resolves the executable path associated with a process.

tr '\0' ' ' < /proc//cmdline — makes the process command-line arguments readable by replacing null separators with spaces.

Replace with the actual process ID.

What should raise suspicion?

Look more closely when you encounter:

A shell spawned by a web server without a clear operational reason.

Programs executing from temporary directories such as /tmp.

Unexpected interpreters, such as Bash, Python, or Perl.

Obfuscated or unusual command-line arguments.

Root-owned processes whose activity has no obvious business justification.

These are investigation indicators, not automatic proof of malicious activity. Legitimate administration and application behavior can produce similar patterns.

4. Network Triage: Who Is the Host Talking To?

Process investigation becomes more useful when combined with network evidence. A suspicious process may have an unexpected listening port or an outbound connection that requires investigation.

Useful commands

ss -t ss -u ss -l ss -n ss -tuln ss -tpn ip addr ip route lsof -i -P -n CommandPurposess -tDisplay TCP socket informationss -uDisplay UDP socket informationss -lDisplay listening socketsss -nShow numerical addresses and portsss -tulnShow listening TCP and UDP sockets numericallyss -tpnShow TCP sockets with numerical addresses and process information where availableip addrDisplay network interfaces and IP addressesip routeDisplay the routing tablelsof -i -P -nList network-related open files and sockets without resolving names or ports

Investigation approach

When examining a suspicious connection:

Find the remote IP address and port.

Identify the process associated with the connection, where possible.

Inspect the process command line.

Identify the user running the process and its executable.

Correlate the activity with logs and DNS evidence.

This process helps connect network activity to the program and account responsible for it.

5. Linux Filters for Log Analysis

SOC analysts frequently work with large amounts of text, including authentication logs, application logs, and command output. Linux filters help search, extract, count, sort, and compare this information.

Essential filters

tee — save output while displaying it

command | tee output.txt

Writes output to both the terminal and a file.

grep — search for matching patterns

grep "error" /var/log/syslog

Finds lines containing the specified pattern.

cut — extract fields

cut -d ':' -f1 /etc/passwd

Extracts the first colon-separated field, which contains usernames in standard /etc/passwd entries.

wc — count lines, words, or characters

wc -l /var/log/auth.log

Counts lines in the specified file.

sort — sort data

sort username.txt

Sorts lines alphabetically by default.

uniq — remove consecutive duplicate lines

sort file.txt | uniq

Sorting first allows identical lines to become adjacent, making duplicates easier to remove.

comm — compare sorted files

comm file1.txt file2.txt

Compares two sorted files and separates lines that are unique to each file from lines shared by both.

These tools can be combined to investigate patterns efficiently without manually reading every line.

6. Basic Linux Tools and File Discovery

Finding files is another important part of a Linux investigation. Analysts may need to locate scripts, configuration files, or other artifacts associated with suspicious activity.

Common commands

find /path -name "filename" locate filename date time command whoami who id

find — searches for files and directories.

locate — searches an indexed file database; results may be outdated.

date — displays the current date and time.

time command — measures command execution time.

whoami — displays the current effective username.

who — displays currently logged-in users.

id — displays user and group identity information.

Useful find options

-name — search by filename.

-type f — match regular files.

-newermt 'YYYY-MM-DD' — match files newer than a specified date.

-size +25M — match files larger than 25 MB.

-user USER — match files owned by a specified user.

-exec ... {} ; — execute a command for each matching result.

For example:

find /tmp -type f -name "*.sh"

This searches /tmp for regular files with the .sh extension.

File discovery should be guided by the incident, and findings should be validated before drawing conclusions.

7. User Management and System Services

Understanding account management and service behavior helps analysts investigate unexpected users, unauthorized changes, and suspicious service activity.

User management commands

useradd userdel usermod chsh

These commands are used to create users, remove users, modify user accounts, and change a user's login shell, respectively.

Service monitoring

systemctl status journalctl -u ssh

systemctl status — checks the status of a service.

journalctl -u ssh — displays journal entries for a service named ssh; the service name may differ across distributions.

These commands can help determine whether a service is running and investigate relevant service logs.

Login shell investigation

chsh

This command changes a user's login shell. Unexpected shell changes may deserve investigation, particularly if they coincide with suspicious account activity.

8. Scheduled Tasks and Persistence

Scheduled jobs are commonly used for backups, maintenance, and automation. However, an attacker with sufficient access may also abuse scheduled tasks to maintain execution.

Cron entries generally follow this structure:

minute hour day-of-month month day-of-week command

Example:

35 * * * * /home/user/backup.sh

This runs the backup script at minute 35 of every hour.

When investigating scheduled jobs, consider:

Is the task expected?

Who owns the script?

Does the script perform legitimate work?

Is the file writable by an unexpected account?

Does the schedule align with suspicious activity?

Scheduled execution alone is normal. The script, ownership, permissions, and execution history provide the necessary context.

9. Putting the Investigation Together

A useful Linux SOC investigation follows the evidence rather than relying on a single command.

Step 1: Review user activity. Check current sessions and recent login history.

Step 2: Inspect processes. Identify unusual process names, parent-child relationships, command-line arguments, and executable paths.

Step 3: Examine network connections. Review listening sockets, remote endpoints, and associated processes.

Step 4: Check permissions and persistence. Investigate relevant files, ownership, service configuration, and scheduled tasks.

Step 5: Correlate evidence. Compare timestamps, authentication records, process execution, network activity, and DNS information where available.

Step 6: Document the conclusion. Summarize what happened, the supporting evidence, what remains uncertain, and what should be investigated next.

Key Takeaways

Day 8 helped me understand how different Linux commands support a structured SOC investigation.

Permissions reveal ownership and access.

Authentication commands help establish user activity.

Process tools expose execution relationships and command-line details.

Network tools help connect processes to communication activity.

Linux filters simplify log analysis.

Service and scheduled-task inspection helps investigate persistence.

The biggest lesson is that individual commands provide observations, but correlating evidence creates an investigation.

I am continuing to strengthen my Linux, SOC, and cybersecurity foundations as part of my longer-term goal of progressing toward VAPT, penetration testing, and Red Team operations.

This article reflects my learning notes and practical study. The commands and observations described here are intended for authorized systems and defensive investigation.

What I'm learning next: How to connect Linux evidence with SIEM alerts and build a clearer incident timeline.

#Cybersecurity #SOCAnalyst #Linux #BlueTeam #IncidentResponse