Post cover image
Photo by Anthony Roberts on Unsplash

August 23, 2026

The Attack Wasn’t Hidden. We Were Looking at the Wrong Evidence.

How attackers leave small technical clues across logs, processes, DNS, authentication, and network traffic

By Fateyaly

12 min read

Read here.

At 2:41 AM, a Linux server started behaving strangely.

CPU usage increased.

Nothing crashed.

No alert fired.

The application was still responding.

The monitoring dashboard showed green.

From a distance, everything looked normal.

Then an administrator noticed something odd:

sshd
 └── bash
      └── python3

The Python process wasn't part of the application's normal architecture.

That single observation changed the investigation.

The question was no longer:

"Why is CPU usage high?"

It became:

"Why did this process exist in the first place?"

That question eventually led through:

Process
   ↓
Parent process
   ↓
User
   ↓
Authentication event
   ↓
Source IP
   ↓
Network connection
   ↓
DNS query
   ↓
File modification
   ↓
Persistence mechanism

Nothing had been hidden particularly well.

The evidence was sitting in different places.

The problem was that nobody had connected it.

And that's one of the most important lessons in incident response:

An attack rarely looks like an attack while you're looking at only one event.

1. The Problem With Security Alerts

Security teams often imagine an attack should produce something obvious:

CRITICAL ALERT
MALWARE DETECTED
REMOTE CODE EXECUTION
ATTACK IN PROGRESS

Real incidents are usually much less cinematic.

You might see:

Failed SSH login

Then:

Successful SSH login

Then:

New process created

Then:

Outbound connection

Then:

New file written

Then:

Scheduled task modified

Each event can have a perfectly legitimate explanation.

Someone mistyped a password.

An administrator logged in.

A script started.

A server contacted an external API.

An application created a file.

A deployment modified a scheduled task.

The problem is not necessarily the events.

It's their relationship in time and context.

2. Start With a Timeline, Not a Theory

One of the easiest mistakes during an incident is deciding what happened before collecting evidence.

For example:

"It's probably malware."

Now every observation gets interpreted through that assumption.

A better approach is:

Evidence
   ↓
Timeline
   ↓
Relationships
   ↓
Hypothesis
   ↓
Verification

Suppose we have:

02:14:03  Failed authentication
02:14:07  Successful authentication
02:14:15  Shell started
02:14:18  Python process created
02:14:22  Outbound connection
02:14:27  New executable written
02:15:03  Scheduled task modified

You don't need to immediately declare an intrusion.

But you absolutely have a reason to investigate.

The timeline gives you something much more valuable than an isolated alert:

sequence.

And sequence is often what turns noise into evidence.

3. Linux Already Knows More Than You Think

One of the reasons Linux is such a useful environment for security investigation is that the operating system exposes a tremendous amount of runtime information.

Processes.

Users.

Files.

Sockets.

Signals.

Environment.

Parent-child relationships.

Open descriptors.

Command lines.

Much of this becomes visible through /proc.

For example:

ps aux

can provide a high-level process snapshot.

For a more targeted view:

ps -ef

And if you find a suspicious PID:

ps -fp <PID>

you can begin asking:

Who owns it?
What command launched it?
When did it start?

But the PID itself isn't the interesting part.

The story behind the PID is.

4. Every Suspicious Process Has a Family Tree

Consider:

systemd
  └── sshd
       └── sshd
            └── bash
                 └── python3

That is much more informative than:

python3

Why?

Because process ancestry gives you context.

A Python process launched by your application might be completely normal:

systemd
   └── application
        └── python3

A Python process spawned by an interactive SSH shell on a production server at an unusual time deserves more scrutiny:

sshd
   └── bash
        └── python3

The binary didn't change.

The context did.

That's why process trees are so valuable.

5. Follow the Parent

If you have a suspicious PID:

ps -fp <PID>

Look at its parent PID.

You can also inspect:

cat /proc/<PID>/status

Look for information such as:

Name:
Pid:
PPid:
Uid:
Gid:

The PPid is particularly useful.

It tells you:

Which process created this process?

Now walk upward.

For example:

PID 8214  python3
PPID 8201  bash
PPID 8188  sshd
PPID 1     systemd

You have a chain.

And every step creates another question.

6. The Process Tree Is an Attack Timeline in Disguise

Imagine you find:

systemd
 └── nginx
      └── php-fpm
           └── sh
                └── curl

You don't immediately know whether it's malicious.

But the structure is interesting.

Ask:

Why did PHP spawn a shell?

Then:

Why did the shell launch curl?

Then:

Where did curl connect?

Then:

What initiated the original PHP request?

Now your investigation has moved from:

Suspicious process

to:

HTTP request
   ↓
Web application
   ↓
Process creation
   ↓
Shell
   ↓
Network connection

That is a much stronger investigative model.

7. Don't Kill the Process Too Quickly

This is one of the hardest habits for people new to incident response.

You see:

suspicious-process

and immediately:

kill <PID>

It feels productive.

But you may have just destroyed evidence.

Before containment, when operationally safe, capture useful information such as:

ps -fp <PID>
readlink -f /proc/<PID>/exe
readlink -f /proc/<PID>/cwd
tr '\0' '\n' < /proc/<PID>/cmdline

And inspect network activity:

ss -tpn

The exact response depends on the incident and environment, but the principle is universal:

Preserve context before destroying the thing you're investigating.

Containment is important.

Evidence is important too.

8. The Executable Can Tell You Where the Process Came From

Suppose:

readlink -f /proc/8214/exe

returns:

/tmp/.cache/python3

That doesn't prove malicious activity.

But it's interesting.

Compare that with:

/usr/bin/python3

The first path deserves considerably more investigation.

Now ask:

Who created that file?
When?
What permissions does it have?
What is its hash?
What process wrote it?
Is it referenced by any startup mechanism?

A suspicious executable is not the conclusion.

It's another node in the evidence graph.

9. /proc Turns a Process Into a Case File

Linux exposes process information through:

/proc/<PID>/

Useful locations include:

/proc/<PID>/exe
/proc/<PID>/cwd
/proc/<PID>/cmdline
/proc/<PID>/environ
/proc/<PID>/fd/
/proc/<PID>/status

Each answers a different question.

/proc/<PID>/exe

What executable is actually running?

/proc/<PID>/cwd

What is its working directory?

/proc/<PID>/cmdline

What arguments were provided?

/proc/<PID>/environ

What environment variables does the process have?

/proc/<PID>/fd/

What files, sockets, pipes, or other descriptors does it currently have open?

/proc/<PID>/status

What user and process metadata are associated with it?

This is why /proc is such a powerful investigative resource.

You're not simply looking at:

"python3 is running."

You're reconstructing:

What?
Where?
Who?
How?
With what?
Connected to what?

10. Then Follow the Network

Suppose the suspicious process is PID 8214.

You want to know:

What is it communicating with?

One useful tool is:

ss -tpn

You may see something conceptually like:

ESTAB
10.0.0.12:42318
203.0.113.25:443
users:(("python3",pid=8214,fd=4))

Now you have another relationship:

PID 8214
   ↓
python3
   ↓
203.0.113.25:443

Again, don't jump immediately to:

"That's the attacker."

The destination could be:

  • a legitimate API
  • a cloud service
  • a monitoring system
  • a software repository
  • an organization's own infrastructure

The important thing is that you now have something concrete to investigate.

11. IP Addresses Are Clues, Not Verdicts

A common mistake is treating an unfamiliar IP address as proof of compromise.

It isn't.

An IP tells you:

Where the connection goes.

It doesn't tell you:

Why the connection exists.

You need context.

Ask:

Which process opened it?
When did it start?
What hostname resolves to it?
Has this server contacted it before?
Is the destination expected?
What protocol is being used?

The strongest evidence comes from combining observations.

12. DNS Can Add the Missing Piece

Suppose the process connects to:

203.0.113.25

A DNS investigation might show:

api.example-service.com

Suddenly the connection looks legitimate.

Or perhaps the infrastructure reveals a domain your organization has never used.

Now you have another lead.

DNS is especially useful because attackers and legitimate software both rely on it.

You may investigate:

dig example.com

or use your organization's DNS telemetry to determine:

Who queried the domain?
When?
From which host?
How often?

The key isn't:

"DNS query = attack."

It's:

"Does this DNS activity make sense in the context of the process that generated it?"

13. Logs Give You the Missing Timestamp

Now suppose the suspicious process started at:

02:14:18

Search authentication logs around that time.

On Debian/Ubuntu systems, authentication events may commonly appear in:

/var/log/auth.log

On systems using systemd's journal:

journalctl --since "02:00" --until "02:30"

You might discover:

02:14:03 Failed authentication
02:14:07 Successful authentication
02:14:15 Session opened
02:14:18 python3 started

Now the process isn't just:

python3

It has a history.

That history is evidence.

14. Correlation Beats Isolation

Consider these events separately:

Failed login

Not unusual.

Successful login

Not unusual.

Python process

Not unusual.

Outbound HTTPS

Not unusual.

New file

Not unusual.

Now combine them:

Failed login
      ↓ 4 seconds
Successful login
      ↓ 8 seconds
Shell created
      ↓ 3 seconds
Python process
      ↓ 4 seconds
Outbound connection
      ↓ 5 seconds
Executable written

That sequence is much more interesting.

This is the essence of incident correlation:

Events become meaningful when you understand their relationships in time, identity, and context.

15. Identity Is the Glue Between Events

Suppose your logs show:

User: deploy

and later:

Process: python3

Can you prove the process belonged to deploy?

Maybe.

But you should verify.

Look at:

ps -eo pid,ppid,user,group,cmd

This can provide a process-oriented view including:

PID
PPID
USER
GROUP
COMMAND

Now you can connect:

Authentication event
      ↓
User
      ↓
Process owner
      ↓
Network activity

That is much stronger than merely saying:

"Something happened around the same time."

16. File Timestamps Can Complete the Story

Suppose you discover:

/tmp/.cache/python3

Check its metadata:

stat /tmp/.cache/python3

You may see timestamps for:

Access
Modify
Change

These timestamps are useful, but don't treat them as perfect forensic truth.

Modern systems, copy operations, extraction tools, timestamp manipulation, and filesystem behavior can complicate interpretation.

The correct approach is correlation.

For example:

02:14:07 Authentication
02:14:18 Process starts
02:14:22 Network connection
02:14:25 File modified

Now the timestamps reinforce one another.

17. Look for Persistence

Suppose the suspicious process disappears after reboot.

That doesn't mean the investigation is over.

Ask:

How could it come back?

Linux persistence can involve many legitimate mechanisms, including:

systemd services
cron jobs
shell startup files
SSH configuration
application startup mechanisms
user-level services

A defender should understand where persistence could exist.

For example:

systemctl list-unit-files

or:

systemctl list-timers

can help you understand service and timer configuration.

For scheduled tasks, examine the appropriate cron locations and user crontabs according to the system's configuration.

The question remains:

What starts this?
Who owns it?
Who can modify it?
When does it execute?

Again:

Trust relationships.

18. Persistence Is Often Boring

This is something defenders should appreciate.

Persistence doesn't have to look like:

evil-malware.bin

It may look like:

backup-update.service

or:

system maintenance script

or:

scheduled task

Names are weak evidence.

Behavior is stronger.

A legitimate-looking service that launches an unexpected binary from an unusual path deserves investigation.

The question isn't:

"Does the name look malicious?"

It's:

"Is the behavior consistent with what this system is supposed to do?"

19. Baselines Are More Valuable Than They Sound

Suppose a server normally communicates with:

Database
Monitoring
Package repositories
Internal APIs

Then suddenly:

Server → New external destination

That doesn't prove compromise.

But it violates the baseline.

Security monitoring becomes much stronger when you understand normal behavior.

For a server, baseline:

Normal processes
Normal users
Normal network destinations
Normal ports
Normal scheduled jobs
Normal DNS queries
Normal file changes

Then detect deviations.

This is one reason anomaly detection is useful.

But beware:

Anomaly does not mean malicious.

It means:

Investigate this because it doesn't fit the expected model.

20. The Most Interesting Process May Be the Parent

Suppose you find:

python3

You investigate it.

But perhaps the more important process is its parent:

python3
   ↑
bash
   ↑
php-fpm

Or:

curl
   ↑
shell
   ↑
application

The parent-child relationship can reveal how execution occurred.

That means process analysis should move both directions:

Child
 ↑
Parent
 ↑
Parent

and:

Parent
 ↓
Children

The first tells you:

Where did this process come from?

The second tells you:

What did this process create?

21. Don't Forget Open Files

A process can tell you what it is.

Its file descriptors can tell you what it is interacting with.

For example:

ls -l /proc/<PID>/fd/

You may find references to:

Files
Sockets
Pipes
Devices

This can help connect a process to resources.

Imagine:

Suspicious process
   ↓
Open socket
   ↓
External IP

or:

Suspicious process
   ↓
Open file
   ↓
Sensitive configuration

Now the process becomes part of a much larger evidence graph.

22. Sometimes the Network Tells You What the Process Won't

Suppose a suspicious process has no obvious command-line arguments.

That doesn't mean it's harmless.

Its network behavior may reveal:

DNS lookup
TCP connection
TLS session
Repeated beacon-like timing
Unexpected destination

Even encrypted traffic can provide metadata such as:

Destination
Port
Timing
Connection frequency
Volume

Encryption protects content.

It doesn't necessarily hide every behavioral signal.

This is why network telemetry and host telemetry complement each other.

23. One Source of Evidence Is Rarely Enough

Good incident investigation resembles a puzzle.

You want independent evidence sources to support the same hypothesis.

For example:

Host evidence

Unexpected process

Identity evidence

Unexpected account activity

Network evidence

Unexpected outbound connection

File evidence

Unexpected executable created

Timeline evidence

All events occur within the same window

Now you have convergence.

That's much stronger than one suspicious log line.

24. The Evidence Graph

At this point, you can model the incident like this:

Source IP
    │
    ↓
Authentication
    │
    ↓
User
    │
    ↓
Shell
    │
    ↓
Process
    │
    ├────────→ File
    │
    ├────────→ Network Socket
    │
    └────────→ Child Process
                    │
                    ↓
              Persistence

This is essentially the defensive version of the attack-chain model from the previous article.

Attackers build paths.

Defenders reconstruct them.

25. The Question That Changes Investigations

Most people ask:

"What happened?"

That's useful.

But during an investigation, a better sequence is:

Who?
 ↓
When?
 ↓
What?
 ↓
How?
 ↓
Where?
 ↓
What next?

For example:

Who?

deploy

When?

02:14:07

What?

Interactive SSH session

How?

Shell → Python process

Where?

203.0.113.25

What next?

File creation → scheduled task modification

Now you've transformed raw telemetry into a narrative.

26. What You Should Never Assume

Incident response is full of dangerous assumptions.

Don't assume:

Unknown IP = attacker

Don't assume:

Root process = malicious

Don't assume:

Unusual filename = malware

Don't assume:

Successful login = compromise

Don't assume:

High CPU = cryptomining

Don't assume:

New outbound connection = data exfiltration

Every one of those observations is a lead.

Security investigation becomes reliable when you distinguish:

Observation

from:

Interpretation

and:

Conclusion

Don't collapse those three into one.

27. A Practical Investigation Workflow

When you encounter a suspicious Linux process, a useful defensive workflow is:

1. Identify the process
        ↓
2. Identify its owner
        ↓
3. Identify its parent
        ↓
4. Identify its executable
        ↓
5. Identify its working directory
        ↓
6. Inspect command-line arguments
        ↓
7. Inspect network connections
        ↓
8. Inspect open files
        ↓
9. Correlate timestamps
        ↓
10. Review authentication activity
        ↓
11. Check persistence mechanisms
        ↓
12. Determine blast radius

The order may change depending on the incident.

The principle doesn't.

Move from the process outward.

28. Determine the Blast Radius

Finding the process is only half the job.

You also need to know:

What could this process access?

Check:

User identity
Group membership
Filesystem permissions
Network reachability
Environment variables
Credentials
Cloud identity
Mounted resources

For example:

id <username>

can help understand group membership.

Then ask:

What files can this identity read?
What services can it access?
What credentials can it use?
What network segments can it reach?

That determines impact.

A compromised process with access only to temporary files is very different from one with access to:

/etc
application secrets
database credentials
cloud credentials
SSH keys
production APIs

29. Detection Isn't About Collecting Everything

More logs don't automatically mean better security.

You could collect:

10 TB/day

and still miss the attack.

The goal is useful telemetry.

You want enough information to answer:

Who?
What?
When?
Where?
How?
What changed?
What happened next?

For Linux environments, useful sources can include:

Authentication logs
systemd journal
process telemetry
DNS telemetry
network flow data
file integrity monitoring
application logs
sudo activity
cloud audit logs

The right combination depends on the environment.

But the principle is universal:

Collect evidence that lets you reconstruct relationships.

30. The Defender's Version of Reconnaissance

Attackers perform reconnaissance before exploitation.

Defenders should perform reconnaissance during investigation.

You are mapping:

Host
 ↓
Users
 ↓
Processes
 ↓
Files
 ↓
Sockets
 ↓
Services
 ↓
External destinations

The difference is intent.

The attacker asks:

"What can I reach?"

The defender asks:

"What did the attacker reach?"

Same technical knowledge.

Opposite purpose.

31. The Incident Was Never One Event

This is the biggest lesson.

An intrusion rarely looks like:

ATTACK

It looks like:

Event A
   ↓
Event B
   ↓
Event C
   ↓
Event D
   ↓
Event E

Each event may be ordinary.

But the sequence isn't.

That's why experienced analysts don't necessarily look for the loudest signal.

They look for relationships.

32. The Security Analyst Is a Storyteller

Not in the creative-writing sense.

In the forensic sense.

At the end of an investigation, you should be able to explain:

At 02:14:07,
account X authenticated from source Y.
At 02:14:15,
a shell session was created.
At 02:14:18,
process Z started.
At 02:14:22,
process Z established an outbound connection.
At 02:14:25,
file A was created.
At 02:15:03,
persistence mechanism B was modified.

That's a story.

But unlike a guess, every important part should be backed by evidence.

The goal isn't simply:

"We think the server was compromised."

The goal is:

"Here is what happened, in what order, under which identity, using which process, affecting which resources, and here's the evidence supporting each step."

That's incident response.

33. What I Would Check First on a Strange Linux Host

If I walked into an investigation and someone said:

"Something weird is happening on this server."

I wouldn't immediately start throwing scanners at it.

I'd establish context.

First:

uptime

Then:

ps -eo pid,ppid,user,lstart,cmd --sort=lstart

Then inspect suspicious processes individually:

ps -fp <PID>

Follow the executable:

readlink -f /proc/<PID>/exe

Check its working directory:

readlink -f /proc/<PID>/cwd

Inspect network state:

ss -tpn

Then correlate with:

journalctl

and the relevant authentication/application logs.

The exact commands aren't the important part.

The reasoning is:

Process
 ↓
Identity
 ↓
Parent
 ↓
File
 ↓
Network
 ↓
Timeline

34. The Most Dangerous Evidence Is Sometimes the Most Ordinary

A suspicious executable is useful.

But consider this sequence:

Successful login

Then:

New shell

Then:

Unexpected command

Then:

Outbound connection

Then:

Persistence

There may never be a single event saying:

"This is malware."

The operating system simply recorded what happened.

The evidence was there.

It just wasn't connected.

35. The Mental Model I Want You to Keep

When investigating any security event, imagine a chain:

IDENTITY
    ↓
PROCESS
    ↓
FILE
    ↓
NETWORK
    ↓
PERSISTENCE
    ↓
IMPACT

Then fill in the blanks.

For example:

deploy
   ↓
bash
   ↓
/tmp/.cache/python3
   ↓
203.0.113.25:443
   ↓
systemd service
   ↓
production application

Now you have something actionable.

You can determine:

  • initial access
  • execution
  • persistence
  • command and control
  • affected assets
  • containment requirements
  • recovery priorities

That's much more useful than:

"There was a suspicious process."

36. The Final Lesson: Don't Hunt for the Smoking Gun

Cybersecurity culture loves the idea of a smoking gun.

One malicious file.

One suspicious IP.

One critical alert.

One obvious command.

Real investigations are often messier.

The evidence is distributed.

One clue lives in authentication logs.

Another lives in the process table.

Another lives in /proc.

Another lives in DNS.

Another lives in network telemetry.

Another lives in filesystem metadata.

Another lives in a scheduled task.

None tells the complete story.

Together, they can.

That's why the best investigators develop a habit that has nothing to do with a particular security tool:

They connect evidence.

They don't ask:

"Is this suspicious?"

They ask:

"What does this connect to?"

Then:

"Who caused that?"

Then:

"What did that process access?"

Then:

"Where did it communicate?"

Then

"What changed afterward?"

And finally:

"What is the complete sequence of events?"

Because the attack wasn't necessarily hidden.

The server may have told you exactly what happened.

The logs may have told you.

The process tree may have told you.

The network may have told you.

The filesystem may have told you.

The problem was that everyone was looking at their own piece of the evidence.

Incident response begins when those pieces become one story.