Malware Analysis & Reverse Engineering

Introduction

Every offensive engagement eventually collides with the defensive question: what does this binary actually do? Whether it's a dropped payload from a red team exercise, a suspicious executable flagged by an EDR, or a sample pulled during incident response, malware analysis and reverse engineering sit at the intersection of red and blue work. This installment covers the discipline of taking apart malicious code — safely, methodically, and with a clear goal: understanding behavior, persistence mechanisms, and indicators of compromise (IOCs).

What It Is

Malware analysis is the process of studying a malicious sample to determine:

• What it does — its functional behavior (keylogging, credential theft, C2 communication, ransomware encryption, etc.)
• How it persists — registry keys, scheduled tasks, services, startup folders, DLL hijacking
• How it communicates — C2 protocols, domains, IPs, beaconing intervals
• How it can be detected — file hashes, strings, network signatures, behavioral patterns

Reverse engineering goes a layer deeper — disassembling and decompiling the binary to understand its internal logic at the assembly or pseudo-code level, rather than just observing external behavior.

Why It Matters

For red teams: Understanding how existing malware achieves persistence, evasion, and C2 communication directly informs custom payload development and helps testers anticipate what defensive tooling will catch.

For blue teams: Analysis output — hashes, YARA rules, network indicators, behavioral signatures — becomes the raw material for detection engineering, SIEM correlation rules, and threat hunting playbooks.

This dual relevance is exactly why malware analysis belongs in a VAPT lifecycle discussion: it's one of the few disciplines that produces artifacts consumed by both sides of the table.

Subtopic 1: Static vs Dynamic Analysis

A mature analysis workflow uses both — static analysis to triage and form hypotheses, dynamic analysis to confirm behavior.

Subtopic 2: PE Structure

Windows executables follow the Portable Executable (PE) format. Understanding its layout is foundational to static analysis:

• DOS Header — legacy header, contains the pointer to the PE header
• PE Header — machine type, timestamp, characteristics
• Optional Header — entry point address, image base, subsystem
• Section Table — defines .text (code), .data (initialized data), .rdata (read-only data), .rsrc (resources)
• Import Table (IAT) — DLLs and functions the binary depends on — often the fastest clue to capability (e.g., WININET.dll suggests network activity, ADVAPI32.dll suggests registry/service manipulation)

Tools like PEStudio and CFF Explorer parse this structure quickly and flag anomalies (unusual section entropy, suspicious imports, missing digital signatures).

Subtopic 3: IDA Free for Disassembly

IDA Free is the entry point into serious reverse engineering — a disassembler that converts machine code back into human-readable assembly, with cross-referencing, function graphing, and basic decompilation support.

Typical workflow:

1. Load the binary and let IDA build the initial disassembly and function list
2. Identify the entry point and trace execution flow
3. Use the Imports window to jump to interesting API calls (CreateRemoteThread, VirtualAllocEx, RegSetValueEx) — common markers of injection or persistence
4. Rename functions and variables as understanding builds, to keep the analysis readable
5. Use the graph view to map decision branches and identify anti-debugging or anti-VM checks

IDA Free is a static analysis tool — pairing it with a debugger (x64dbg, or IDA's own debugger in the paid version) enables live, instruction-by-instruction dynamic analysis.

Subtopic 4: YARA Rules

YARA is a pattern-matching tool built specifically for identifying and classifying malware based on textual or binary patterns. A YARA rule is essentially a signature that can be deployed across a fleet of endpoints, a SIEM, or a malware repository.

Basic structure of a rule:

rule Suspicious_Persistence_Marker
{
    meta:
        description = "Flags binaries referencing common persistence registry paths"
        author = "VAPT Lab"
strings:
        $reg1 = "CurrentVersion\\Run" wide ascii
        $api1 = "RegSetValueExA"
    condition:
        uint16(0) == 0x5A4D and all of them
}

rule Suspicious_Persistence_Marker
{
    meta:
        description = "Flags binaries referencing common persistence registry paths"
        author = "VAPT Lab"
strings:
        $reg1 = "CurrentVersion\\Run" wide ascii
        $api1 = "RegSetValueExA"
    condition:
        uint16(0) == 0x5A4D and all of them
}

This rule checks that the file is a valid PE (MZ header) and contains both a Run-key registry reference and the API call used to write it — a lightweight but effective heuristic for persistence-focused malware. YARA rules generated during analysis feed directly into detection tooling — including custom Wazuh decoders, discussed below.

Subtopic 5: Sandbox Analysis

Sandboxes provide an isolated, instrumented environment to detonate a sample safely and capture its behavior automatically. Key capabilities:

• File system and registry monitoring — what gets created, modified, deleted
• Network capture — C2 domains, IPs, protocols, beacon patterns
• Process tree visualization — parent/child spawning, injection targets
• API call logging — sequence of Windows API calls made at runtime

Isolation is non-negotiable: sandbox VMs should be network-segmented (or routed through a controlled fake-internet setup), snapshot-based for quick resets, and never share credentials or infrastructure with production/lab AD environments. Cross-contamination between a malware sandbox and an Active Directory lab defeats the entire purpose of isolation.

MITRE ATT&CK Mapping

Blue Team Detection: Wazuh Integration

Analysis output should never stay static — it should feed detection. A practical pipeline from this discipline into a Wazuh-monitored environment:

1. File Integrity Monitoring (FIM): Configure Wazuh's syscheck module to watch common persistence paths (Run keys, Startup folder, Tasks directory) so any write matching analysis findings triggers an alert.
2. YARA integration: Wazuh supports active-response YARA scanning — rules developed during analysis (like the example above) can be dropped into the Wazuh YARA ruleset and triggered on file creation events.
3. Custom decoders/rules: Build Wazuh rules around specific API call patterns or registry writes surfaced during dynamic analysis, mapped to the MITRE ATT&CK IDs above for SOC triage clarity.
4. Network indicators: IOCs (C2 domains/IPs) extracted from sandbox runs can be pushed into Wazuh via threat intel integration or custom log analysis rules for outbound connection alerting.

This closes the loop: a sample analyzed on the offensive/research side becomes a concrete, testable detection on the defensive side — the same philosophy that's run through this entire series.

Key Takeaways

• Static and dynamic analysis are complementary, not competing — use both.
• PE structure and import tables often reveal capability before a single instruction is executed.
• IDA Free is the standard entry point for serious disassembly work.
• YARA rules turn one-off analysis into repeatable, deployable detection.
• Sandbox isolation is a hard requirement, not a best practice — never blend malware analysis infrastructure with lab AD environments.
• Every analysis finding has a natural home in a SIEM like Wazuh — this is where red-side research becomes blue-side detection.

This post is part of an ongoing VAPT learning series documenting hands-on offensive and defensive security work in a lab environment. Previous parts covered web application testing, social engineering, post-exploitation, wireless attacks, SIEM, Detection & Blue team concepts.