Post cover image

August 6, 2026

Reverse Engineering 101: Principles, Tools, and Methodologies

Welcome, traveler! From basic hardware logic to practical reverse engineering, this article serves as a complete beginner’s guide. We’ll…

By Sw33tBit

16 min read

Welcome, traveler! From basic hardware logic to practical reverse engineering, this article serves as a complete beginner's guide. We'll place a special focus on static analysis using Ghidra, walking through essential analysis techniques like code coverage, signature matching, and symbolic execution.

Before diving deep into actual reverse engineering, it is recommended to understand the fundamentals of how processes are loaded into memory and their overall layout. It is also definetely worth to be familiar with binary formats, specifically the different sections where code and data reside.

This foundational knowledge helps you understand how malicious actors can hide code that executes before the main function is even called (such as through TLS Callbacks in Windows), and clarifies why certain structures behave the way they do.

How Does a Computer Work? (in short)

Before we can deconstruct a compiled program, it is worth understanding the machine that runs it. While modern software feels incredibly complex, it is just a collection of simple building blocks arranged into a larger system. By tracing the path from raw electronic signals up to structured logic, we can see exactly how abstract code is transformed into physical execution.

The Rule of Context: Binary Has No Inherent Meaning

At its lowest level, a computer operates entirely on ones and zeros. But why exactly ones and zeros? This binary system exists because computers are built on Boolean algebra, which is physically represented by electrical states.

None

However, it is crucial for a reverse engineer to remember that raw binary has no inherent meaning — context is everything.

The exact same sequence of ones and zeros might represent an executable instruction (an opcode) on one CPU architecture, but perform a completely different action — or just be interpreted as raw data — on another.

The Engine of Everything: Universal Logic

At the physical hardware level, these binary states are controlled by transistors acting as microscopic switches. By wiring these transistors together, we create logic gates, and the most used among them is the NAND gate. Because NAND is logically universal, you can deduce and create every other logical operation using only NAND gates. It is the basic block of all computation.

None

The Ladder of Abstraction

The entire architecture of a computer is built upon the concept of abstraction. The system is separated into distinct layers, where each layer acts as a simplified interface hiding the immense complexity below it.

We take incredibly small, simple components and stack them to create highly complex machinery.

For example by stringing together those universal NAND gates, we can build Full Adders and Arithmetic Logic Units (ALUs) and we have the math core of a CPU.

And how about memory? At its core, hardware memory is just a logical feedback loop. The most basic memory cell, known as an R-S flip-flop, uses this continuous loop to sustain an electrical state, allowing the machine to effectively "remember" a bit of data.

None

Some kind of layers of abstration is OSI table that is used for isolating problems to a specific layer.

With a CPU to compute and memory to store state, we have the physical foundation required to execute programs.

Process Memory Layout

Once a program is executed, the operating system creates a new process and allocates it its own isolated virtual address space. This space is logically split into distinct memory sections:

None

How a Process is Born

If we dive into the granular details of execution — let's say we run a program from the terminal using ./program

The operating system (in Unix/Linux environments) begins by creating a new process via the fork() syscall. This relies on a resource-management technique called Copy-on-Write (CoW).

pid_t fork(void);

Once the new process is created, it invokes the execve() syscall to replace the current process memory with the new program.

int execve(const char *pathname, char *const argv[], char *const envp[]);

Behind the scenes, execve() performs a critical sequence of operations to properly load your binary into memory:

  1. Validation (if it is valid binary)
  2. Clearing the Address Space (inherited from the fork() )
  3. Parsing the Binary Header (of .text , .data , .bss )
  4. Memory Mapping (mmap) — creates our address space
  5. Setting up the Stack (pushes argc, argv, envp )

Common Binary Formats

As mentioned during the execve() breakdown, the operating system must parse the binary header to understand how to load the file. Depending on the operating system you are analyzing, you will primarily encounter two binary formats.

Portable Executable (PE)

The standard format for Windows executables

Extension: (.exe, .dll, .sys, .scr) Signature: 4D 5A (MZ)or 50 45 00 00 (PE\0\0)

Executable and Linkable Format (ELF)

The standard format for Unix and Linux systems

Extension: (None, .so, .ko) Signature: 7F 45 4C 46 (.ELF)

Function Execution

Once the binary is loaded into memory and execution begins, the CPU relies on small, ultra-fast storage locations called registers to process data and keep track of state. While modern CPUs have many registers, a few are critical to know when reverse engineering:

ebp/rbp — base pointer
edi/rdi — index pointer
eax/rax — general usage pointer
eip/rip/pc — instruction pointer

If we want a program to execute smoothly, we must establish strict rules to ensure that jumping between functions doesn't accidentally corrupt data or overwrite registers.

To solve this, compilers enforce a Calling Convention. This is an agreed-upon standard that dictates exactly how a function should be called, such as whether arguments are passed through specific CPU registers, pushed onto the stack, or a mix of both.

When a function is called, it creates its own isolated memory workspace known as a Stack Frame, which is pushed onto the Call Stack.

None

To set up and tear down this frame without destroying the calling function's data, compilers inject standard sequences of assembly at the start and end of a function, known as the Prologue and Epilogue.

Here is what standard prologue and epilogue looks like:

; x86-64 prologue/epilogue
push rbp        ; 1. Save the Base Pointer of the caller
mov  rbp, rsp   ; 2. Set the current Stack Pointer as the new Base Pointer
sub  rsp, 0x20  ; 3. Allocate space on the stack for local variables (e.g., 32 bytes)
; <function instructions>
mov rsp, rbp    ; 1. Destroy the local variables (restore the stack pointer)
pop rbp         ; 2. Restore the caller's Base Pointer
ret             ; 3. Return to the caller

The Heap (Dynamic Memory Allocation)

Unlike the stack, which generally requires the size of its variables to be known at compile time, the heap allows memory to be allocated dynamically during execution.

The heap is a large pool of memory dedicated to this runtime allocation. When a program needs to store data whose size is unpredictable until the program is actually running, it uses the heap.

  • Manual Lifespan (developer controls memory)
  • Unstructured Layout (uses linked lists called bins)
  • Stack frames are temporary
None

Methods used in RE

When analyzing a binary, reverse engineers generally divide their workflow into two main categories: static and dynamic analysis, often supplemented by advanced automated techniques.

None

Static analysis

Static analysis involves examining the program's code without actually executing it. This includes disassembling the binary, analyzing its control flow, and reading the assembly or decompiled pseudocode. It is the safest way to analyze malware, as the malicious code is never actively run on your machine.

We will dive into this method a little bit later.

Dynamic analysis

Dynamic analysis is the process of observing a running binary. This allows you to get a feel for how the program acts in a controlled environment.

There are a few different approaches you can take:

Debugging — attaching to the process and seeing what it does step by step (pause execution at critical moments to inspect CPU registers and memory)

  • Tools: GDB (Linux), winedbg (PE debugger for Linux), x64dbg (Windows)

API and system monitoring — we can attach library/program that hooks to certain calls (Side note: binaries often try to detect or overwrite these calls — a known method for bypassing tools like ptrace)

  • Tools (Linux): strace (system), ltrace (library)
  • Tools (Windows): Sysinternals Process Monitor, API Monitor

Sandboxing — executing binary in monitored and isolated virtual machine and recording everything (network traffic, registry, filesystem — read/write calls …)

  • Tools: Any.run, Triage, Cuckoo Sandbox

Others / Automated analysis

Symbolic Execution

  • Instead of running a program with normal inputs (like a specific string or number), symbolic execution replaces the input with "symbols."
  • The execution engine then calculates the mathematical boundaries required to reach specific paths in the code.

For example, if we know an input must be a printable ASCII character, we can constrain our solver like this:

state.solver.add(byte >= 32)
state.solver.add(byte <= 126)

(We will dive much deeper into the details of this in the #Symbolic Execution section later).

Taint Analysis

  • This technique focuses on tracking a specific piece of unverified data (called the "taint" — most commonly user input) as it flows through the program.
  • The goal is to see if this tainted data reaches a vulnerable function (often called a "sink" or "pit") without first passing through a "sanitizer" function that cleans the data.

If you are interested in this type of analysis, I recommend you to read through additional resources.

Dynamic Binary Instrumentation (DBI)

  • DBI frameworks allow you to inject custom code into a running process on the fly.
  • This provides incredible power to intercept functions, modify arguments, and interact directly with the live memory of the target application without needing to modify the executable file itself.

Tools: Frida, Intel Pin, Qiling

Patching (modifying the binary)

  • Sometimes, you need to permanently change how the binary acts. Patching involves modifying the raw assembly instructions in the compiled file.
  • For example, if you find a core branching instruction, you can reverse the flow (changing a JZ (Jump if Zero) to a JNZ (Jump if Not Zero))
  • You can completely remove unwanted operations by overwriting them with NOP (No Operation) instructions.

Tools: Ghidra, IDA, Cutter and radare2

Ghidra

Ghidra is a free, open-source Software Reverse Engineering (SRE) framework created by the NSA. Written in Java, it is a powerful, completely free alternative to IDA Pro.

Its core features include a disassembler, a decompiler, and a robust scripting engine.

Weirdness of Ghidra (Special shortcuts)

Ghidra uses its own pseudo-macros to shorten output and make decompiled C code cleaner. However, this can sometimes lead to confusion.

For example:

  • CONCAT22(a,b) <=> (a << 16) | b → 32-bit
  • x.4_4 <=> bytes 4–7 of x

So from where does this syntax come from?

When you load a binary into Ghidra, it doesn't translate assembly directly into C code — doing so would require writing a completely different decompiler for every CPU architecture.

Instead, Ghidra uses a language called SLEIGH to translate every CPU's assembly into a universal, architecture-agnostic language called P-Code. From this P-Code, Ghidra constructs the C code.

In short can be described as: Input (Bytes) => SLEIGH (P-Code) => Optimizer (dead variables / branches) => Type Engine => Output (C-Code)

If you ever want to copy-paste Ghidra's decompiled C code into a compiler (like GCC or Clang), you will need to define these macros manually, like:

#define CONCAT44(a, b) (((unsigned long long)(a) << 32) | (unsigned int)(b))
#define SUB84(a, offset) ((unsigned int)((unsigned long long)(a) >> (offset * 8)))
#define ZEXT48(a) ((unsigned long long)(unsigned int)(a))

Deducing Types and Sizes

Even though Ghidra's auto-generated variable names look random, they contain structural hints:

  • uVar1, uVar2: Unsigned Variable.
  • iVar1, iVar2: Signed Integer Variable.
  • puVar1, pbVar1: Pointer to Unsigned Variable, Pointer to Byte.
  • cVar1, pcVar1: Char variable, Pointer to Char (often a string).
  • DAT_00402000: Raw Data located at address 0x00402000 (likely a global variable).
  • local_10, local_14: Stack variables (the numbers usually indicate the offset on the stack).
  • in_FS_OFFSET / in_GS_OFFSET: Refers to the FS or GS segment registers (fsbase usually used for canary).
None None None

Based on the knowledge, we can now just find canary bits using grep (0x28 offset is used for x64)

When reading decompiled code, look for library calls like malloc, fgets, scanf, or read.

char *fgets(char *str, int n, FILE *stream);

fgets(input, 0x80, stdin);

Or for loops:

for (i = 0; i < 10; i++) {
    *(int *)(local_28 + i * 4) = 0;
}

From this example, we can see that it goes through local_28 like it's array of ints of size 10.

Custom struct (offsets + size)

You might also spot memory offset patterns indicating a custom struct:

*(int *)(param_1 + 0x0) = 5;
*(char **)(param_1 + 0x4) = "Hello";
*(short *)(param_1 + 0x8) = 10;

When you encounter logic like this, you can deduce it's a structure:

/*
example, it don't have to reflect this exact structure
but still can help you later on
*/
struct MyString {
  int size;
  char *string;
  short m_size;
};

In Ghidra, you can create a custom type for it via the Data Type Manager (Right-click your binary name -> New -> Structure)

Then retype the variable using CTRL+L or by right-click menu.

Equetes/Enums

An interesting feature of Ghidra is the Equate. It is simply a text label applied to a specific, raw number — much like a #define macro in C.

It is perfect for one-off constants (e.g., changing 0x80000000 to GENERIC_READ).

Unlike an Equate, an Enum is an actual Type that you apply to a variable. Once typed, Ghidra will automatically translate those numbers everywhere that variable is used (e.g., Error codes, or States like Running, Stopped, Paused).

Cross-Referencing

When you know what specific string you are looking for (like an error message or a password prompt), you can search for it and use cross-referencing (XREF) to track down the exact function utilizing that string.

None

Suppose we would know that we now the correct result should include words "Excellent work", we can search in memory and find this string (using S shortcut or menu Search)

None

and from there we know exactly where we need to end up, without going through the whole binary and through all the nested functions.

Code Coverage

Code coverage is the process of tracking exactly which lines of assembly code (specifically, which Basic Blocks) were executed when a program ran. You can collect this data using drrun (a tool from the DynamoRIO DBI framework) and the drcov module.

DynamoRIO acts like a virtual machine running your target binary, allowing you to monitor memory and execution.

  • drrun -t drcov -- ./binary [arguments]

This produces a log that can be loaded directly into decompiler (using a plugin like Lighthouse for IDA or Cartographer for Ghidra).

This is incredibly useful for revealing crucial branches, such as VM detection evasion in malware, or determining which functions are actually reachable during runtime.

None

Symbolic execution

As discussed earlier, symbolic execution is powerful because we treat user input as algebraic symbols rather than concrete values. We can apply constraints to these symbols, pruning the tree of possibilities and solving mathematical equations instead of brute-forcing logic.

Note: You must understand the binary's structure for this to work. If a program sends data to a remote server for validation, symbolic execution won't help you. But if the checking algorithm is local, we are green.

Angr is a powerful Python framework for binary analysis, utilizing a solver called Claripy (built on top of Z3). However, it suffers from a major hurdle:

Path explosion

When Angr evaluates a branch (like an if/else statement) that depends on a symbolic variable, it doesn't know which path to take.

Consequently, it duplicates the current program state to explore both. If we have 10 consecutive if statements, Angr generates 2¹⁰ (1024) states.

Our RAM will fill up rapidly, the Z3 solver will choke on massive equations, and your script will hang forever.

To reduce or eliminate path explosion, we can:

  • Enable Veritesting (tries to merge states back together after a branch)
  • Restrict the Search Space (find and avoid)
  • Hook Complex Functions (skip/replace printf, strlen, or malloc)
  • Limit Loops (if symbolic variable depends on loop)
  • Add Constraints Early (example: limit yourself on ASCII)
  • Length Limiting (limit the total number of basic blocks evaluated)
  • Unicorn Engine (Combine fast concrete execution with symbolic execution)

In this example we are hooking complex functions (replacing them) and adding constraints early on:

import angr
import claripy

# auto_load_libs = False = use SimProcedures, do not load libc...
proj = angr.Project('./e4sy_RE.bdsec', auto_load_libs=False)

# Dummy procedures to skip irrelevant functions
class ReturnZero(angr.SimProcedure):
    def run(self, *args):
        # BVV = known value (do not resolve)
        return claripy.BVV(0, 64)

class SilentPrint(angr.SimProcedure):
    def run(self, *args, **kwargs):
        return claripy.BVV(1, 32)

# Force strcspn to instantly return 41
# - bypassing the entire length check
class FakeStrcspn(angr.SimProcedure):
    def run(self, s, reject):
        return claripy.BVV(41, 64)

for sym, hook in [
    ("rand",   ReturnZero()),
    ("time",   ReturnZero()),
    ("clock",  ReturnZero()),
    ("puts",   SilentPrint()),
    ("printf", SilentPrint()),
    ("strcspn", FakeStrcspn()),
]:
    proj.hook_symbol(sym, hook, replace=True)

# BVS = unknown value (symbolic value)
# - 41 symbolic bytes
flag_chars = [claripy.BVS(f'c_{i}', 8) for i in range(41)]
flag = claripy.Concat(*flag_chars)

state = proj.factory.entry_state(stdin=angr.SimFileStream(name='stdin', content=flag, has_end=False))

# Standard printable ASCII constraints
for ch in flag_chars:
    state.solver.add(ch >= 0x20)
    state.solver.add(ch <= 0x7e)

simgr = proj.factory.simulation_manager(state)
main_func_addr = proj.loader.find_symbol('main').rebased_addr
# OR proj.loader.main_object.min_addr (returns imagebase)

SUCCESS_ADDR = main_func_addr + 0x5b8
FAILURE_ADDR = main_func_addr + 0x38b

simgr.explore(find=SUCCESS_ADDR, avoid=FAILURE_ADDR)

if simgr.found:
    print("\n[+] Found a path!")
    print(simgr.found[0].posix.dumps(0))
else:
    print("\n[-] Unsat!")

Another interesting thing is PIE (Position Independent Executable), whether you use address or offset depends on whether the binary is PIE (that can be checked with the checksec tool).

Approaches to static analysis

When beginning a static analysis session, we typically adopt one of the following mindsets:

  • Bottom-Up Approach: Starting at the smallest functions or system calls and working backward to see how they fit into the bigger picture
  • Top-Down Approach: Starting at main() or the entry point and following the broad logic branches downward
  • Outside-In Approach: Starting at known interactions (like GUI inputs, network sockets, or specific string references) and tracing how the program handles them (XREFs)
  • Pattern-Recognition Approach: Recognizing standard compiler boilerplate, common library functions, or known algorithms without needing to read every instruction

Signature matching

Instead of manually reading every line of code, we can use signatures to identify known, constant data within a binary.

The true power of signatures lies in their ability to rapidly evaluate binaries at scale. Advanced signature engines can apply XOR operations and various transformations to the search patterns. This means that even if a binary has been tampered with or slightly obfuscated, you can still spot reused functions that you have already analyzed.

Additionally, there are vast repositories of public signatures for various malware families that you can leverage out-of-the-box.

  • Tools: Yara rules, FLIRT (FunctionID in Ghidra), Detect It Easy

In Ghidra, you can use the ghidra_yara plugin to generate custom rules directly from the code you are analyzing and scan the binary for existing rule matches. Or https://github.com/Neo23x0/yarGen that will extracts strings directly from the binary.

Applying what we learned earlier about stack frames, we can write a simple YARA rule to detect standard x64 function prologues:

rule detect_x64_function_prologues {
    meta:
        description = "Detects x64 function prologues"
        author = ""
        date = ""

    strings:
        // push rbp; mov rbp, rsp
        $prologue_standard = { 55 48 89 E5 }

        // push rbp; mov rbp, rsp; sub rsp, XX
        $prologue_alloc_short = { 55 48 89 E5 48 83 EC ?? }
        $prologue_alloc_long  = { 55 48 89 E5 48 81 EC ?? ?? ?? ?? }

        // (no RBP save): sub rsp, XX
        $prologue_leaf_short = { 48 83 EC ?? }
        $prologue_leaf_long  = { 48 81 EC ?? ?? ?? ?? }
    condition:
        any of them
}
None

By running YARA from the command line with the -s flag (which prints the matching strings and their locations)

We can find the exact file offset of the match. We then add this offset to the binary's image base address in your disassembler to jump straight to the hidden or undocumented function.

A Note on Ghidra Extensions

I want to quickly mention a minor annoyance regarding Ghidra extensions. While they add incredible functionality, managing them can sometimes be frustrating.

Because Ghidra updates frequently, older extensions will often refuse to load due to strict version matching. To keep your workflow intact after a Ghidra update, you will frequently need to recompile your extensions manually against your current installation.

You can do this using the Gradle build system included in the extension's directory:

export GHIDRA_INSTALL_DIR="/<yourpath>/ghidra-public"
cd $EXTENSION_DIR # with build.grandle
"$GHIDRA_INSTALL_DIR/support/gradle/gradlew" -PGHIDRA_INSTALL_DIR="$GHIDRA_INSTALL_DIR"

While YARA is fantastic for malware hunting and custom logic, FLIRT (Fast Library Identification and Recognition Technology) is designed specifically for a different problem: identifying statically linked library functions.

When developers compile a program statically, standard C/C++ library functions (like printf, memcpy, or malloc) are embedded directly into the executable. Without debugging symbols, the disassembler has no idea what these are and will label them with generic names (e.g., FUN_00401020).

FLIRT solves this by analyzing the raw bytes of these unknown functions and matching them against a massive database of known library signatures.

While incredibly useful, byte-matching techniques are somewhat fragile. Because they rely on specific sequences of assembly, they can break easily. If the target binary was compiled with heavy optimizations (like -O3), or if it was written in modern, LLVM-backed languages like Rust, the resulting assembly structure changes drastically. In these scenarios, traditional signatures often fail to recognize the functions.

Addictional resources:

Anatomy of the Portable Executable (PE) Format - Deep Hacking Discover the internal structure of the Windows PE format: DOS headers, NT Headers, sections and data directories for…

What Is Taint Analysis? A Guide for Developers and Security Researchers 1. Introduction

Dynamic binary instrumentation (DBI) with DynamoRio Learn how to build your own dynamic binary instrumentation (DBI) tool with open-source DynamoRIO to enable malware…

Solve equations and mathematical games with Z3 solver How to efficiently solve problems ranging from simple mathematical equations to advanced logical satisfiability.

Understanding Inter-process Communication in Linux: Fork, Pipes, Shared Memory, and Message Queues Boost Efficiency & Coordination: Mastering Linux IPC with Forking, Pipes, Shared Memory & Message Queues. Code examples…

https://quackie.at/files/reverse_engineering_rust_programs.pdf

GitHub - BitR13x/GhidraSignatures: script that will load signatures from file and apply it into… Script that will load signatures from file and apply it into Ghidra (string matching) - BitR13x/GhidraSignatures

GitHub - BitR13x/r2ai-workflow: AI workflow for radare2 - Connect to Ollama/NVIDIA-api models with… AI workflow for radare2 - Connect to Ollama/NVIDIA-api models with context taken from binary function through radare2…

https://quackie.at/files/reverse_engineering_rust_programs.pdf

Conclusion

With these concepts, we are now have a solid foundation for diving into reverse engineering and static analysis.

From the layers of abstraction — through basic hardware logic to how an operating system manages memory — compiled binaries don't have to be a black box anymore. You know how processes are built, how to use Ghidra and more.

At the end of the day, reverse engineering is really just solving a puzzle the compiler left behind. The best way to improve is to do your own research. Try writing some simple C programs (https://github.com/BitR13x/C-Reverse), compile them, and throw them straight into Ghidra to see exactly what they look like on the other side.

If you enjoyed this article, clap and follow me! Thanks for reading, and I wish you the best luck on your journey👋.