Reverse Engineering: How It Works, the Workflow, and the Basic Rules

What it is

Reverse engineering (RE) means working backwards from a finished product to understand how it was made and how it works. In software, that usually means analysing a compiled program without its source code.

How it works: the core idea

When code is compiled, it turns into machine code that the CPU runs:

Source code  →  [compiler]  →  Machine code (binary)
   (C, Rust, Go…)                 (numbers the CPU executes)

Source code  →  [compiler]  →  Machine code (binary)
   (C, Rust, Go…)                 (numbers the CPU executes)

Compiling throws information away: variable names, comments, structure and often function names. Reverse engineering rebuilds as much of that meaning as it can:

Binary  →  Disassembler  →  Assembly  →  Decompiler  →  C-like pseudocode  →  Human understanding

Binary  →  Disassembler  →  Assembly  →  Decompiler  →  C-like pseudocode  →  Human understanding

The one fact that makes RE possible: the CPU has to be able to run the program, so all of its logic is in the file. It may be hidden, optimized or obfuscated, but it can't be removed.

The standard workflow

1. Define the goal

Decide what you want to know. Examples: "Does this malware steal credentials?", "How does this file format work?" or "Where is the licence check?" A clear question keeps you from getting lost in millions of instructions.

2. Triage: identify the target

• File type and format (PE on Windows, ELF on Linux, Mach-O on macOS, APK on Android)
• CPU architecture (x86–64, ARM64…)
• Is it packed, compressed or obfuscated? High entropy is a sign.
• Tools: file, Detect It Easy, entropy checks

3. Surface reconnaissance

• Strings: messages, URLs, file paths, keys, error text
• Imports and exports: which system functions it calls (network, crypto, file, registry)
• Metadata: compiler, timestamps, embedded resources, signatures

This step often answers most of the question cheaply.

4. Static deep-dive

• Load it into a disassembler or decompiler (Ghidra is free)
• Find the entry point and main
• Follow interesting strings and imports back to the code that uses them (cross-references)
• Rename functions and variables as you work out what they do. This is most of the job.

5. Dynamic analysis

• Run it in an isolated environment: a VM or sandbox, especially for malware
• Set breakpoints at key functions and inspect registers and memory
• Watch file, network and process activity

6. Confirm and iterate

Test your theory by changing an input, patching an instruction, or predicting an output. If the prediction fails, go back to step 4.

7. Document

Write up your findings: what it does, how, the evidence, and indicators (hashes, domains, behaviours). Undocumented RE work is effectively lost.

Basic rules

Legal and ethical rules

1. Only analyse what you own or are authorized to analyse. Your own code, CTF challenges, crackme sites, malware samples for research, or work under a contract or engagement.
2. Read the licence. Many EULAs forbid reverse engineering. How enforceable that is depends on the country.
3. Know the legal exceptions, broadly. In the US, DMCA §1201(f) allows some RE for interoperability, and court cases such as Sega v. Accolade (1992) treated RE for interoperability as fair use. In the EU, the Software Directive (Art. 6) allows decompilation for interoperability. Security research has some protections, but the details vary. This isn't legal advice, so check before doing serious work.
4. Don't use findings to cause harm. No cracking for piracy, and no exploiting systems you don't own.
5. Disclose vulnerabilities responsibly. Report them to the vendor and give them time to fix the problem before going public.

Safety rules

1. Never run unknown binaries on your main machine. Use a VM with snapshots and no shared folders.
2. Isolate the network when running malware, or simulate it with tools like INetSim or FakeNet.
3. Assume samples are live and dangerous. Store them password-protected (infected is the common convention) and with renamed extensions.

Technical rules of thumb

1. Go from broad to deep. Strings and imports first, assembly later.
2. Don't read every instruction. Focus on the code tied to your goal, and skip library and boilerplate code.
3. Rename everything you understand. Meaningful names build up into a readable program.
4. Trust behaviour over assumptions. Code can lie: strings can be decoys and functions can be misleadingly named.
5. Record your work. Ghidra project notes, comments and a running log let you resume later and help others.

Useful facts

• Ghidra, a full-featured RE suite, was built by the NSA and released free and open source in 2019. It made professional-grade decompiling widely available.
• IDA Pro by Hex-Rays has been the industry-standard disassembler since the 1990s.
• Clean-room reverse engineering made the PC clone industry possible. In the early 1980s, Compaq and Phoenix Technologies reimplemented the IBM PC BIOS: one team documented its behaviour, and a separate team that never saw IBM's code wrote a new version from that description.
• WannaCry (2017) was slowed when researcher Marcus Hutchins, analysing the malware, found an unregistered domain it checked before spreading. Registering that domain acted as a kill switch.
• Stuxnet (2010) took months of RE by multiple security firms to understand. It's one of the most-studied pieces of malware ever.
• RE isn't only for security. It's used for interoperability (file formats, drivers, protocols), game modding, legacy software maintenance when the source is lost, and hardware and firmware analysis.
• Compiler settings make a big difference. The same 10-line program can be easy to read at -O0 and hard to read at -O3.