Post cover image

July 26, 2026

The Linux Developer’s Toolbox: Debugging

Learn how Linux developers investigate crashes, trace system behavior, and uncover elusive bugs.

By Kobi Toueg

9 min read

Continue reading the complete article here — free access.

gdb

📖 What is it?

GDB (GNU Debugger) is the standard debugger for native Linux applications. It allows developers to pause a running program, inspect its internal state, execute code step by step, examine memory, and analyze crashes. Although many IDEs integrate graphical front ends, GDB itself is primarily a command-line tool and has long been the foundation of debugging on Linux systems.

🛠️ What does it do?

GDB executes a program under debugger control or attaches to an already running process. Developers can set breakpoints, step through the code, inspect variables, evaluate expressions, examine memory, view the call stack, and switch between threads.

To display source code, function names, local variables, and line numbers, the executable should be compiled with debug symbols. Without debug symbols, GDB can still debug the program, but much of the information available to the developer is limited.

Unlike the graphical debuggers provided by IDEs such as Visual Studio, Eclipse, or Xcode, GDB primarily provides a command-line interface. While it has a learning curve, its text-based design makes it lightweight and suitable for debugging applications running on remote servers, containers, or systems accessed over SSH.

Example:

gcc -g program.c -o program
gdb ./program

(gdb) break main
(gdb) run
(gdb) next
(gdb) print counter
(gdb) bt

I think this is more consistent with the philosophy of your Linux series: introduce the concept, show what it does, avoid diving into compiler optimization flags unless the article is specifically about building software.

💡 Why should every Linux developer know about it?

Many software defects cannot be diagnosed from log files alone. GDB allows developers to observe the exact state of a program while it is executing or immediately after it crashes, enabling them to identify issues such as invalid memory accesses, incorrect program state, race conditions, and unexpected control flow. It is one of the most important tools in the Linux development ecosystem and an essential skill for native C and C++ developers.

Core dumps

📖 What is it?

A core dump is a file that captures the complete state of a process at the moment it crashes. It contains the process's memory, CPU registers, threads, call stacks, and other execution information, allowing developers to investigate the failure after the program has already terminated.

🛠️ What does it do?

When a program terminates due to a fatal error, such as a segmentation fault, the Linux kernel automatically attempts to generate a core dump capturing the process's state at the moment of the crash. Whether the dump is actually written depends on the system configuration and the process's resource limits.

Linux uses the ulimit shell command to configure resource limits for processes before they are started. Different options control different resources, such as the maximum stack size (-s), the maximum number of open files (-n), and the maximum size of a core dump (-c). Since the kernel attempts to write the entire process image, the core dump size limit must be large enough to accommodate it. A common configuration during development is to remove this limit:ulimit -c unlimited

After the application crashes, the core dump can be examined together with the original executable:

gdb ./my_program core

If the executable contains debug symbols, GDB can display the source code, variables, and function names corresponding to the crash.

Common commands include:

(gdb) bt              # Display the call stack
(gdb) frame 0         # Select the crashing stack frame
(gdb) list            # Display the source code around the current line
(gdb) print variable  # Inspect a variable
(gdb) info threads    # List all threads

Using these commands, developers can determine where the crash occurred, inspect the program's state, and view the surrounding source code without reproducing the failure.

💡 Why should every Linux developer know about it?

Many crashes occur on production systems where attaching a debugger is impossible or where the process has already terminated. A core dump preserves the exact state of the application at the moment of failure, allowing developers to perform post-mortem debugging on another machine later. Combined with GDB and debug symbols, it often provides enough information to identify the root cause of a crash without reproducing it.

strace

📖 What is it?

strace is a Linux diagnostic tool that traces the system calls made by a process. Since every interaction between a user-space application and the Linux kernel occurs through system calls, strace provides a detailed view of how an application communicates with the operating system.

🛠️ What does it do?

strace intercepts and logs each system call together with its arguments, return value, and any reported error. It can trace a program from the moment it starts or attach to an already running process.

For example, to trace a program:

strace ./my_program

Or attach to an existing process:

strace -p 12345

Typical output looks like:

openat(AT_FDCWD, "/etc/config.conf", O_RDONLY) = 3
read(3, "configuration...", 4096)             = 1024
write(1, "Application started\n", 20)          = 20
close(3)                                     = 0

From this output, developers can determine which files were accessed, whether network connections succeeded, how processes were created, why a system call failed, or whether the application is blocked waiting for I/O.

💡 Why should every Linux developer know about it?

Many application failures are not caused by bugs in the program itself but by problems interacting with the operating system. A missing configuration file, insufficient permissions, an unavailable network resource, or a failed system call can often be identified immediately with strace without modifying the application's code.

Because it operates outside the application, strace is especially valuable when source code is unavailable or when debugging production systems. It is often one of the first tools Linux developers reach for when an application behaves unexpectedly or simply "doesn't work."

ltrace

📖 What is it?

ltrace is a Linux diagnostic tool that traces calls made by an application to dynamically linked shared libraries. These libraries can be system libraries, third-party libraries, or application-specific shared libraries. As the program executes, ltrace displays the library functions being invoked, together with their arguments and return values, allowing developers to observe how the application interacts with external code.

🛠️ What does it do?

ltrace intercepts calls to functions in dynamically linked shared libraries at runtime. It can either start a program under its control or attach to an already running process.

To trace a program from startup:

ltrace ./my_program

To attach to an existing process:

ltrace -p 12345

In either case, ltrace displays library calls as they occur. When attaching to a running process, tracing begins at the moment of attachment and cannot display library calls that occurred earlier.

Typical output looks like:

malloc(1024)                          = 0x55b4f4f642a0
printf("Connecting to %s\n", "db")    = 19
my_encrypt("secret")                  = 0
free(0x55b4f4f642a0)                  = <void>

The trace shows the library function being called, the arguments passed to it, and its return value. ltrace can monitor calls to any dynamically linked shared library, whether it is a system library such aslibc, a third-party library such as OpenSSL, or a shared library developed specifically for your application.

Because ltrace displays function arguments, it can also expose sensitive information passed to library functions, such as SQL queries, authentication tokens, API keys, passwords, or plaintext data provided to encryption or networking libraries. For this reason, it should be used with appropriate care when debugging production applications.

Functions that are statically linked into the executable do not appear in the trace because their code becomes part of the executable during the linking process rather than being called through a shared library. Likewise, functions inlined by the compiler cannot be traced.

💡 Why should every Linux developer know about it?

Modern applications rely heavily on shared libraries for functionality such as memory management, encryption, networking, database access, image processing, and application-specific business logic. ltrace reveals which library functions are being executed, the arguments they receive, and the values they return, making it invaluable for diagnosing incorrect API usage, unexpected library behavior, integration problems, and bugs inside custom shared libraries. At the same time, understanding that ltrace can expose sensitive information passed through library APIs helps developers better appreciate the importance of Linux process isolation and debugging permissions.

Sanitizers

📖 What is it?

Sanitizers are a family of compiler-assisted runtime analysis tools that detect programming errors while an application is executing. Instead of changing the operating system or debugger, the compiler inserts additional checks into the generated executable. These checks monitor the program at runtime and immediately report memory errors, data races, undefined behavior, and other defects that can otherwise be extremely difficult to diagnose.

Sanitizers are primarily used during development, testing, and continuous integration to identify bugs before software reaches production.

🛠️ What does it do?

A sanitizer is enabled by compiling the application with the appropriate compiler option. The compiler inserts additional runtime checks into the executable that validate operations as the program runs and immediately report violations, along with the source file, line number, stack trace, and the location where the problem originated.

The most commonly used sanitizers are:

AddressSanitizer (ASan) — Detects heap and stack buffer overflows, use-after-free errors, double frees, invalid frees, and other memory corruption bugs.

g++ -fsanitize=address -g my_program.cpp -o my_program

ThreadSanitizer (TSan) — Detects data races and synchronization errors in multithreaded applications.

g++ -fsanitize=thread -g my_program.cpp -o my_program

UndefinedBehaviorSanitizer (UBSan) — Detects operations that invoke undefined behavior according to the C and C++ language standards, such as signed integer overflow, invalid shifts, and invalid pointer operations.

g++ -fsanitize=undefined -g my_program.cpp -o my_program

LeakSanitizer (LSan) — Reports memory leaks when the application terminates.

g++ -fsanitize=leak -g my_program.cpp -o my_program

MemorySanitizer (MSan) — Detects reads of uninitialized memory.

clang++ -fsanitize=memory -g my_program.cpp -o my_program

Note: Unlike the other commonly used sanitizers, MemorySanitizer is primarily supported by the Clang/LLVM toolchain. GCC does not provide equivalent support, which is why this example uses clang++ instead of g++.

Because sanitizers instrument the executable during compilation, they introduce additional runtime overhead and increased memory usage. As a result, they are typically enabled for debugging, testing, and continuous integration rather than production deployments.

💡 Why should every Linux developer know about it?

Memory corruption, undefined behavior, and data races are among the most difficult bugs to reproduce and debug in native applications. Sanitizers detect many of these problems at the moment they occur and provide precise diagnostic information, often identifying both the location of the failure and the code that caused it. Compared to traditional dynamic analysis tools such as Valgrind's Memcheck, they generally impose significantly lower runtime overhead, making them practical for everyday development and automated testing. Today, many large C and C++ projects build and execute dedicated sanitizer configurations as part of their continuous integration process, making sanitizers an essential part of the modern Linux developer's debugging toolbox.

Valgrind

📖 What is it?

Valgrind is an open-source dynamic analysis framework for Linux that provides a collection of tools for debugging and profiling native applications. Its tools help developers detect memory errors, thread synchronization problems, memory leaks, excessive memory usage, and performance bottlenecks. Although it is not part of the Linux operating system itself, it is included in the package repositories of virtually all major Linux distributions and has become a standard tool for C and C++ development.

Unlike a debugger, which allows developers to inspect a running program interactively, Valgrind executes the application inside an instrumentation framework that observes its behavior at runtime and reports errors or performance statistics as the program executes.

🛠️ What does it do?

Valgrind is a framework containing several analysis tools, each designed to diagnose a different class of problems.

Memcheck (the default tool) detects memory leaks, invalid memory accesses, use-after-free errors, double frees, invalid frees, and reads of uninitialized memory.

valgrind ./my_program

For explicitly:

valgrind --tool=memcheck ./my_program

Memory errors are reported immediately upon occurrence during execution, while the final memory leak analysis is produced when the application terminates. For long-running applications such as database servers or web servers, developers typically run Valgrind in a development or test environment, exercise the application using regression or stress tests, and then shut it down gracefully. Allowing the application to perform its normal cleanup significantly reduces false positives and makes it much easier to distinguish genuine memory leaks from memory that would normally be released during an orderly shutdown.

Helgrind analyzes multithreaded applications and reports synchronization problems such as incorrect mutex usage and potential data races.

valgrind --tool=helgrind ./my_program

DRD is another thread analysis tool that detects data races and synchronization errors.

valgrind --tool=drd ./my_program

Massif profiles heap memory usage over time, helping developers understand where an application allocates memory and how its memory footprint evolves.

valgrind --tool=massif ./my_program

Callgrind profiles function execution and generates detailed call graphs that help identify performance bottlenecks.

valgrind --tool=callgrind ./my_program

Cachegrind analyzes instruction execution and simulates CPU cache behavior, helping developers optimize memory access patterns and reduce cache misses.

valgrind --tool=cachegrind ./my_program

Because Valgrind instruments every instruction executed by the program, it introduces significantly higher runtime overhead than compiler-based sanitizers. However, it does not require recompiling the application, making it useful for analyzing existing executables as well as applications for which the source code or build process is unavailable.

💡 Why should every Linux developer know about it?

Valgrind has long been one of the essential tools for developing native Linux applications. Its collection of analysis tools supports both correctness analysis and performance analysis within a single framework. Memcheck detects many of the same memory errors as compiler-based sanitizers while also identifying reads of uninitialized memory without requiring a specially compiled executable. At the same time, tools such as Callgrind, Cachegrind, and Massif help developers understand application performance, cache efficiency, and memory usage. Although compiler-based sanitizers are often preferred in everyday development for their lower overhead, Valgrind remains invaluable for analyzing existing binaries, testing long-running services, and investigating complex runtime behavior without rebuilding the application.

Because Valgrind operates entirely in user space, it cannot analyze memory allocated by the Linux kernel. Kernel developers instead rely on dedicated tools such as KASAN (Kernel Address Sanitizer) to detect invalid memory accesses and kmemleak to identify leaked kernel allocations.

🤝

Kobi Toueg Engineering Leader | AI, Security & System Architecture | Principal Software Engineering & Technical Leadership

LinkedIn: https://www.linkedin.com/in/kobi-toueg