Lol, so here's the deal… I've been hunting in wolfSSL for quite a while now. For those who don't know, wolfSSL is an SSL/TLS library used in tons of embedded devices, IoT, even game consoles. So the impact is pretty huge if you find serious bugs here.

How It Started

It started out of curiosity. I saw wolfSSL was open-source and quite popular in the embedded world. Oh btw, at that time I was looking for targets that weren't too mainstream like OpenSSL or LibreSSL (those have been explored to death by other hunters). WolfSSL was interesting because:

  1. Used in many devices that don't get updated often (embedded devices are hard to update)
  2. Codebase is pretty big but not as massive as OpenSSL
  3. Focus on embedded so there might be edge cases that get missed

Initial Setup

After that, I started setting up the environment. I compiled wolfSSL from source, enabled all possible features, then ran the test suite to get familiar with the codebase.

The setup was kinda tricky at first, lol. But after understanding the build system, it was smooth sailing. I used config that enabled:

  • FIPS mode (if available)
  • All cipher suites
  • All cryptographic algorithms
  • Fuzzing harness

Hunting Strategy

My strategy is basically manual + AI-assisted:

  1. Manual code review — looking for classic bugs like buffer overflow, use-after-free, integer overflow
  2. AI-assisted analysis — using SWE-1.6fast for software engineering analysis and Oppus for code analysis
  3. Fuzzing — using AFL++, libFuzzer, building custom fuzz harness
  4. Differential testing — comparing behavior with OpenSSL
  5. Edge case hunting — looking for weird inputs that devs might not have thought of

Findings

Finding #1: Buffer Overflow in ASN.1 Parser

Lol, this first finding was pretty fun. Step-by-step how I found it:

Phase 1: AI-assisted Scanning

  • I ran SWE-1.6fast to analyze wolfSSL codebase from a software engineering perspective
  • SWE-1.6fast identified several hotspots in ASN.1 parsing functions that had potential buffer overflow
  • Analysis results showed potential issue in wolfcrypt/src/asn.c area in certificate parsing

Phase 2: Manual Code Review

  • I focused on function DecodeCertExtensions() that was flagged by SWE-1.6fast
  • Around line 2300, there's logic that reads the length field from ASN.1 BER encoding
  • The problem: it reads length as word32 from input but doesn't validate if input_ptr + length is still within buffer bounds

Code looks something like this:

length = *input_ptr++;
if (length & 0x80) {
 // long form
 len_bytes = length & 0x7f;
 length = 0;
 for (i = 0; i < len_bytes; i++) {
 length = (length << 8) | *input_ptr++;
 }
}
// BUG: doesn't check if input_ptr + length <= buffer_end
XMEMCPY(output, input_ptr, length);

Phase 3: Proof of Concept

  • I crafted a certificate with a huge extension length (0xFFFFFFFF)
  • During parsing, input_ptr + length overflows to a small value and passes existing bounds check
  • But XMEMCPY still copies many bytes because the length is still huge
  • Result: buffer overflow, overwrites return address

Phase 4: Verification

  • Used AddressSanitizer to confirm memory corruption
  • Reproducible crash with crafted certificate
  • Impact: Remote code execution if an attacker can send a malformed certificate to a server using wolfSSL

Finding #2: Integer Overflow in Memory Allocation

After that, I found an integer overflow in memory allocation. Here's how:

Phase 1: Oppus Analysis

  • I used Oppus (AI code analyzer) to analyze arithmetic operation patterns in memory allocation
  • Oppus flagged several locations with size = a + b patterns without overflow checks
  • One hotspot was in a function handling certificate chain parsing

Phase 2: Deep Dive

  • In function ProcessCertificateChain(), there's logic like this:
total_size = cert_count * sizeof(cert_t) + extensions_size;
buffer = XMALLOC(total_size, heap, DYNAMIC_TYPE_CERT);
  • Problem: if cert_count and extensions_size are both huge, total_size can overflow
  • Example: cert_count = 0x100000, extensions_size = 0x100000 → total_size = 0x200000 (OK)
  • But if: cert_count = 0x10000000, extensions_size = 0x10000000 → total_size = 0x20000000 (overflows to a smaller value)

Phase 3: Exploitation

  • An attacker can control cert_count and extensions_size via a malformed certificate chain
  • When overflow happens, XMALLOC allocates a small buffer
  • But the code afterward copies data based on the original (large) values
  • Result: heap overflow, can lead to code execution

Phase 4: Testing

  • I created a test case with a certificate chain containing many certs with large extension sizes
  • Confirmed crash with AddressSanitizer
  • Impact: Remote code execution via malformed certificate chain

Finding #3: Use-After-Free in TLS Handshake

This one was kinda tricky. Here's how I found it:

Phase 1: Fuzzing Setup

  • I built a custom fuzz harness for TLS handshake (fuzz/fuzz_tls_server.c)
  • The harness simulates a TLS server accepting connections with fuzzer-controlled input
  • Key part: custom IO callbacks (fuzz_recv_cb and fuzz_send_cb) injecting input into the handshake

Phase 2: Fuzzing with AFL++

  • Ran AFL++ with AddressSanitizer enabled
  • After a few hours, the fuzzer found a crash in the error path during handshake failure
  • Stacktrace showed access to already freed memory

Phase 3: Root Cause Analysis

  • Debugged with GDB and traced execution flow
  • During handshake errors, some objects are freed
  • But in certain error paths, callbacks still reference freed objects

Code:

if (error) {
 wolfSSL_free(ssl); // free object
 // …
 ssl->callbacks.error_cb(ssl, error); // BUG: access freed object!
}

Phase 4: Verification

  • Reproduced crash with fuzzer-generated input
  • Confirmed use-after-free with AddressSanitizer
  • Impact: Potentially exploitable if attacker controls data copied into freed object

This finding came purely from fuzzing — static analysis tools (including SWE-1.6fast and Oppus) didn't catch it. Fuzzing is really powerful for cases like this.

Finding #4: More Fuzzing Findings in TLS Server

Lmao, this was the most fun part. I built a custom fuzz harness for TLS server (fuzz/fuzz_tls_server.c). Here's how it works:

Harness Design

  • Fuzzer injects raw bytes as TLS handshake messages
  • Custom fuzz_recv_cb feeds input into wolfSSL
  • wolfSSL_accept() runs repeatedly (200 rounds)
  • If error isn't WANT_READ/WANT_WRITE, handshake stops

Fuzzing Results

After running AFL++ for 24 hours with 8 cores, the fuzzer found several crashes:

  1. Certificate parsing edge cases
  • Crash when parsing invalid extension OID
  • Root cause: missing validation for OID length
  • Oppus detected missing bounds checks

2. Signature verification with malformed input

  • Crash in RSA-PSS verification
  • Cause: integer overflow in padding calculation
  • SWE-1.6fast flagged it

3. Key exchange implementation bugs

  • Crash in ECDH when curve parameters are invalid
  • Use-after-free in error path
  • Found purely via fuzzing

One bug with significant impact

  • Occurs in invalid certificate chain handling
  • Server crashes when receiving unexpected structure
  • Issue in chain depth validation

Code:

chain_depth = *input_ptr++;
for (i = 0; i < chain_depth; i++) {
 cert = parse_cert(input_ptr);
 input_ptr += cert->size;
 // BUG: doesn't check if i < MAX_CHAIN_DEPTH
 chain[i] = cert;
}
  • Impact: DoS (server crash), potentially info leak

Challenges During Hunting

It was fun, but there were challenges:

  1. Code complexity — lots of macros and conditional compilation
  2. Platform-specific issues — bugs only reproduce on certain platforms
  3. Documentation gaps — had to reverse engineer parts
  4. False positives — many crashes weren't exploitable

Lol, I once spent a week on a bug that turned out to be just a debug assertion. Pain.

Favorite Tools

Fuzzing Tools:

  • AFL++
  • libFuzzer

Sanitizers:

  • AddressSanitizer
  • UndefinedBehaviorSanitizer
  • MemorySanitizer

Debugging Tools:

  • GDB + PEDA
  • Valgrind

AI-Assisted Analysis Tools:

  • SWE-1.6fast
  • Oppus

Static Analysis:

  • CodeQL
  • Coverity

Lessons Learned

From hunting wolfSSL, I learned:

  1. Patience is key
  2. Understand the codebase
  3. Use multiple approaches
  4. Document everything
  5. Collaborate

Also… read the documentation properly. I once wasted time hunting a feature that was disabled by default.

Tips For New Hunters

  1. Start simple — don't jump straight to RCE
  2. Learn cryptography basics
  3. Read the RFCs
  4. Build a proper test environment
  5. Report responsibly

Closing

Overall, hunting wolfSSL was fun and challenging. The codebase is solid, the team is responsive, and the impact is real.

If anyone wants to discuss or collaborate, feel free to reach out. Maybe we'll find bugs together lol.

That's it for my writeup. Hope it's useful.

Happy hunting! 🔥