August 9, 2026

How a Floating-Point Error Becomes a Browser Exploit

Every number your CPU processes as a double is 64 bits. Most of those bit patterns represent actual numbers. But there's a corner of the…

By m0ms3c

7 min read

Every number your CPU processes as a double is 64 bits. Most of those bit patterns represent actual numbers. But there's a corner of the encoding — over 4.5 quadrillion distinct bit patterns — reserved for a single concept: "this is not a number."

That concept is NaN. And if you've only seen it as the annoying NaN that pops up in your JavaScript console, you're missing the bigger picture. NaN is one of the most abused primitives in modern browser exploitation.

This article walks through the full journey: what NaN actually is at the bit level, how CPUs and compilers handle it (and get it wrong), how JavaScript engines pack entire type systems inside NaN's unused bits, and how exploit developers weaponize that packing to take over your browser.

The Bit Layout Nobody Reads

A 64-bit IEEE 754 double looks like this:

[S] [  Exponent (11 bits)  ] [        Mantissa / Payload (52 bits)           ]

One sign bit, eleven exponent bits, fifty-two mantissa bits. When all 11 exponent bits are 1 and the mantissa is zero, you get Infinity. When all 11 exponent bits are 1 and any mantissa bit is non-zero, you get NaN.

Here's the part that matters: the spec only needs 12 bits to identify a value as NaN (the exponent plus one mantissa bit). The remaining 51 bits of the mantissa are free. The CPU doesn't care what's in them. They're just along for the ride.

Remember this. It's the foundation of everything that follows.

Two Flavors: Quiet and Signaling

Bit 51 of the mantissa (the most significant bit) splits NaN into two types:

qNaN:  0 11111111111 1xxx...xxx   ← bit 51 = 1, propagates silently
sNaN:  0 11111111111 0xxx...xxx   ← bit 51 = 0, triggers FP exception

Quiet NaN flows through arithmetic without raising any alarm. qNaN + 5.0 just gives you another qNaN. Signaling NaN fires an Invalid Operation exception the moment the FPU touches it. In practice, you'll almost never encounter a signaling NaN in the wild — every runtime and language generates quiet NaN exclusively.

The Equality That Isn't

NaN breaks the most basic assumption in programming: a value equals itself.

let x = NaN;
x === x;   // false
x !== x;   // true
x < 0;     // false
x > 0;     // false
x === NaN; // false

Every comparison involving NaN returns false, except !=. This is by design — NaN means "the result is undefined," and the IEEE 754 committee decided that undefined values should never compare equal to anything. Including themselves.

This gives you the oldest trick in the book for detecting NaN without library calls:

bool is_nan(double x) {
    return x != x;
}

It also means NaN infects computation chains. 0.0 / 0.0 produces NaN. That NaN flows into the next operation, which produces another NaN, and so on. One bad input poisons every downstream result. That's intentional — you can check once at the end instead of guarding every intermediate step.

When Your Compiler Deletes Your Security Checks

The broken equality of NaN creates a problem for compiler optimizers. By default, compilers respect IEEE 754 rules and preserve NaN behavior. But when you turn on aggressive optimization, the deal changes.

The -ffast-math Trap

GCC and Clang's -ffast-math flag (and its component -ffinite-math-only) tells the compiler: "assume NaN never exists in this program." That unlocks algebraic simplifications like treating x == x as always true and x * 0.0 as always zero.

Now look at this:

if (isnan(x)) {
    handle_error();  // sanitization, bounds check, abort
}
process(x);

With -ffast-math, the compiler can reason: "NaN doesn't exist, so isnan() is dead code." It removes the entire branch. Your sanitization disappears from the binary. The unchecked value hits process() directly.

This isn't hypothetical. I've seen real codebases where -ffast-math was enabled project-wide for performance, silently gutting floating-point validation across the entire build. If you're auditing a C/C++ codebase, check the build flags before trusting any NaN guard.

How CPUs Actually Check for NaN

At the assembly level, NaN detection comes down to a single CPU flag. But the mechanism differs between architectures, and the difference has real consequences for exploitation.

x86_64

The UCOMISD instruction compares two scalar doubles and sets flags in RFLAGS:

ucomisd xmm0, xmm0    ; compare value with itself
jp      .is_nan        ; Parity Flag set → unordered → NaN

When either operand is NaN, the comparison is "unordered" and the CPU sets the Parity Flag (PF). Compilers emit JP (jump if parity) to branch on NaN.

The security-relevant detail: x86 FPUs preserve NaN payloads. If you feed a NaN with a specific bit pattern into an ADDSD, the output NaN keeps those bits (with the quiet bit forced to 1). Your embedded data survives arithmetic. This property is what makes NaN-Boxing reliable on x86.

ARM64

ARM uses FCMP and checks the NZCV condition flags:

fcmp    d0, d0
b.vs    .is_nan        ; VS (overflow set) → unordered → NaN

Here's the critical difference: most ARM64 FPU instructions perform NaN canonicalization. When an operation produces or propagates a NaN, the hardware overwrites the payload with the default NaN (0x7FF8000000000000). Your embedded data is destroyed.

This means any technique that stores information in NaN payloads and passes them through floating-point arithmetic is architecture-dependent. What works on x86 will silently break on ARM. Keep this in mind when we get to NaN-Boxing.

NaN-Boxing: An Entire Type System in 8 Bytes

Here's where NaN stops being an arithmetic curiosity and becomes an engineering tool.

JavaScript is dynamically typed. Every variable can hold a number, a string, an object, null, undefined, a boolean — anything. The runtime needs to track both the value and the type of every single variable. The naive approach is a tagged union: 8 bytes for a type tag, 8 bytes for the value, 16 bytes total per variable. In a language where everything is a value, that overhead adds up fast.

NaN-Boxing solves this by cramming everything into exactly 8 bytes — the same size as a raw double.

The Trick

If the 64-bit value is a normal double (not NaN), the runtime reads it directly as a floating-point number. Zero overhead, zero conversion.

If the top bits match the NaN pattern, the value is not a float — it's a tagged payload. The runtime extracts a type tag and a value from the remaining bits:

Regular double:  [       any valid non-NaN IEEE 754 encoding                ]
                 └─ read directly as a float, no tag check needed
Boxed pointer:   [1111111111111] [TAG: PTR ] [   48-bit heap address        ]
Boxed int32:     [1111111111111] [TAG: INT ] [   32-bit integer value        ]
Boxed special:   [1111111111111] [TAG: SPEC] [  null / undefined / bool      ]
                  ↑ NaN marker                ↑ type tag    ↑ actual value

Why 48 bits for pointers? Because on current 64-bit systems, only 48 bits of virtual address space are actually used. A heap pointer fits inside a NaN payload with room left over for the tag.

The result: a single 64-bit word encodes every JavaScript type. Floats pass through at hardware speed. Everything else is one bitwise check and a mask away.

The Security Boundary

NaN-Boxing creates a single 64-bit boundary between raw floating-point data and typed object pointers. On one side, bits are arithmetic. On the other, they're memory addresses.

If an attacker can make the engine read the wrong side — interpret a pointer as a float, or a float as a pointer — the entire type system collapses.

Crossing the Boundary: addrof and fakeobj

This is where NaN goes from interesting to dangerous. Modern browser exploits against JIT engines (V8, SpiderMonkey, JavaScriptCore) almost universally build on two primitives that exploit the NaN-Boxing boundary.

How the Confusion Happens

JIT compilers optimize JavaScript by speculating on types. If a function is called a hundred times with an array of doubles, the JIT generates a fast path that reads elements as raw unboxed doubles — no type check, no NaN-tag inspection, straight to the metal.

The bug occurs when the attacker forces a mismatch between what the JIT thinks is in the array and what's actually there. Common triggers:

  • Side-effects in callbacks — a valueOf() or toString() override that mutates the array's element type while the JIT isn't looking
  • Bounds check elimination — the JIT removes a length check that was actually necessary
  • Range analysis errors — the compiler infers a type that's too narrow for the actual runtime value

When the JIT thinks it's reading a double[] but the array actually contains NaN-boxed object pointers, the type confusion is live.

addrof(obj) — Reading a Pointer as a Float

The attacker passes a JavaScript object into a JIT-compiled function that expects a double array. The JIT skips the NaN-tag check (it "knows" everything is a double) and returns the NaN-boxed pointer bits as a raw floating-point number.

The attacker converts those float bits back to an integer. The result is the 48-bit heap address of the object.

ASLR is defeated. The attacker now knows where any reachable object lives in memory.

fakeobj(addr) — Writing a Float as a Pointer

The inverse. The attacker writes a crafted double — whose bit pattern encodes a chosen memory address — into an array that the JIT treats as containing boxed objects. The JIT reads the raw float bits as a NaN-boxed pointer and returns it as a valid JavaScript object.

The attacker now holds a reference to an arbitrary memory address, treated by the engine as a legitimate object.

The Chain

With these two primitives, the standard exploitation flow is:

  1. Leak addresses of key objects — ArrayBuffer backing stores, JIT code pages, internal structures
  2. Forge a fake ArrayBuffer whose backing store pointer targets arbitrary memory
  3. Read and write through the fake ArrayBuffer → arbitrary r/w across the process
  4. Overwrite JIT-compiled code or a function pointer → code execution
  5. Chain with a sandbox escape for full system compromise

All of this starts with 51 unused bits in a floating-point NaN.

How Engines Fight Back

The NaN-Boxing attack surface has driven serious hardening:

At the hardware level, ARM64's NaN canonicalization destroys embedded payloads, making NaN-based data smuggling unreliable. Pointer Authentication (PAC) on Apple Silicon cryptographically signs pointer bits, so forged NaN-boxed pointers fail validation and crash. Memory Tagging (MTE) on newer ARM chips tags each memory granule, making fake objects hit tag mismatches on access.

At the JIT level, modern engines insert speculation guards that re-check type assumptions before fast-path access. If the speculation was wrong, the code bails out to the interpreter. Some engines apply NaN masking — bitwise operations that force extracted pointers into valid address ranges before dereference.

At the architecture level, V8's sandbox (Ubercage) confines all JavaScript heap objects within a 1TB virtual memory cage. Even with full arbitrary r/w over the V8 heap, the attacker can't directly touch process memory outside the cage. V8 also uses 32-bit compressed pointers (offsets within the cage) instead of raw 64-bit addresses, which changes the NaN-Boxing equation. Control Flow Integrity (CFI) validates indirect call targets, making function pointer overwrites harder to weaponize.

None of these are silver bullets. Each raises the cost of exploitation. Combined, they've turned what was once a straightforward type confusion into a multi-stage research project.

The Takeaway

NaN is a masterclass in how encoding decisions compound.

A committee in the 1980s needed a way to say "this computation failed" without crashing the program. They reserved a slice of the floating-point encoding space, left 51 bits unused, and moved on. Three decades later, language implementers discovered those bits were the perfect place to hide a type system. And a few years after that, exploit developers discovered that the boundary between "these bits are a number" and "these bits are a pointer" is one of the most reliable attack surfaces in modern software.

The broader pattern applies beyond NaN: any encoding that multiplexes different semantic types into a shared bit representation creates a type confusion surface. Tagged pointers, union types in C, pointer compression — same tradeoff every time. Performance for attack surface. Understanding where those boundaries live is fundamental to both breaking and defending modern runtimes.

I do vulnerability research on V8 and Chromium. Everything described here is for defensive and educational purposes.