AES GCM Nonce Reuse: The Forbidden Attack Explained

AES GCM is one of the most used authenticated encryption modes on the internet. It gives you both secrecy and a check that nobody tampered with the message, and it is fast. It also has one rule that, if broken, turns a strong cipher into a weak one: every message encrypted under a given key must use a fresh, unique nonce. AES GCM nonce reuse breaks both halves of that promise. Repeat a nonce under the same key and an attacker can read relationships between your messages and, worse, forge valid tags for messages you never sent.

How AES GCM works, briefly

GCM stands for Galois Counter Mode. It bolts two pieces together. The first is a keystream. AES runs in counter mode, so AES never touches your plaintext directly. It encrypts a sequence of counter blocks built from the nonce, producing a stream of pseudo random bytes, and your ciphertext is the plaintext XORed with that keystream.

The second piece is the authentication tag. GCM computes a polynomial MAC called GHASH over the ciphertext and any associated data. GHASH uses a secret value called H, which is AES applied to an all zero block under your key. The result is mixed with a value derived from the nonce to form the tag the receiver checks.

So one encryption produces two outputs that both depend on the nonce: the keystream that hides the plaintext, and the tag that detects tampering. The nonce is the only thing that changes between two messages under the same key. That is why it has to be unique.

Why AES GCM nonce reuse is catastrophic

Reuse hits both pieces. Think of it as two failures that happen together.

Failure one: the keystream repeats

Counter mode turns a block cipher into a stream cipher, and the cardinal rule of any stream cipher is that you never reuse the keystream. With GCM the keystream is fully determined by the key and the nonce. Same key, same nonce, same keystream. So if you encrypt two messages, P1 and P2, with the same key and nonce, you get:

C1 = P1 XOR keystream C2 = P2 XOR keystream C1 XOR C2 = (P1 XOR keystream) XOR (P2 XOR keystream) = P1 XOR P2

C1 = P1 XOR keystream C2 = P2 XOR keystream C1 XOR C2 = (P1 XOR keystream) XOR (P2 XOR keystream) = P1 XOR P2

The keystream cancels out. The attacker now holds the XOR of two plaintexts and never needed the key. If one message is known or guessable, the other falls out directly. Even with two unknown messages, the XOR of English text or structured JSON leaks a lot. No key was broken. Secrecy evaporated because the same mask was used twice.

Failure two: the forbidden attack recovers the auth key

This is the part that surprises people. The integrity guarantee collapses too. GHASH is a polynomial evaluated at the secret point H in a finite field, and the tag is, roughly, that polynomial plus a nonce dependent mask.

When two messages share a nonce, that mask is identical for both. Subtract one tag equation from the other and the mask cancels, the same way the keystream did. What remains is a polynomial in a single unknown, H. Every coefficient is known except H, so solving for the roots in the field recovers the authentication subkey.

Reusing a nonce does not just leak one message. It can hand the attacker the key to GHASH, after which they forge valid tags for messages of their choosing and the receiver accepts them as genuine.

Once H is known, the attacker can compute the GHASH of any ciphertext they like and produce a tag the receiver will verify. That is the forbidden attack, the documented consequence of doing the one thing GCM tells you never to do.

How nonces actually get reused in the wild

Nobody writes "reuse the nonce" on purpose. It happens through ordinary mistakes.

Random 96 bit nonces and the birthday bound

The recommended nonce for GCM is 96 bits, often generated at random because that is easy. Random looks safe, but random values collide. The birthday bound says you expect a repeat after roughly two to the power of n over two values for an n bit space, which for 96 bit nonces is around two to the 48th messages under one key. For a single laptop that is plenty of headroom. For a fleet of servers sharing one key at high volume, two to the 48th is reachable, and a single collision is enough to start the attack.

Counters that reset on reboot

A counter based nonce is safer than random, but only if it never goes backward. A device that keeps its counter in memory and restarts at zero after a crash or reboot will re emit nonces it already used. Same key across reboots plus a counter that resets equals guaranteed reuse.

One key shared across many encryptors

Spread the same key across several machines and you have to coordinate their nonces. If two of them independently pick from the same range, they emit the same nonce under the same key. Cloning a virtual machine that already holds key and counter state quietly duplicates nonces across every clone.

How to defend against it

The defenses are concrete and they stack. None of them require new cryptography.

• Never reuse a nonce under a key. Treat that as a hard invariant. Everything below serves this one rule.
• Prefer a deterministic counter over randomness. A 96 bit nonce made of a per message counter that strictly increases and never wraps avoids birthday collisions entirely. Persist the counter so a reboot cannot rewind it.
• If you must use random nonces, cap messages per key. Stay well under the birthday bound, then rotate.
• Rotate keys often. A fresh key resets the whole nonce space and limits how much data a recovered H exposes.
• Use a misuse resistant scheme when reuse is plausible. AES GCM SIV derives its internal value from both the nonce and the message, so a repeated nonce leaks only whether two identical plaintexts were sent, never the authentication key. It is the right default for distributed encryptors that cannot perfectly coordinate.
• Coordinate nonces across machines. If many encryptors share a key, give each a distinct nonce prefix so their ranges cannot overlap, or give each its own key.
• Generate nonces from a real CSPRNG. A weak or seeded generator produces predictable or repeating values. Use the platform secure random source.

Notice the shape of this bug. It is not a flaw in AES. It is a broken assumption about how the mode is used, that every nonce is unique. The same pattern shows up across cryptography, where the math is sound but a usage rule gets quietly violated. A padding oracle attack turns a tiny error message into full plaintext recovery, and JWT algorithm confusion tricks a verifier into trusting the wrong key. In every case the primitive is fine and the integration around it breaks.

The assumption that breaks

AES GCM nonce reuse is a clean example of a security property that depends entirely on how the system is built around the cipher. The encryption is strong and the mode is strong. The failure lives in a counter that resets, a key copied to too many places, or a random nonce drawn one too many times. Those assumptions get missed in code review because the library call looks correct on its own line. This is the kind of bug you find by asking what a system assumes and then checking whether anything can make that assumption false. An early signal we find encouraging: a frontier model drove the full methodology on its own and identified and verified real access control and injection issues in test applications it had not seen before. That is what an autonomous researcher is built to do. Read more on our about page.

Frequently asked questions

What is AES GCM nonce reuse?

It is the mistake of encrypting two or more messages with the same key and the same nonce in AES GCM. GCM needs a unique nonce per key because the nonce determines both the counter mode keystream and the value mixed into the authentication tag. Reuse it and the keystream repeats, which lets an attacker XOR two ciphertexts to cancel the mask and recover plaintext relationships, and it also exposes the GHASH authentication subkey. It is a usage error, not a flaw in AES itself.

What is the forbidden attack on AES GCM?

The forbidden attack is the integrity failure that follows a repeated nonce. The GCM tag is a polynomial in a secret value H, plus a mask that depends only on the nonce. When two messages share a nonce, that mask is identical, so subtracting the two tag equations cancels it and leaves a polynomial in the single unknown H. The attacker knows the ciphertexts and tags, so they solve for the roots in the finite field and recover H. With H known, they can forge valid tags for arbitrary messages the receiver will accept.

How do nonces get reused by accident?

Several ordinary ways. Random 96 bit nonces collide after roughly two to the 48th messages under one key because of the birthday bound, which a busy fleet can reach. Counter based nonces stored only in memory reset to zero after a crash or reboot and re emit old values. Sharing one key across many machines or cloning a virtual machine that already holds key and counter state lets two encryptors pick the same nonce. A weak random source can also repeat or predict nonces.

How do you prevent AES GCM nonce reuse?

Treat a unique nonce per key as a hard rule. Prefer a strictly increasing counter that is persisted so a reboot cannot rewind it. If you use random nonces, cap messages per key well under the birthday bound and rotate keys often. Give each encryptor a distinct nonce prefix or its own key so distributed machines cannot overlap. When reuse is plausible, use a misuse resistant scheme like AES GCM SIV, which limits the damage of a repeated nonce to leaking only whether two plaintexts were identical, never the authentication key.

Originally published at https://security.unboundcompute.com on June 25, 2026.