This writeup demonstrates a nonce reuse vulnerability in an insecure AES-GCM implementation, where using the same key and nonce across multiple messages causes the keystream to be reused. By exploiting this flaw and knowing part of one plaintext, an attacker can recover the keystream and use it to decrypt other ciphertexts produced with that same key and nonce. The writeup was inspired by my participation in the CyberQ CTF, which focused on cryptography and AI challenges.

Terminology

  • Counter: a value that increments for each block of plaintext.
  • Nonce: a unique value used in cryptographic operations.
  • Keystream: the output from the encrypted nonce and counter, which is XORed with the plaintext.
  • GMAC (Galois Message Authentication Code): a MAC based on Galois field operations for message authentication.
  • GHASH: the hashing function used within GMAC to compute the authentication tag.
  • Additional Authenticated Data (AAD): optional input data that is authenticated but not encrypted, typically packet headers or metadata.
  • Authentication Tag: a short value generated during encryption that verifies both the integrity and authenticity of the data and AAD.
  • Hash Subkey (H): a 128-bit value obtained by encrypting an all zero block with AES using the secret key, essential in computing the authentication tag.

AES-GCM Overview

AES-GCM is one of the most widely used encryption modes in modern cryptography. It combines the AES algorithm with Galois/Counter Mode (GCM) to provide both confidentiality and integrity in a single operation. GCM uses counter mode to generate a unique keystream for encrypting plaintext blocks, ensuring ciphertext uniqueness for each combination of key and nonce. This uniqueness prevents common attacks such as pattern analysis that are possible in simpler modes like Electronic Codebook (ECB), where identical plaintext blocks produce identical ciphertexts. AES-GCM also incorporates GMAC to authenticate both the ciphertext and any additional authenticated data through the GHASH function.

The diagrams below illustrate the AES-GCM encryption process. The keystream is generated by encrypting a combination of a nonce and an incrementing counter using the private key, and each plaintext block is then XORed with this keystream to produce the ciphertext.

None
Figure 1 — AES-GCM encryption flow

Message Authentication Process

Note: This section is not essential to recovering the keystream. You may skip it if you're not interested in the AES-GCM authentication process.

In message authentication, the first counter block is used to generate the initial part of the keystream, which is reserved for message authentication. The additional authenticated data are processed in the first GHASH operation, then each ciphertext block is XORed with the previous GHASH result. Each GHASH step also involves multiplication with the hash subkey H. Once all ciphertext blocks have been processed, the lengths of the additional data and ciphertext are XORed in. The final GHASH result is then XORed with the first counter block to generate the authentication tag, confirming that the data has not been modified and originates from a trusted source.

None
Figure 2 — Encryption & authentication tag generation

Root cause & Impact of Nonce Reuse in AES-GCM

Nonce reuse in AES-GCM is a serious security issue because of how the mode uses XOR during encryption. When the same nonce is reused with the same key, AES generates the same keystream for any message that reuses that nonce. Since ciphertext blocks are created by XORing the plaintext with this keystream, an attacker can XOR two ciphertexts together to cancel out the keystream, revealing the XOR of the two plaintexts. This result doesn't directly show the messages, but it exposes their bitwise relationship, where bits that are the same appear as 0s, and bits that differ appear as 1s. In other words, it shows how the two plaintexts differ at every position. Even worse, if one plaintext block is known it can reveal all other plaintext blocks in the same position across messages that reused the nonce.

Note: The amount of keystream that can be recovered depends on how much plaintext is known.

The illustration below shows four messages that are sent using the same nonce and secret key. A malicious actor who knows the second block plaintext of message 2, can easily recover the keystream for that block, which lets the actor decrypt the second block of all three other messages, compromising confidentiality for those blocks.

None
Figure 3 — Nonce reuse leading to keystream recovery

The CyberQ Challenge Setup

The CyberQ CTF challenge called "Groudhog Day"provides two ciphertexts and a structured key-value plaintext containing only the keys. The second plaintext (not provided) contains the corresponding values placed at the exact same offsets as the keys in the first plaintext. The goal of the challenge is to recover the full keystream used to encrypt the messages so the plaintexts can be reconstructed.

#Provided Data

ciphertext1: 
"ee16c4b4a5ba5937af8d513741371c43a7cd9951ce68a8414c5e4e6c0895f30f0933f7bfd26575704b26f9428478d970237d09939893b989b553a5af9589858aa349c1372dbee5f6cfda3b4d9800b5b2402989ca5ff49b78d86a5488990a807b7f9f4712a8b2af5ecae941e01903c7f8575d8afe1319850f6beea979fbeb9713375f2ad3468b85ac8d42c729cb13"


ciphertext2: 
"b53cc4b4a7a14c2cbe944c355b371e0be697dc1d806aa441665e4e6e158ef8050131f6bdc865773d0262b503ca3cdb7c235709939a9fa889b749a5add8c8d588af49eb372dbcf2fdf1c3325a9a1ab5b01360c59c1aa69974d84054889b1d9b7774925714aaa8af5c89bb00ae5b4695fa5b5da0fe131b900965f2bb7df9e39509375d798707d9c3edc10ec5039639"


plaintext1:
{
"status": "??????",
"profile": "???????",
"app": "???",
"dbRole": "??????",
"timeout": "???????",
"frontend": "????????",
}

Keystream Recovery Process

Since the values in plaintext2 exist at the same offsets as the keys in plaintext1, a partial keystream can be generated to decrypt the key values from plaintext2. The keystream is considered partially correct because the provided plaintext is incomplete, it only contains the keys and the values are missing. As a result, the generated keystream is only accurate at positions where the plaintext matches the original message. By using the recovered values to complete plaintext1, the full and correct keystream can then be obtained.

Step 1: Partial Keystream Extraction

The following script converts plaintext1 and ciphertext1 into byte representation and XORs them while also ensuring both have the same length, to get the partial keystream.

#keystream1.py

import string
import sys


c1_hex = "ee16c4b4a5ba5937af8d513741371c43a7cd9951ce68a8414c5e4e6c0895f30f0933f7bfd26575704b26f9428478d970237d09939893b989b553a5af9589858aa349c1372dbee5f6cfda3b4d9800b5b2402989ca5ff49b78d86a5488990a807b7f9f4712a8b2af5ecae941e01903c7f8575d8afe1319850f6beea979fbeb9713375f2ad3468b85ac8d42c729cb13"

# Directly define the plaintext structure instead of building it
p1_text = """{
 "status": "??????",
 "profile": "???????",
 "app": "???",
 "dbRole": "??????",
 "timeout": "???????",
 "frontend": "????????"
}"""

# Turn the plaintext into byte representation
p1_bytes = p1_text.encode("utf-8")

# XOR the incomplete plaintext with the ciphertext to get a partially correct keystream (both in byte representation)
def xor_bytes(a, b):
 return bytes(x ^ y for x, y in zip(a, b))

# Turn the cipher hex into byte representation
try:
 c1 = bytes.fromhex(c1_hex)
except Exception as e:
 print("Error parsing c1_hex:", e)
 sys.exit(1)

# L sets both inputs to the same length so they can be XORed safely
L = min(len(c1), len(p1_bytes))
keystream_segment = xor_bytes(c1[:L], p1_bytes[:L])

#print
print("Keystream hex value:(not fully accurate yet)")
print(keystream_segment.hex())
#keystream1.py output

user@user-ubuntu:~/Desktop/rsa-gcm-nr$ python3 keystream1.py 
Keystream hex value:(not fully accurate yet)
951ce49487c92d56dbf822157b173e7c98f2a66ef14a8461467e6e4e78e79c69605f929de845574f7419c67dbb47fb5c037729b3baf2c9f99769858daab6baa88f69cb170d9c81949db55728ba3a95907f16b6f560cbb954f86074a8bb7ee9161af032668a888f7cf5d67edf263cf8da7b7d80de333be37d0480dd1c958fb529177d15ec79b4ba93b27de523b6

Step 2: Value Recovery

Once the partial keystream is obtained, it is XORed with ciphertext2 to reveal the plaintext values.

# values.py

# XOR the second ciphertext with the keystream retrieved previously (both in byte representation)
def xor_bytes(a: bytes, b: bytes) -> bytes:
 return bytes(x ^ y for x, y in zip(a, b))

c2_hex = "b53cc4b4a7a14c2cbe944c355b371e0be697dc1d806aa441665e4e6e158ef8050131f6bdc865773d0262b503ca3cdb7c235709939a9fa889b749a5add8c8d588af49eb372dbcf2fdf1c3325a9a1ab5b01360c59c1aa69974d84054889b1d9b7774925714aaa8af5c89bb00ae5b4695fa5b5da0fe131b900965f2bb7df9e39509375d798707d9c3edc10ec5039639"

keystream_hex = "951ce49487c92d56dbf822157b173e7c98f2a66ef14a8461467e6e4e78e79c69605f929de845574f7419c67dbb47fb5c037729b3baf2c9f99769858daab6baa88f69cb170d9c81949db55728ba3a95907f16b6f560cbb954f86074a8bb7ee9161af032668a888f7cf5d67edf263cf8da7b7d80de333be37d0480dd1c958fb529177d15ec79b4ba93b27de523b6"

# Turn both values into byte representations
c2 = bytes.fromhex(c2_hex)
ks = bytes.fromhex(keystream_hex)

# L sets both inputs to the same length so they can be XORed safely
L = min(len(c2), len(ks))
recovered = xor_bytes(c2[:L], ks[:L])

# print
print("Recovered P2 in hex:")
print(recovered.hex())
print()
print("Recovered P2 (utf-8, prefix; non-decodable bytes replaced):")
print(recovered.decode("utf-8", errors="replace"))
# values.py output

user@user-ubuntu:~/Desktop/rsa-gcm-nr$ python3 values.py
Recovered P2 in hex:
202020202068617a656c6e20202020777e657a7371202020202020206d69646c616e642020202072767b737e717b202020202020206d617020202020727e6f2020202020202073696c766572202020206c7673697a6d202020202020206372616e626572202020207c6d7e717d7a6d202020202020207374617266616c6c202020206c6b7e6d797e7373202020

Recovered P2 (utf-8, prefix; non-decodable bytes replaced):
     hazeln    w~ezsq       midland    rv{s~q{       map    r~o       silver    lvsizm       cranber    |m~q}zm       starfall    lk~my~ss

Step 3: Full Keystream Recovery

After the values are recovered by values.py, the plaintext in keystream1.py is updated to produce the full keystream.

# keystream2.py

import string
import sys

c1_hex = "ee16c4b4a5ba5937af8d513741371c43a7cd9951ce68a8414c5e4e6c0895f30f0933f7bfd26575704b26f9428478d970237d09939893b989b553a5af9589858aa349c1372dbee5f6cfda3b4d9800b5b2402989ca5ff49b78d86a5488990a807b7f9f4712a8b2af5ecae941e01903c7f8575d8afe1319850f6beea979fbeb9713375f2ad3468b85ac8d42c729cb13"

# Directly define the plaintext structure instead of building it
p1_text = """{
"status": "hazeln",
"profile": "midland",
"app": "map",
"dbRole": "silver",
"timeout": "cranber",
"frontend": "starfall"
}"""

# Turn the plaintext into byte representation
p1_bytes = p1_text.encode("utf-8")

# XOR the incomplete plaintext with the ciphertext to get a partially correct keystream (both in byte representation)

def xor_bytes(a, b):
 return bytes(x ^ y for x, y in zip(a, b))

# Turn the cipher hex into byte representation
try:
 c1 = bytes.fromhex(c1_hex)
except Exception as e:
 print("Error parsing c1_hex:", e)
 sys.exit(1)

# L sets both inputs to the same length so they can be XORed safely
L = min(len(c1), len(p1_bytes))
keystream_segment = xor_bytes(c1[:L], p1_bytes[:L])

# print
print("Keystream hex value:")
print(keystream_segment.hex())
#keystream2.py output

user@user-ubuntu:~/Desktop/rsa-gcm-nr$ python3 keystream2.py 
Keystream hex value:
951ce49487c92d56dbf822157b173e2bc6b7fc3da04a8461467e6e4e78e79c69605f929de845571d22429523ea1cfb5c037729b3baf2c9f99769858df8e8f5a88f69cb170d9c81949db55728ba3a95903340e5bc3a86b954f86074a8bb7ee9161af032668a888f7ca99b208e7b66b5da7b7d80de333be37d0480dd1c958fb529177d59a727f9e3cde12ee523b6

The keystream has been recovered which would allow a malicious actor to decrypt all messages using the same nonce and private key.

Note: this setup is for demonstration. While a perfectly aligned second plaintext is uncommon in the wild, even a small known fragment of one plaintext lets an attacker recover the corresponding bytes of every message encrypted with the same keystream.

Conclusion

This challenge demonstrates how dangerous nonce reuse can be in AES-GCM. When the same key and nonce are reused, the entire encryption process breaks down, allowing attackers to recover plaintexts even if they know only part of one message. It is essential to ensure that every operation uses a unique nonce to keep your data safe and your encryption reliable.

Huge thanks to CENSUS and Dr. Tasos Keliris for making this writeup possible.

If you have any questions or feedback, please don't hesitate to reach out via LinkedIn.

References