AES-GCM is widely used mode of operation of the AES symmetric cipher. AES is considered by prominent cryptographers as the gold standard of a block cipher, being probably as safe as this kind of algorithm can get and the GCM is one of the main modes of operations used to provide authentication through an AEAD scheme considered very secure.

AES-GCM Encryption

AES operates on fixed 128-bit blocks. To encrypt data of arbitrary length, AES-GCM uses counter mode (CTR) to turn the block cipher into a stream cipher.

Given a key, a nonce, and a plaintext, the process starts by deriving an initial counter block J0​ from the nonce. In the standard case where the nonce is 96 bits (12 bytes), J0 is formed by appending a fixed 32-bit value to the nonce.

From J0​, a sequence of counter blocks J1,J2,…,Jn is generated by incrementing the last 32 bits. Each counter block is then encrypted with AES under the given key to produce a sequence of keystream blocks:

The ciphertext is obtained XORing each plaintext block with the corresponding keystream block.

The overall algorithm looks like this:

This will accomplish the confidentiality part of this mode of operation.

AES-GCM Authentication and Integrity

The AES-GCM also provides authentication purposes with the use hashing over a binary Galois field. This works by producing a tag assures integrity and authentication on the ciphertext and AAD ( Additional Authenticated Data).

First a secret hash key is derived from the AES key.

Then the hash function GHASH is used to compute a value S from the AAD,ciphertexts and lengths.

This function works through a polynomial muiltiplication on the Galois field.

The final tag is then produced using the J0 value mentioned before

Nonce reuse

Nonce reuse is perhaps the most classical mistake in AES implementations in general consisting of reusing the number meant to be used at once as the acronym suggests. The GCM mode is also sensitive to this kind of attack since the mode depends upon a nonce in order to not have trivial attacks involving XOR properties.

The whole attack idea is based on the fact that if the same key and nonce are used twice:

Because the nonce is the same the J0 will also be the same and so the sequence and the keystream.

The confidentiality property breaks in this case taking advantage of the use of CTR mode and some properties of XOR operation.

Even though the attacker would not be able to extract the plaintext of the message, just the XOR version. If the attacker knows certain cribs like the header of a message or other information, the unknown parts of the message can be extracted.

For the integrity part, the collapse is related to the mentioned use of the GHASH which, for performance's sake is a linear function which will produce this form of tag for all the messages.

In the case of nonce reuse the attacker will see the same J0 and AES_k(J0) and so

Since GHASH is linear :

This is catastrophic: with enough messages, an attacker can apply linear algebra to recover the authentication key and forge valid tags for arbitrary ciphertexts. Because the field is fixed to GF(2¹²⁸) and GHASH is a low-degree polynomial, even a relatively small number of nonce reuses can be sufficient. In realistic scenarios such as traffic analysis, where parts of the plaintext (like headers) are known dozens or even fewer reused nonces may be enough to break authentication.

AAD misunderstanding

AAD is meant to contain important metadata about the sender party and so if the developer misunderstands which data should or should not be added to the tag, the results can be disastrous even creating privilege escalation.

Let's say, a developer is building an API with the following request format

request = {
    "user_id": 65537,
    "role": "user",
    "ciphertext": ENC_K(nonce, {"action": "transfer", "amount": 10000}),
    "tag": ---
}

An attacker who captures this request would see the user_id, role, ciphertext and the tag. The attacker could simply change the role to admin and keep all the other fields the same and the tag would be still valid since those are not included in the AAD. This will result in a privilege escalation attack.

Conclusion

Even though AES-GCM design is solid and quite fast, there are certain considerations that developer should be aware of when implementing this algorithm, as the attacks for this kind of algorithm generally do not occur at the mathematical level , but on some subtle implementation choices that may look not important on a first glance like the deliberate or accidental reuse of nonce or a miscalculation of what should be or not in the AAD.