Post cover image

July 17, 2026

A 65,529-Sample Lie: How a Tiny TIFF Fooled GDAL Into Asking for 16 GB

A malformed SamplesPerPixel tag reached GDAL’s GTiff driver, triggered a 16.4 GB allocation request, and led to a codec-aware upstream fix

By Aleens-labs

6 min read

Three kilobytes. That is all it took. A TIFF file just 3,155 bytes long convinced GDAL to reach for more than 16 billion bytes of memory — roughly five million times its own size.

The file did not contain a 16 GB image payload. It contained a small set of TIFF metadata values, including an abnormal SamplesPerPixel value of 0xFFF9, or 65,529 samples per pixel. When GDAL's GTiff driver processed that metadata through its TIFF decoding path, the calculated uncompressed buffer size became approximately 16.4 GB.

The first mitigation I proposed was a fixed 256 MB allocation ceiling. That pull request was later superseded by a more precise, codec-aware design from the GDAL and libtiff maintainer. The final fix was merged into GDAL's internal libtiff implementation.

This article explains the finding, the evidence, why the first patch was not the final answer, and how the upstream solution evolved.

Quick Summary

Project: GDAL Component: GTiff driver and GDAL's internal libtiff Tested version: GDAL 3.8.4 Input size: 3,155 bytes Malformed value: SamplesPerPixel = 0xFFF9(65,529) Allocation requested: 16,382,250,000 bytes, approximately 16.4 GB Observed impact: TIFF processing failure and denial of service through disproportionate memory allocation Initial proposal: GDAL pull request #14400 Final upstream fix: GDAL pull request #14416 and commit deaeb11 Related identifier: CVE-2026-36849 for the underlying libtiff issue RCE status: Not demonstrated or claimed

The vulnerable behavior can be summarized as follows:

small TIFF input
-> oversized SamplesPerPixel metadata
-> enormous calculated decoded-strip size
-> approximately 16.4 GB allocation request
-> processing failure / denial of service

The 3 KB Input

The proof-of-concept TIFF used these relevant properties:

File size: 3,155 bytes
Image dimensions: 500 x 500
BitsPerSample: 8
SamplesPerPixel: 65,529
SHA-256: 396cd4b7ccaf1c0d769b42456f78a2c6b5786cd694ad15e7499faa4157d6a6d6

The critical size calculation was:

500 x 500 x 65,529 x 1 byte
= 16,382,250,000 bytes
= approximately 16.4 GB

This is a classic resource-asymmetry problem: a tiny attacker-controlled input influences metadata that expands into a vastly larger memory requirement.

Reproducing the GDAL Behavior

The test was performed with GDAL 3.8.4 on Ubuntu 24.04 under WSL2. The tested environment used the system libtiff 4.5.1.

The behavior was reproduced with gdal_translate:

gdal_translate crash.tiff /tmp/output.tif
echo "Exit: $?"

The observed output was:

ERROR 2: ./frmts/gtiff/gtiffdataset_read.cpp, 3264:
cannot allocate 1x16382250000 bytes

ERROR 1: crash.tiff, band 1:
IReadBlock failed at X offset 0 and Y offset 0

real 0m0.150s
Exit: 1

The most important evidence is the requested allocation size: 16,382,250,000 bytes. The operation failed in approximately 150 milliseconds.

I describe this carefully as a processing failure and denial-of-service condition. The output proves that GDAL attempted the disproportionate allocation and could not complete the requested conversion. It does not, by itself, prove a segmentation fault, memory corruption, or remote code execution.

Where GDAL Entered the Picture

GDAL's GTiff driver relies on libtiff functionality. At the time of testing, the TIFF metadata could reach buffer-size calculations without an effective guard against this particular compressed-to-uncompressed size mismatch.

The problem was not simply that large TIFF files exist. Legitimate geospatial images can be enormous, and a universal low memory ceiling would break valid workloads. The security problem was that a very small compressed input could claim metadata that implied an implausibly large decoded buffer.

That distinction became important during patch review.

The First Patch: A 256 MB Ceiling

I opened GDAL pull request #14400 with a narrowly scoped mitigation. It added TIFFOpenOptionsSetMaxSingleMemAlloc() with a 256 MB limit in VSI_TIFFSetOpenOptions().

The goal was straightforward:

prevent a single malformed TIFF operation
from requesting an unreasonably large allocation

The proposal also highlighted that GDAL already used a cumulative allocation limit but did not have the corresponding single-allocation guard in this path.

The pull request was closed on April 21, 2026, with the note that it had been superseded by pull request #14416. This is an important distinction: #14400was not the final patch, but the underlying problem was addressed through a different implementation.

Why a Fixed Limit Was Not the Best Final Design

A fixed 256 MB ceiling is easy to understand, but GDAL processes legitimate datasets that may require allocations above that size. A hardcoded threshold can stop the malicious case while also rejecting valid high-resolution imagery.

The better question is not simply:

Is this allocation larger than 256 MB?

It is:

Is the claimed compressed size realistic for the expected decoded size
and the active compression codec?

That leads to a more contextual defense. Different TIFF codecs have different maximum practical compression ratios, and the ratio may depend on tile size, strip size, sample count, and codec configuration.

The Codec-Aware Fix

GDAL pull request #14416 imported the TIFFGetMaxCompressionRatio() approach into GDAL's internal libtiff and used it in the encoded strip and tile allocation paths.

Conceptually, the new logic allows the decoder to ask the active codec for a realistic maximum compression ratio. The allocation path can then reject a strip or tile whose compressed size is not credible relative to the expected uncompressed size.

The implementation added codec-specific ratio handling across multiple files, including support for compression schemes such as Fax, JPEG, LERC, LZMA, LZW, PackBits, WebP, ZIP, and others. Codecs that cannot provide a reliable maximum can report that the ratio is unknown.

This approach is more precise than a universal memory ceiling:

  • It targets implausible compressed-to-uncompressed expansion.
  • It preserves legitimate large-image workflows where the encoded data is credible.
  • It accounts for differences between compression codecs.
  • It protects the allocation path before the oversized buffer request occurs.

Pull request #14416 was merged into GDAL master on May 4, 2026, as commit deaeb11.

The same functionality is also listed in the libtiff 4.7.2 release notes, where TIFFGetMaxCompressionRatio() is used by _TIFFReadEncodedTileAndAllocBuffer() and _TIFFReadEncodedStripAndAllocBuffer().

Security Impact

The confirmed impact is uncontrolled resource consumption leading to TIFF-processing failure. A service that automatically processes attacker-controlled TIFF files through GDAL may expose this behavior through an upload endpoint, geospatial data pipeline, remote-sensing ingestion system, conversion service, or thumbnail generator.

Whether the attack vector is actually remote depends on the application embedding GDAL. GDAL itself is a library and command-line toolkit; it does not automatically turn every malformed file into a network attack.

For an exposed service, the practical consequences may include:

  • failure of the current conversion or analysis job;
  • worker-process termination under restrictive memory policies;
  • memory pressure in long-running image-processing services;
  • repeated resource exhaustion when the same input is automatically retried;
  • reduced availability of a geospatial processing pipeline.

No confidentiality or integrity impact was demonstrated. No out-of-bounds access, instruction-pointer control, or code execution was observed in this GDAL finding.

Upstream Remediation Timeline

The remediation evolved publicly through the GDAL and libtiff projects. The initial proposal introduced a simple per-allocation ceiling, while the final implementation replaced that fixed threshold with codec-aware validation:

April 20, 2026  Initial GDAL mitigation opened as PR #14400
April 21, 2026  PR #14400 closed as superseded by PR #14416
May 4, 2026     PR #14416 merged into GDAL as commit deaeb11

This progression matters because the first patch documented and blocked the immediate oversized-allocation behavior, while the superseding patch addressed the deeper invariant: whether the compressed strip or tile size is realistic for the decoded size and active codec. The important public outcome is that GDAL received a more precise, codec-aware mitigation without imposing a universal low memory ceiling on legitimate datasets.April 21, 2026 PR #14400 closed as superseded by PR #14416 May 4, 2026 PR #14416 merged into GDAL as commit deaeb11 July 14, 2026 Google OSS VRP report closed

I do not infer a technical severity conclusion from the word "closed" alone. The important public result is that the allocation path received an upstream, codec-aware mitigation.

What I Learned from the Patch Review

The first correct observation is not always the final correct design.

My initial proposal focused on the immediate symptom: one allocation was far too large. The maintainer's solution focused on the deeper invariant: compressed data should not claim a decoded size beyond what its codec can realistically produce.

That is a better engineering outcome. It addresses the malicious case without imposing an arbitrary global ceiling on legitimate geospatial workloads.

For vulnerability researchers, there are several useful lessons:

  • Reproduce the exact allocation size and code path.
  • Separate an operation failure from a process crash.
  • Do not convert a library finding into a universal network-severity claim.
  • Treat an initial patch as a proposal, not as proof that the design is final.
  • Follow superseding pull requests and record the fix that actually lands.
  • Describe confirmed denial of service without inflating it into memory corruption or RCE.

Disclosure Status

The GDAL mitigation is public and merged. The initial pull request #14400 remains useful as the original report and proposed guard, while pull request #14416 and commit deaeb11 represent the final upstream implementation.

The related libtiff functionality is included in libtiff 4.7.2.

References

Closing Thoughts

A 3 KB TIFF did not need to contain 16 GB of pixel data. It only needed metadata that convinced the decoder to ask for a 16.4 GB buffer.

The final fix did more than block one oversized number. It gave the allocation path a way to reason about what each codec could realistically produce.

That is the strongest outcome of coordinated open-source vulnerability research: not merely stopping one proof of concept, but improving the invariant that protects every caller of the affected code.