Jetty.project: GzipHandler invalid input zip size on large (over 2,147,483,647 bytes) request body content

[Auo]

Jetty version
9.4.24.v20191120

Java version
openjdk 11.0.3 2019-04-16
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.3+7)

OS type/version
macOS Catalina
version 10.15.1

Description
GZipHandler will fail to inflate streams that are over 4GB, the error originate in GZipContentDecoder

...
                    case ISIZE:
                    {
                        _value += (currByte & 0xFF) << 8 * _size;
                        ++_size;
                        if (_size == 4)
                        {
                            if (_value != _inflater.getBytesWritten())
                                throw new ZipException("Invalid input size");
...

How to replicate
Setup Jetty with GZipHandler and specify setInflateBufferSize so Jetty will try and inflate your request.

Create dummy data:

dd if=/dev/urandom of=too_big.xml bs=1048576 count=4097

curl -X POST -H "Content-Encoding: gzip" -T too_big.xml.gz "http://localhost:port/"

Will throw: java.util.zip.ZipException: Invalid input size

Possible solutions
Don't validate the size if a certain amount of bytes have been written OR cast _inflater.getBytesWritten() to int

[joakime]

It's not that it doesn't validate, we are using int to track sizes.
Which would limit the file size to 2,147,483,647 (roughly 2GB).

We need to update that code to use long

[joakime]

I have been able to replicate.

There are 2 problems, one in the ISIZE calculation (need to use long to track value), and one in the ISIZE validation (need to honor modulo 2^32 evaluation rule in RFC 1952: Section 2.3.1).

If I only correct for the modulo 2^32 validation check, there's still some issues pointing to the fact that _value is an int.

Integer.MAX_VALUE = 2,147,483,647

  ISIZE = 0xff | _value = 255 (0xff)
  ISIZE = 0xff | _value = 65,535 (0xffff)
  ISIZE = 0xff | _value = 16,777,215 (0xffffff)
  ISIZE = 0x7f | _value = 2,147,483,647 (0x7fffffff)
  2,147,483,647 != (2,147,483,647 & ffffffff) [2,147,483,647]
  (_value != (inflater.getBytesWritten() & UINT_MAX) [result of modulo])


Integer.MAX_VALUE + 1 = 2,147,483,648

  ISIZE = 0x0 | _value = 0 (0x0)
  ISIZE = 0x0 | _value = 0 (0x0)
  ISIZE = 0x0 | _value = 0 (0x0)
  ISIZE = 0x80 | _value = -2,147,483,648 (0x80000000)
  -2,147,483,648 != (2,147,483,648 & ffffffff) [2,147,483,648]
  (_value != (inflater.getBytesWritten() & UINT_MAX) [result of modulo])


UINT_MAX -1 = 4,294,967,294

  ISIZE = 0xfe | _value = 254 (0xfe)
  ISIZE = 0xff | _value = 65,534 (0xfffe)
  ISIZE = 0xff | _value = 16,777,214 (0xfffffe)
  ISIZE = 0xff | _value = -2 (0xfffffffe)
  -2 != (4,294,967,294 & ffffffff) [4,294,967,294]
  (_value != (inflater.getBytesWritten() & UINT_MAX) [result of modulo])


UINT_MAX = 4,294,967,295

  ISIZE = 0xff | _value = 255 (0xff)
  ISIZE = 0xff | _value = 65,535 (0xffff)
  ISIZE = 0xff | _value = 16,777,215 (0xffffff)
  ISIZE = 0xff | _value = -1 (0xffffffff)
  -1 != (4,294,967,295 & ffffffff) [4,294,967,295]
  (_value != (inflater.getBytesWritten() & UINT_MAX) [result of modulo])


UINT_MAX + 1 = 4,294,967,296

  ISIZE = 0x0 | _value = 0 (0x0)  
  ISIZE = 0x0 | _value = 0 (0x0)
  ISIZE = 0x0 | _value = 0 (0x0)
  ISIZE = 0x0 | _value = 0 (0x0)
  0 != (4,294,967,296 & ffffffff) [0]
  (_value != (inflater.getBytesWritten() & UINT_MAX) [result of modulo])

[joakime]

I've addressed the bitshifting of long and i am preparing a PR.

[joakime]

Opened PR #4417