Ktor: httpclient gzip decoding failed

[rockyou]

Ktor Version

1.2.1

Ktor Engine Used(client or server and name)

Http Client with Apache engine

JVM Version, Operating System and Relevant Context

JDK 11,MAC OS

Feedback

If ContentEncoding feature turned on

    install(ContentEncoding) {
                gzip()
                identity()
            }

and get content from any website which supports compression

An exception thrown:

java.lang.IllegalStateException: Gzip padding invalid.

[e5l]

Hi @rockyou, can you provide the request example? The request log also would be nice.

[rockyou]

Hi! Thanks so much for quick response.

Test code:

    runBlocking {
        val client = HttpClient(Apache) {

            install(ContentEncoding) {
//                gzip()
                identity()
            }
        }
        client.call("http://mockbin.com/request") {
            method = HttpMethod.Post
        }.response.use { println(it.readText()) }

    }

when gzip-encoding is disabled,the request result is correct.
when gzip-encoding is enabled,exception is thrown:

Exception in thread "pool-1-thread-3" java.lang.IllegalStateException: Gzip padding invalid.
    at io.ktor.util.EncodersJvmKt$inflate$1.invokeSuspend(EncodersJvm.kt:55)
    at io.ktor.util.EncodersJvmKt$inflate$1.invoke(EncodersJvm.kt)
    at kotlinx.coroutines.io.CoroutinesKt$launchChannel$job$1.invokeSuspend(Coroutines.kt:123)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
    at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:238)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

[luishiga]

In ktor/ktor-utils/jvm/src/io/ktor/utilEncoders.jvm the check padding should be only size?
line 55:
check(padding.contentEquals(GZIP_HEADER_PADDING)) { "Gzip padding invalid." }

change the condition to something like
check(padding.size == GZIP_HEADER_PADDING.size) { "Gzip padding invalid." }

[cy6erGn0m]

It looks like the server doesn't support compression:

# curl -H 'Accept-Encoding: gzip,deflate' -D - "http://mockbin.com/request" | head -n 25
Server: openresty/1.13.6.2
Date: Thu, 05 Sep 2019 12:55:33 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 975
Connection: keep-alive
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET
Access-Control-Allow-Headers: host,connection,x-forwarded-for,x-forwarded-proto,x-forwarded-host,x-forwarded-port,x-real-ip,kong-cloud-request-id,kong-client-id,user-agent,accept,accept-encoding,x-request-id,via,connect-time,x-request-start,total-route-time
Access-Control-Allow-Credentials: true
X-Powered-By: mockbin
Vary: Accept, Accept-Encoding
Etag: W/"3cf-2Py4h7YOgGd9URbRiiAuyUyhJFQ"
Via: kong/0.36-1-enterprise-edition
X-Kong-Upstream-Status: 200
X-Kong-Upstream-Latency: 5
X-Kong-Proxy-Latency: 0
Kong-Cloud-Request-ID: 6c5a42489ea1baf2b5c2298877a45df9

{
  "startedDateTime": "2019-09-05T12:55:33.237Z",
  "clientIPAddress": "81.3.129.2",
  "method": "GET",
  "url": "http://mockbin.com/request",
  "httpVersion": "HTTP/1.1",

[Nohus]

This issue also affects me, for servers that definitely support compression, for example requesting https://www.google.com also fails with java.lang.IllegalStateException: Gzip padding invalid.

[Nohus]

Here is a self contained, runnable example:

suspend fun main() {
    HttpClient(Apache) {
        ContentEncoding {
            gzip()
        }
    }.get<String>("https://ktor.io/")
}

It results in a java.lang.IllegalStateException: Gzip padding invalid.
If you replace gzip() with deflate(), it works fine.

[pecanw]

Any progress guys?

[pecanw]

I'm neither kotlin nor gzip expert, but I have downloaded the source code, and found that

internal val GZIP_HEADER_PADDING: ByteArray = ByteArray(7)

is never written to, so it contains just seven zero bytes, but then it is compared to the padding read from gzip header:

    if (gzip) {
        val header = source.readPacket(GZIP_HEADER_SIZE)
        val magic = header.readShortLittleEndian()
        val format = header.readByte()
        val padding = header.readBytes()

        check(magic == GZIP_MAGIC) { "GZIP magic invalid: $magic" }
        check(format.toInt() == Deflater.DEFLATED) { "Deflater method unsupported: $format." }
        check(padding.contentEquals(GZIP_HEADER_PADDING)) { "Gzip padding invalid." }
    }

According to the gzip specification:

Offset   Length   Contents
  0      2 bytes  magic header  0x1f, 0x8b (\037 \213)  
  2      1 byte   compression method
                     0: store (copied)
                     1: compress
                     2: pack
                     3: lzh
                     4..7: reserved
                     8: deflate
  3      1 byte   flags
                     bit 0 set: file probably ascii text
                     bit 1 set: continuation of multi-part gzip file, part number present
                     bit 2 set: extra field present
                     bit 3 set: original file name present
                     bit 4 set: file comment present
                     bit 5 set: file is encrypted, encryption header present
                     bit 6,7:   reserved
  4      4 bytes  file modification time in Unix format
  8      1 byte   extra flags (depend on compression method)
  9      1 byte   OS type

after the magic header and compression method there are additional 7 bytes which are not supposed to be zero.

So we should either use @luishiga 's proposed solution to check just the size (which dos not make much sense as we always read private const val GZIP_HEADER_SIZE: Int = 10 bytes as header), or remove the check completely.

However, there might be some extra header present. In this case we have to read some more bytes before getting the body. If nobody gets involved I'll try to do some more research and try to fix the code...

[InsanusMokrassar]

This problem represented for version 1.3.0 too. So, is there some news about fixes of that?

[yyq2013]

I have same issue with 1.3.0:
failed at line 55 of EncodersJvm.kt:
check(padding.contentEquals(GZIP_HEADER_PADDING)){ "Gzip padding invalid." }
padding:
[0, 0, 0, 0, 0, 0, 3]
expected GZIP_HEADER_PADDING:
[0, 0, 0, 0, 0, 0, 0]

they not equal.

Can it just compare padding length?

[pecanw]

I have implemented the correct gzip header check and created a pull request.

[e5l]

Fixed in 1.3.2

[jittya]

gzip still getting failed

            HttpClient(Apache) {
                Logging {
                    logger = Logger.SIMPLE
                    level = LogLevel.ALL
                }
                ContentEncoding {
                    gzip()
                }
            }.get<String>("https://ktor.io/")

RESPONSE https://ktor.io/ failed with exception: kotlinx.coroutines.JobCancellationException: Parent job is Completed; job=JobImpl{Completed}@fbc4134

[pecanw]

I had already found the problem as well - see issue #1780. As a workaround you can set logging to a different level (see a sample code linked to the issue - ktorGzipTest)