Superagent: 'incorrect header check' parsing gzip response on every call but the initial one

[matthiasg]

I am seeing the following error on every request but the first to a resource (using 1.7.2). It might be related to #410

 { [Error: incorrect header check] errno: -3, code: 'Z_DATA_ERROR', response: null } Error: incorrect header check
  at Zlib._handle.onerror (zlib.js:363:17)
From previous event:
....

The Server resource headers returned (so it supports gzip). The server is a hapi server with default options (version 13)

Connection →
Connection
Options that are desired for the connection
keep-alive
Date → Tue, 08 Mar 2016 06:48:58 GMT
Transfer-Encoding → chunked
cache-control → no-cache
content-encoding → gzip
content-type → application/json; charset=utf-8
vary → accept-encoding

The resource is being called by a very simple setup:

    request.get("http://localhost:8963/commands/failed")
      .end( (err,res)=>{
        if(res.ok)
          callback(null, res.body)
        else
          callback(`ERR: ${res.status} > ${res.text}`)
       });

Calling the same resource from curl or a browser etc shows no issues.

[kornelski]

Can you check v1.8.2?

[gwely]

I'm having this issue with superagent (v1.8.3) and supertest (v1.2.0).

Here is my request code:

var api = supertest(config.get('baseUrl'));
// var api = require('superagent');

api.post('/api/token')
    .set('Accept', 'application/json')
    .type('form')
    .send(loginData)
    .expect('Content-Type', /json/)
    .expect(200)
    .end(function (postErr, postResponse) {
        if (postErr) {
            console.log(postErr);
            return done(postErr);
        }

        access_token = postResponse.body.access_token;
        token = 'Bearer ' + access_token;

        api.get('/swagger/docs/v1')
        // with superagent - api.get(config.get('baseUrl') + '/swagger/docs/v1')
            .set({
                'accept': 'application/json;charset=utf-8,*/*',
                'Accept-Encoding': 'gzip,deflate,sdch',
                'Accept-Language': 'en-US,en;q=0.8',
                'Authorization': token
            })
            .end(function (getErr, getResponse) {
                if (getErr) {
                    console.log(getErr);
                    return done(getErr);
                }

                var filename = path.resolve(__dirname, 'schema', 'v1.json');
                jsonfile.writeFile(filename, getResponse.body, function (writeError) {
                    if (writeError) {
                        console.log(writeError);
                        return done(writeError);
                    }

                    done();
                });

            });
    });

I've tried the get request without any headers set other than Authorization and various combinations of what's above.

I get the same error:

{ [Error: incorrect header check] errno: -3, code: 'Z_DATA_ERROR', response: null }
Error: incorrect header check
  at Zlib._handle.onerror (zlib.js:363:17)

The same request works perfectly fine in postman

[gwely]

Also works fine with request

var options = {
    url: config.get('baseUrl') + '/swagger/docs/v1',
    headers : {
        'Authorization': token
    }
};

request(options, function (getErr, getResponse, body) {
    if (getErr) {
        console.log(getErr);
        done(getErr);
    }

    console.log(JSON.parse(body).info.version); // v1
    done();
});

[kornelski]

To fix this issue I need to see HTTP headers from server response and request body, if there was any.

[gwely]

The request body is empty.

I got the following HTTP headers from the response from a successful request via Postman.

Cache-Control →private
Content-Encoding →deflate
Content-Length →15544
Content-Type →application/json; charset=utf-8
Date →Tue, 03 May 2016 14:04:54 GMT
Server →Microsoft-IIS/7.5
X-AspNet-Version →4.0.30319
X-Powered-By →ASP.NET

[kornelski]

if the body is empty, but the server is saying Content-Length: 15544, then it's the server's fault it doesn't work. The response is invalid, and gzip is right to complain that an empty body is not the 15KB of gzip data promised by the server.

[gwely]

Oh sorry the _request_ body is empty. The _response_ body is quite large and also something I can't really post publicly. Is there something I can check for you?

[kornelski]

Oh, OK, so in that case it should work. The other thing that stands out in the headers you've posted is encoding deflate rather than the more common gzip, so maybe we have a bug in there — I'd have to set up a test case for it.

[kornelski]

If you'd like to help, it'd be super helpful if you wrote a unit test that exposes the bug.

[lcgg110]

superagent.get('http://www.test.com').set('Accept-Encoding','gzip, deflate').end(callback())

this test case is not support
error:
incorrect header check
Error: incorrect header check
at Zlib._handle.onerror (zlib.js:366:17)

[futpib]

In case anybody is looking for a workaround:

    const res = await agent
        .get(url)
        .set('Accept-Encoding', 'identity')

[DavidTPate]

Thanks for the workaround @futpib that definitely addresses the issue.

Also, you can change the encoding to something that superagent can handle such as gzip instead of identity as well.

[nebulousdog]

going forward, do we know what versions of sub-dependencies should be used going foward to prevent this?