Node: Z_DATA_ERROR on this site

[ile]

I get Z_DATA_ERROR on this url when requesting a page with request and when gzip is true: http://www.marketsnare.com/ ... so it seems compression related.

http://www.marketsnare.com/ { Error: incorrect header check
    at Zlib._handle.onerror (zlib.js:370:17) errno: -3, code: 'Z_DATA_ERROR' }

| software | version
| ---------------- | -------
| request | 2.80.0
| node | v6.10.3 and 8.1.4
| npm | 3.10.10
| Operating System | Ubuntu 16.04

Requesting the page without gzip works.

I think this shouldn't happen.

Also reported this here: https://github.com/request/request/issues/2713

[KryDos]

Hi @ile, the site is not sending gzipped data back to the client.
As far as I can see the error is expected (see test case in request repository).

Probably not related to Node or zlib and it's library specific. I don't even think it's a bug.

[ile]

Hmm, if I do this:

var request = require('request');

request({ url: 'http://www.marketsnare.com/', gzip: true }, (err, response, body) => {
    if (err) {
        console.error(err);
    }
    else {
        console.log(body);
    }
});

Then I get:

ile@pc:/src$ node test5.js 
{ Error: incorrect header check
    at Zlib._handle.onerror (zlib.js:370:17) errno: -3, code: 'Z_DATA_ERROR' }
ile@pc:/src$ 

So, something is wrong...?

[Trott]

It seems odd that looking at the Network tab in Chrome dev tools for the site, it indicates that the site is using gzip compression, but checking the site using https://checkgzipcompression.com/?url=http%3A%2F%2Fwww.marketsnare.com%2F, it says it is not using compression.

[Trott]

@ile Your code works fine for me (no errors) with request 2.81.0. Can you update to 2.81.0?

[Trott]

Actually, it works for me with 2.80.0 too. I wonder if you might be behind a proxy server or something?

[Trott]

Also: @ile What happens if you try a more common URL, say http://www.google.com?

[ile]

Ok, this seems to work on another computer (Mac) but not on an Ubuntu 16.04 (two different machines). Here's the info of one of the Ubuntus:

Linux pc 4.4.0-83-generic #106-Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

So maybe it's OS/architecture dependent, not sure. I can test some more, but I have actually used this same code on production and it has worked for tens of thousands of urls before. This one url seems to be the only problematic so far.

[Trott]

This one url seems to be the only problematic so far.

Interesting. Perhaps that site chooses not to compress (or compresses incorrectly) with certain User-Agent strings or something like that, ignoring the Accepts-Encoding header in those cases. It seems like a stretch, but it would explain the behavior you see along with the fact that I see the site compressing the payload when I look at headers in Chrome Dev Tools but I get "compression not implemented" when I use checkgzipcompression.com on the site.

Actually, the fact that checkgzipcompression.com reports the site as not compressing makes me wonder if I should see if I can find out if they're using Node.js or not. If not, it's a good indication that the problem is with the site and nothing to do with Node.js. But if they are using Node.js, then that still leaves open the possibility of a Node.js or request bug in here somewhere.

[bnoordhuis]

Occam's razor suggests a website bug more than anything else.

Actually, the fact that checkgzipcompression.com reports the site as not compressing makes me wonder if I should see if I can find out if they're using Node.js or not.

It identifies itself as Microsoft-IIS/8.5.

[ile]

Were anybody else able to reproduce this, or is it just me?

[Trott]

Were anybody else able to reproduce this, or is it just me?

I think @addaleax looked at it for a bit last week and might have even tried to reproduce it without request or something, but I don't remember what conclusions she arrived at (if any).

[addaleax]

I also could reproduce this on Ubuntu, but only with request < 2.80.0 (which makes sense to me, I supplied a fix to request 2.80.0 for something that may or may not be related to your problem).

A wire capture, or even better a dump of the exact data that request passes to the gunzip stream, would be really helpful.

[ile]

Now everything seems to work, I'm not sure why. I can't reproduce this error anymore. Not with request 2.80.0 or 2.81.0 (I'm not sure when the 2.81.0 was released - I thought I was using the latest at the time of the bug report).

So, I think this can be closed for now as it doesn't seem to happen anymore.

[bnoordhuis]

I'll close it out, thanks for following up.

[rogierslag]

Still encountering this issue.

For a reproducable test case you can do the following:

1. checkout https://github.com/inventid/opengraph-cacher/tree/zlib-bug-example
2. Install node 8.5.0
3. npm install
4. node index.js
5. In your browser, visit http://localhost:7070/opengraph?url=http://www.mars.com/uk/en/careers/graduates-students/our-programmes/european-finance-development-programme
6. You will notice an error

In app.js of request (part of node_modules), you can add console.log(err, response, body); on line 458. It will clearly show the error

{ url: 'http://www.mars.com/uk/en/careers/graduates-students/our-programmes/european-finance-development-programme',
  timeout: 10000,
  gzip: true,
  headers: { 'user-agent': 'request.js' },
  encoding: null,
  jar: 
   RequestJar {
     _jar: CookieJar { enableLooseMode: true, store: { idx: {} } } } }
{ Error: incorrect header check
    at Inflate.zlibOnError (zlib.js:152:15) errno: -3, code: 'Z_DATA_ERROR' } undefined undefined

Adding options.gzip = false to just above it ensures it works fine.

[rogierslag]

Any update on this issue? Some incorrect servers still do not work properly like this

[addaleax]

@rogierslag It’s hard to tell without extra information what kind of issue exactly you are seeing, but passing the a finishFlush: zlib.constants.Z_SYNC_FLUSH option to the zlib stream may or may not help you – it will make zlib accept truncated data, if that’s what you want.