Node: Unexpected Z_BUF_ERROR when using zlib

[olalonde]

• Version: 6.x
• Platform: Linux
• Subsystem: zlib

There's been a few Z_BUF_ERROR reported on https://github.com/ttezel/twit:

{ Error: unexpected end of file
    at Zlib._handle.onerror (zlib.js:370:17) errno: -5, code: 'Z_BUF_ERROR' }

https://github.com/ttezel/twit/issues/300
https://github.com/ttezel/twit/issues/269
https://github.com/ttezel/twit/issues/252
https://github.com/ttezel/twit/issues/247

Googling for the error code only brings back to one of those issues or https://github.com/nodejs/node/issues/2043.

It's not clear to me what that error means... The readable stream piping to zlib ended too early? Here's the twit code that creates the zlib stream: https://github.com/ttezel/twit/blob/master/lib/streaming-api-connection.js#L89

[addaleax]

@olalonde Okay, first of all, slightly offtopic but still: You’ll probably want to use gunzip.setEncoding('utf8'); rather than calling .toString('utf8') for each decoded chunk. The latter one won’t handle multibyte characters that cross chunk boundaries cleanly.

And then… it’s pretty hard to tell what’s going on or whether this even is a bug in core without a reproducible test case. Could you add console.log debugging to the self.response.on('data') listener to see what data actually gets passed to zlib?

[olalonde]

Thanks for the suggestion. I was reporting this here to be helpful in case there was a bug in core. I'm not the maintainer or owner to that project. I'll try that out though. If you think it's not a bug with zlib feel free to close.

[addaleax]

If you think it's not a bug with zlib feel free to close.

I don’t know, but I would love to help find that out! :)

[bnoordhuis]

I'll go ahead and close out the issue. Can reopen when new information is available.

[CTassisF]

I was having this problem running Node.js v6.9.1 (from NodeSource) on Linux.
Also tried the changes suggested by @addaleax but the problem persisted.
I decided to downgrade Node.js to v4.6.1 (from Debian Stretch official mirror) and now the problem is gone.
I would like to troubleshoot it more, because I think it might be related to Node.js, but I do not have any ideas on what to do. Do you think identifying the latest Node.js version that runs this library without crashing zlib.js would help?

[bnoordhuis]

@CTassisF If you can get git bisect to pinpoint the commit, that would help tremendously, but even knowing the last known-good release could prove quite useful.

[addaleax]

@CTassisF I would also still really recommend trying to give log some/all of the data that is being passed to zlib.

[CTassisF]

@bnoordhuis Thanks for your reply!
Unfortunately I won't be able to test further because this zlib.js error was being triggered by an issue in Twitter's streaming backend that was also affecting many other Twitter libraries (including other programming languages) and that was fixed yesterday. There is a post on Twitter Developer Forum regarding this issue.
I'm back running Node.js v7.0.0 from NodeSource and everything is looking fine. I will keep you posted if it happens again.
Thanks again :)

[dege88]

I also encountered this issue, and can provide you with a test that can reproduce the issue:

While using request to fetch the homepage of a website I got this strange issue on newer node versions, everything is fine on 4.6:

$ nvm run --lts=argon reproduceissue.js
Running node LTS "argon" -> v4.6.2 (npm v2.15.11)
working without gzip
working with gzip
$ nvm run --lts=boron reproduceissue.js
Running node LTS "boron" -> v6.9.1 (npm v3.10.8)
not working with gzip
{"errno":-5,"code":"Z_BUF_ERROR"}
working without gzip

Here's the code used to reproduce issue:

var request = require('request');

request({
    url: 'http://www.mantovanispa.it',
    gzip: true
}, function(error, response, body)
{
    if(!error)
    {
        console.log('working with gzip');
    }
    else
    {
        console.log('not working with gzip');
        console.log(JSON.stringify(error));
    }
});

request({
    url: 'http://www.mantovanispa.it'
}, function(error, response, body)
{
    if(!error)
    {
        console.log('working without gzip');
    }
    else
    {
        console.log('not working without gzip');
        console.log(JSON.stringify(error));
    }
});

The issue seems linked to gzip, maybe this site use a strange way to handle compression, but on older versions of node this won't happen.

@bnoordhuis can I help you to debug in some other ways?

[addaleax]

curl -H 'Accept-Encoding: gzip' http://www.mantovanispa.it | gunzip does warn about gzip: stdin: unexpected end of file, so there’s definitely something weird going on on the server side

[addaleax]

@dege88 fyi, I filed https://github.com/request/request/pull/2492 for the request module specifically (which should address your particular case)