Node: Undocumented breaking change in stream.pipeline?

[aduh95]

• Version: 14.0.0
• Platform: any
• Subsystem: stream, doc

What steps will reproduce the bug?

It seems Node.js 14 introduced a breaking change in the stream.pipeline API, however the docs doesn't reference any change related to Node 14.

Refs: https://github.com/maxogden/extract-zip/issues/94

TLDR of the above issue is that sometimes the promisified version of pipeline never resolves, causing the program to hang or to exit with unfinished promises.

How often does it reproduce? Is there a required condition?

I have a limited knowledge of the Node.js Stream API, I haven't been able to isolate the bug in the issue I referenced above.

What is the expected behavior?

If there is a breaking change on Node.js 14, it should be documented, at least in the YAML metadata.

Ideally there would be a migration guide to workaround the breaking change.

What do you see instead?

Last documented change references 13.10.0.

Additional information

Probably related to https://github.com/nodejs/node/pull/32158.

CC: @ronag

[Himself65]

I think we have another undocumented semver-major change

like #32987

[ronag]

32987

@Himself65 how is that related to streams?

[ronag]

@aduh95 Do you think you could provide a full example on how to reproduce?

[Himself65]

@Himself65 how is that related to streams?

Absolutely not, I just thought this's maybe an undocumented update issue like that one. Because the commit https://github.com/nodejs/node/commit/1428a92492469b7ea1ee91acd204854a615be4eb which author mentioned to only release in V14

[ronag]

FYI, I've moved the conversation to https://github.com/maxogden/extract-zip/issues/94 for now until I know how to reproduce the issue.

[targos]

I had a similar issue with another zip extracting library. I can try to make a repro.

[ronag]

I had a similar issue with another zip extracting library. I can try to make a repro.

Thanks. I'm struggling to make it fail. So far all my tries succeed.

[ronag]

Ok, I managed to make it fail, pre https://github.com/nodejs/node/pull/32967.

@aduh95 Can you confirm whether this is still a problem after https://github.com/nodejs/node/pull/32967?

[targos]

@ronag see https://github.com/targos/bug-zip-pipeline

Expected: node test.js should print two lines (start/finished reading zip file) and the zip file should be fully extracted at result/.

Actual:

• Node.js 13: works
• Node.js 14.0.0: only one file is extracted and one line is printed
• Latest 15.0.0 nightly: only one file is extracted and one line is printed

Edit:

If pipeline is replaced with manual pipe + on('finish'), it works in all versions.

[aduh95]

@aduh95 Can you confirm whether this is still a problem after #32967?

Yes, I can confirm it is still a problem. I am also able to reproduce using @targos repo:

$ npm install
$ ../node/out/Release/node --version
v15.0.0-pre
$ ../node/out/Release/node test.js
start reading zip file
$ node13 --version
v13.13.0
$ node13 test.js 
(node:53994) ExperimentalWarning: The ESM module loader is experimental.
start reading zip file
finished reading zip file

[ronag]

The problem seems to be that the read stream never emits a 'close' event event though it probably should. Either need to fix the read stream or make finished even more strict in regards to when to assume it should wait for 'close'. Digging...

[ronag]

So the problem goes back to fd-slicer which implements its own destroy() function overriding the stream default autoDestroy behavior, thus it never emits 'close'.

https://github.com/andrewrk/node-fd-slicer/blob/master/index.js#L102

At the moment I'm a little unsure how to fix this without basically to large degree disabling the willEmitClose behavior https://github.com/nodejs/node/blob/master/lib/internal/streams/end-of-stream.js#L75.

[ronag]

Neither fd-slicer not yauzl seems to be maintained anymore 😞 so fixing them does not feel likely. https://github.com/thejoshwolfe/yauzl/pull/115

[jasnell]

/cc @mafintosh @mcollina

[ronag]

I think this should fix it, https://github.com/nodejs/node/pull/33058

[mcollina]

@ronag could you also send a PR that list the semver-major changes to the history of pipeline/finished etc?

[ronag]

@ronag could you also send a PR that list the semver-major changes to the history of pipeline/finished etc?

Sure, though I don't quite understand what that means in practice? In the documentation? Where and how? i.e. what is "the history of pipeline/finished etc"?

https://github.com/nodejs/node/pull/33065

[ronag]

Ah, you mean the changes: section?

[barisusakli]

Can anyone confirm if this issue is fixed? File uploads still fail with a simple app with connect-multiparty. https://github.com/expressjs/connect-multiparty/issues/29

[aduh95]

I can confirm it is fixed on Node.js 14.1.0, I haven't tested the app you referenced though.

[StephenLynx]

14.3 and multi-party is still broken tbh fam

[exe-dealer]

here is my minimal bug reproduction https://github.com/exe-dealer/node-iss-33050
node v10 works fine, but node v14 is broken