Node: Piping twice to same Writable stream via two different Readable stream

[amit-upadhyay-IT]

I am trying to concatenate a string and a Readable stream (the readable stream is pointing to a file which may have data in multiple chunks, i.e. the file may be large) into one writable stream so that the writable stream can finally be written to a destination.

I am encrypting the string and content of the file and then applying zlib compression on them, then finally I want to pipe them to the writable stream.

To achieve this, I can:

a) Convert the file content into a string then concatenate both the string then encrypt, do compression and then finally pipe it into the writable stream. But this is not possible because the file may be big in size, thus I can't convert its content to string.

b) I can first encrypt and compress the string then convert the string into a stream then pipe it into the writable stream after that is done completely, pipe the file contents into the same writable stream.

To do so, I have written this:

var crypto = require('crypto'),
    algorithm = 'aes-256-ctr',
    password = 'd6FAjkdlfAjk';

var stream = require('stream');
var fs = require('fs');
var zlib = require('zlib');

// input file
var r = fs.createReadStream('file.txt');
// zip content
var zip = zlib.createGzip();
// encrypt content
var encrypt = crypto.createCipheriv(algorithm, password, iv);

var w = fs.createWriteStream('file.out');

// the string contents are being converted into a stream so that they can be piped
var Readable = stream.Readable;
var s = new Readable();
s._read = function noop() {};
s.push(hexiv+':');
s.push(null);
s.pipe(zlib.createGzip()).pipe(w);

// start pipe when 'end' event is encountered
s.on('end', function(){
    r.pipe(zip).pipe(encrypt).pipe(w);
});

What I observe is:

Only the first pipe is done successfully, i.e. the string is written to the file.out. The second pipe doesn't make any difference on the output destination. At first, I thought that the reason might be due to asynchronous behaviour of pipe. So, for this reason, I am piping the file content after the first piping is closed. But still, I didn't get the desired output.

Why is the second time I am unable to pipe to same write stream?

• Version:
• Platform:
• Subsystem:

[addaleax]

Hey, I hope I’m getting your question right here – your code seems almost correct, with only one gotcha: stream#pipe takes an end options that defaults to true (documentation), so when the first .pipe(w) finishes, the writable at w is also automatically closed.

So I think all you want to do is replace s.pipe(zlib.createGzip()).pipe(w); with s.pipe(zlib.createGzip()).pipe(w, {end: false});?

[amit-upadhyay-IT]

Oh, yes, I just didn't know how to keep the writable stream open. Thank you so much :+1: