Node: Need a way to detect end of stream on custom Writable Streams implementing _write

[GeorgeBailey]

It seems that _write should be called with null as the data, when the end of stream is reached.

The problem I am having is that I need to add additional data to the end of the stream, and the finish event should not be emitted until after that is completed. I need a way to control this, so I'm thinking a new _end function with a callback, or perhaps a callback on _write(null) would do the trick.

[GeorgeBailey]

Actually, I'm surprised this was not already in place. (or is it?)

[jhamhader]

Why not call writableStream.end('last data', cb)?

[brendanashworth]

I think the prefinish event (which may be undocumented) would fit your requirements. Maybe try that for a spin?

[vkurchatkin]

I think the prefinish event (which may be undocumented) would fit your requirements. Maybe try that for a spin?

It's not public. Take a look at this PR: https://github.com/nodejs/node/pull/2314

[brendanashworth]

@vkurchatkin ah, thanks for pointing me to that. @GeorgeBailey they might be interested in hearing your use case.

[GeorgeBailey]

they might be interested in hearing your use case.

@brendanashworth, I am piping an http request and response over stdio to isolate various webapps in a process tree. This way an individual webapp can be killed without interrupting the whole server. I need to call res.end() in the parent process after res.end() in the child process is called.

( I realize a more common approach to process isolation is to set up localhost HTTP servers and listen on various TCP ports; using a proxy. However, for our environment we determined the stdio approach was more advantageous. )

I can certainly see other cases where detecting end of stream would be appropriate. (i.e. an encoding stream that has to add trailing data to the end of a compressed file format)

[GeorgeBailey]

Why not call writableStream.end('last data', cb)?

@jhamhader, There needs to be a way to distinguish data from end of stream. The easiest way to do this is just overriding the write and end properties to begin with.

The method you suggest is less efficient as it requires parsing the stream for a pre-determined string. (similar to a mime id)

My understanding is that Writable Streams are intended to provide a more robust solution via an override to _write.

[Trott]

Did anything ever come of this? If not, is anything likely to come of it? (Basically, trying to figure out if this should be closed or if something could happen to get things moving, as it's been dormant for 7 months...)

[olalonde]

Also need this... Use case is to convert http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#upload-property to a sync call that returns a write stream (I'm writing an adapter and have to conform to a stream returning interface which I have no control over).

// aws-sdk, callback api
s3.upload({ Key: 'somekey', Body: readStream }, (err) => {
  // etc.
})

const { PassThrough } = require('stream') 
const createWriteStream = (key) => {
  const through = new PassThrough()
  s3.upload({ Key: key, Body: through }, (err) => {
    // etc.
  })
  return through
}

The above doesn't cut it because the finish event is fired on through before the s3.upload callback is called. There doesn't seem to be any way to have through wait for the callback before emitting finish with the current stream implementer API.

[mcollina]

This is solved by #12828 and the new _final  method that has been added to Node 8.0.0. Please have a look.