Node-rdkafka: Piping consumer stream to Rx, handling backpressure

[michallevin]

I would like to pipe the consumer stream to Rx and to handle backpressure. My logic consists of reading from a Kafka topic (using the consumer stream), normalizing the data, and uploading to AWS's S3.
I thought of:

1. "Pulling" messages from the consumer stream, and continuing to pull only when the service is ready to handle more messages (after uploading to S3). Is this possible with the new consumer stream?
2. "Pipe" the stream directly to an Rx observable. However, I do not know of a way to tell the Kafka stream to "stopAndWait" in case of backpressure.

[webmakersteve]

The Kafka Consumer stream will not read unless the node stream calls read on it. The way node streams work is that the write stream has a write callback, and it will only read as fast as it can write. The read events are "deferred" in the sense that the read callback is expected to push data eventually, and the writes will not process anything until they get the readable event, which happens when there is data to read.

Node streams should provide what you are looking for out of the box provided you do not call the write callback in your piped to write stream until you have fully uploaded the message to s3.

If you have a code sample I'd be happy to take a look.

[webmakersteve]

Going to close this issue because of inactivity, but if you're still having difficulties feel free to comment or re-open if needed.