Node-postgres: Is there a way to pause and resume query output?

[jlfaber]

I'm using the eventEmitter model to handle the results from a query that produces a large number of rows. The handler has some async components, so I'd really like to be able to keep the next 'row' event from firing until after the current one has completed processing.

Ideally this would take to form of pause() and resume() calls on the query object itself.

 var query = client.query("select * from big_data_table");
 query.on('row', function (row) {
    query.pause(); // turn off the firehose
    return do_something_async(row, function() {
      query.resume(); // turn back on the firehose
    });
 });

[brianc]

What you probably want to do is treat query results as a stream. You can use this, which I've been using heavily the past couple weeks and it's been working great for me: https://github.com/brianc/node-pg-query-stream

I have another implementation of a raw cursor object which is more in line with how postgres itself works, but I've not brought it up to the state where it is fit to release.

I'm trying to work on more experimental things outside of the main repository instead of just adding every feature here and having to support a giant ball of mud. :p

[brianc]

As a follow up. pg-query-stream will exert back pressure on the database so it will only return 100 rows or so at a time into the stream. You can consume the output from the query just as you would consume any other readable stream with the whole

stream.once('readable', function() {
  var row = stream.read()
})

madess

[jlfaber]

Interesting! Thanks
-joe

[TeddyPerkins77]

Hi! sorry for re-opening this, but I was wondering if you were able to achieve what you were looking for @jlfaber ?

I tried following @brianc advice but I couldn't achieve that behavior that Joe is looking for through pg-query-stream.

I figured out that maybe there's another option available to this date , I'm basically looking for exactly this:
"The handler has some async components, so I'd really like to be able to keep the next 'row' event from firing until after the current one has completed processing."

Thanks guys! any feedback will be greatly appreciated.

[jgoux]

I'm interested as well. I'd like to handle an async process inside the data handler, and receive the next row only when my async process in the handler is over.

[brianc]

You can also try https://github.com/brianc/node-pg-cursor which gives you very direct control of when and how many rows you return at a time.

[TeddyPerkins77]

@brianc thank you for quick response! I ended up using query-stream.

@jgoux this worked for me buddy:

var QueryStream = require('pg-query-stream')
var pg = require('pg')

var conString = "yourConnectionString";
var client = new pg.Client(conString);

client.connect(function(err) {
       if(err) {
             return console.error('could not connect to postgres', err);
        }

    if(err) throw err;
    var query = new QueryStream(queryStr, values)
    var stream = client.query(query)

    stream.on('data', function(row) {
        stream.pause();
            //handle your async process here
        stream.resume();
    })
    stream.on('end', function() {
        client.end();
    })
});

Cheers!

[jhagege]

Thanks for your code !
I have an issue with it though, stream.on('end' gets called BEFORE the last resume (last row).

Thanks a lot.

[TeddyPerkins77]

Anytime! I noticed that behavior too (since data its not the same as 'results'), for that I had to use flags in order to process the last record only if I am at the 'end' event and there isn't any other record being processed on 'data'. So basically:

.on('end', function() {

    // Received End Event
    receivedEE = true;

    // Process Last Row
    if(receivedEE && !processingRow)
    {
      processLastRow();
    }

});

where processLastRow consists of whatever you are doing to your rows on 'data'.

[jhagege]

Thanks a lot !

On 18 July 2016 at 03:55, danteleon [email protected] wrote:

Anytime! I noticed that issue too, for that I had to use flags in order to
process the last record only if I am at the 'end' event and there isn't any
other record being processed on 'data'. So basically:

.on('end', function() {

// Received End Event
receivedEE = true;

// Process Last Row
if(receivedEE && !processingRow)
{
processLastRow();
}

});

where processLastRow consists of whatever you are doing to your rows on
'data'.

---

From: cyberjoac [email protected]
Sent: Sunday, July 17, 2016 8:03 PM
To: brianc/node-postgres
Cc: danteleon; Comment
Subject: Re: [brianc/node-postgres] Is there a way to pause and resume
query output? (#460)

Thanks for your code !
I have an issue with it though, stream.on('end' gets called BEFORE the
last resume (last row).

Thanks a lot.

You are receiving this because you commented.
Reply to this email directly, view it on GitHub<
https://github.com/brianc/node-postgres/issues/460#issuecomment-233200951>,
or mute the thread<
https://github.com/notifications/unsubscribe-auth/ASuruJ52z3WMQyoFq424UP7P-3JVwb4qks5qWoqOgaJpZM4BJfYS

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/brianc/node-postgres/issues/460#issuecomment-233214290,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHKEDM6oxZfehR-eCnAdGoYhCEQuaZxwks5qWs8cgaJpZM4BJfYS
.