Jsforce: Data fetch get stuck for large datasets

[jbonigomes]

Hi there,

I am experiencing a problem fetching large datasets (roughly half million rows), my query looks a little like this:

conn
  .sobject('Account')
  .select('the,fields,I,need')
  .execute({ autoFetch: true, maxFetch: 1000000 })
  .on('record', (res) => console.log(res))
  .on('error', (err) => console.log(err))
  .on('end', () => console.log('all done'));

Not always, but more often than not, my callback stops being called and it never reaches the "on.('end')", neither it gives me any errors.

Is there something I am doing wrong?

Thanks

[jbonigomes]

Correction, it did throw me an error this time, even though the same data set was imported before:

RangeError: Maximum call stack size exceeded

[jbonigomes]

Correction, I no longer get the Maximum call stack error, the app just hangs, no response, no errors.

[jbonigomes]

The issue was that I was inserting the results into my MongoDB, the writing process would take longer than the results received from SalesForce, I fixed the problem by pushing the results to an array first, then, when the process ends, insert the records into MongoDB.