Node-mssql: Error on streaming huge recordset

[AlexSenchenko]

Hi @patriksimek please consider following issue.

"mssql": "^3.3.0",

  mssql: Object.assign({}, {
        stream: true,
        options: {encrypt: false},
        debug: true,
        requestTimeout: 50000,
        pool: {
            max: 10,
            min: 0,
            idleTimeoutMillis: 100000,
        },
    }),

Hi, I have error on while streaming 1000.000.
It begin streaming normally but after interruption can't proceed.

Code looks like following:

 const requestGenerator = MssqlAdaptor.Factory(mssql);
            requestGenerator.then((adaptor) => {
                let request = adaptor.request;


                request.on('error', function (err){
                    console.log('Request error');
                    console.log(err);

                });
                request.on('row', function (data){
                    console.log('Next data: ' + data);
                    elasticAdaptor.addRawData([data]);

                });
                request.on('done', function (){
                    elasticAdaptor.replaceOldIndex().then(function (err, result) {
                        done(null, "Finished");

                    });
                });

                adaptor.executeRequest(query);
            });

Request error
{ RequestError: Timeout: Request failed to complete in 50000ms
    at Request.userCallback (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/mssql/lib/tedious.js:802:25)
    at Request.callback (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/tedious/lib/request.js:33:27)
    at Connection.message (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/tedious/lib/connection.js:1209:31)
    at Connection.dispatchEvent (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/tedious/lib/connection.js:519:45)
    at MessageIO.<anonymous> (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/tedious/lib/connection.js:439:23)
    at emitNone (events.js:86:13)
    at MessageIO.emit (events.js:185:7)
    at ReadablePacketStream.<anonymous> (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/tedious/lib/message-io.js:92:15)
    at emitOne (events.js:96:13)
    at ReadablePacketStream.emit (events.js:188:7)
    at readableAddChunk (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/readable-stream/lib/_stream_readable.js:198:18)
    at ReadablePacketStream.Readable.push (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/readable-stream/lib/_stream_readable.js:157:10)
    at ReadablePacketStream.Transform.push (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/readable-
[mssql_debug_1486634130964.txt](https://github.com/patriksimek/node-mssql/files/763287/mssql_debug_1486634130964.txt)
[mssql_debug_1486634516673.txt](https://github.com/patriksimek/node-mssql/files/763293/mssql_debug_1486634516673.txt)
[mssql_debug_1486634844538.txt](https://github.com/patriksimek/node-mssql/files/763294/mssql_debug_1486634844538.txt)
[mssql_debug_1486634900132.txt](https://github.com/patriksimek/node-mssql/files/763291/mssql_debug_1486634900132.txt)
[mssql_debug_1486634978027.txt](https://github.com/patriksimek/node-mssql/files/763292/mssql_debug_1486634978027.txt)




stream/lib/_stream_transform.js:123:32)
    at ReadablePacketStream._transform (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/tedious/lib/message-io.js:57:16)
    at ReadablePacketStream.Transform._read (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/readable-stream/lib/_stream_transform.js:159:10)
    at ReadablePacketStream.Transform._write (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/readable-stream/lib/_stream_transform.js:147:83)
    at doWrite (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/readable-stream/lib/_stream_writable.js:313:64)
    at writeOrBuffer (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/readable-stream/lib/_stream_writable.js:302:5)
    at ReadablePacketStream.Writable.write (/Users/alse/src/Forks/EyeFitU.NodeJs.Server/node_modules/readable-stream/lib/_stream_writable.js:241:11)
    at CleartextStream.ondata (_stream_readable.js:561:20)
    at emitOne (events.js:96:13)
    at CleartextStream.emit (events.js:188:7)
  name: 'RequestError',
  message: 'Timeout: Request failed to complete in 50000ms',
  code: 'ETIMEOUT',
  number: 'ETIMEOUT',
  lineNumber: undefined,
  state: undefined,
  class: undefined,
  serverName: undefined,
  procName: undefined }

[AlexSenchenko]

mssql_debug_1486634978027.txt

[tracker1]

If it's taking more than 50 seconds to stream all the records, it's your timeout limit set in your settings... if you need more, you may want to bump up to how long you think it will take to read/process all the rows in question.

[AlexSenchenko]

@tracker1 thanks. What actually max valid timeout boundary?

[tracker1]

It's in your config as requestTimeout ... What would be valid is whatever you actually need... you may want to set it to something really big, like 10 minutes, time how long the process takes, then reign it in to closer to what you actually need. It depends on the context... If you're doing requests in a web server, 50 seconds is reasonable... for a migration/import/export script, it's whatever you need it to be.

[munir131]

I think streaming should be free from whatever set in requestTimeout

[dhensby]

This is most likely because requests didn't implement any kind of pausing or back pressure. This is now implemented in v5