Google-api-nodejs-client: ECONNRESET error with large sheet read with v 32.0.0

[ovaris]

Hi, after upgrading to new Axios based sheets api, I'm having troubles reading rather large sheet,
after few minutes it throws error, see below.

I'm able to read smaller sheets, but request is notable slower compared to v24 of the client.

I'm reading sheet as following:

const response = await sheetsAPI.spreadsheets.get({
    auth: jwtClient,
    spreadsheetId,
    includeGridData: true
  });

(tested with NodeJS 9.9.0 and latest 10.5)

{ Error: read ECONNRESET
    at TLSWrap.onread (net.js:602:25)
  errno: 'ECONNRESET',
  code: 'ECONNRESET',
  syscall: 'read',
  config: 
   { adapter: [Function: httpAdapter],
     transformRequest: { '0': [Function: transformRequest] },
     transformResponse: { '0': [Function: transformResponse] },
     timeout: 0,
     xsrfCookieName: 'XSRF-TOKEN',
     xsrfHeaderName: 'X-XSRF-TOKEN',
     maxContentLength: 2147483648,
     validateStatus: [Function],
     headers: 
      { Accept: 'application/json, text/plain, */*',
        'Accept-Encoding': 'gzip',
        'User-Agent': 'google-api-nodejs-client/32.0.0 (gzip)',
        Authorization: 'Bearer ya29.c.xxx-xxx-xxx' },
     method: 'get',
     url: 'https://sheets.googleapis.com/v4/spreadsheets/xxx',
     paramsSerializer: [Function],
     data: undefined,
     params: { includeGridData: true } },
  request: 
   Writable {
     _writableState: 
      WritableState {
        objectMode: false,
        highWaterMark: 16384,
        finalCalled: false,
        needDrain: false,
        ending: false,
        ended: false,
        finished: false,
        destroyed: false,
        decodeStrings: true,
        defaultEncoding: 'utf8',
        length: 0,
        writing: false,
        corked: 0,
        sync: true,
        bufferProcessing: false,
        onwrite: [Function: bound onwrite],
        writecb: null,
        writelen: 0,
        bufferedRequest: null,
        lastBufferedRequest: null,
        pendingcb: 0,
        prefinished: false,
        errorEmitted: false,
        bufferedRequestCount: 0,
        corkedRequestsFree: [Object] },
     writable: true,
     _events: 
      { response: [Function: handleResponse],
        error: [Function: handleRequestError] },
     _eventsCount: 2,
     _maxListeners: undefined,
     _options: 
      { protocol: 'https:',
        maxRedirects: 21,
        maxBodyLength: 2147483648,
        path: '/v4/spreadsheets/xxx?includeGridData=true',
        method: 'get',
        headers: [Object],
        agent: undefined,
        auth: undefined,
        hostname: 'sheets.googleapis.com',
        port: null,
        nativeProtocols: [Object],
        pathname: '/v4/spreadsheets/xxx',
        search: '?includeGridData=true' },
     _redirectCount: 0,
     _redirects: [],
     _requestBodyLength: 0,
     _requestBodyBuffers: [],
     _onNativeResponse: [Function],
     _currentRequest: 
      ClientRequest {
        _events: [Object],
        _eventsCount: 6,
        _maxListeners: undefined,
        output: [],
        outputEncodings: [],
        outputCallbacks: [],
        outputSize: 0,
        writable: true,
        _last: true,
        upgrading: false,
        chunkedEncoding: false,
        shouldKeepAlive: false,
        useChunkedEncodingByDefault: false,
        sendDate: false,
        _removedConnection: false,
        _removedContLen: false,
        _removedTE: false,
        _contentLength: 0,
        _hasBody: true,
        _trailer: '',
        finished: true,
        _headerSent: true,
        socket: [TLSSocket],
        connection: [TLSSocket],
        _header: 'GET /v4/spreadsheets/xxx?includeGridData=true HTTP/1.1\r\nAccept: application/json, text/plain, */*\r\nAccept-Encoding: gzip\r\nUser-Agent: google-api-nodejs-client/32.0.0 (gzip)\r\nAuthorization: Bearer ya29.c.xxx-xxx-xxx\r\nHost: sheets.googleapis.com\r\nConnection: close\r\n\r\n',
        _onPendingData: [Function: noopPendingOutput],
        agent: [Agent],
        socketPath: undefined,
        timeout: undefined,
        method: 'GET',
        path: '/v4/spreadsheets/xxx?includeGridData=true',
        _ended: false,
        res: [IncomingMessage],
        aborted: undefined,
        timeoutCb: null,
        upgradeOrConnect: false,
        parser: null,
        maxHeadersCount: null,
        _redirectable: [Circular],
        [Symbol(isCorked)]: false,
        [Symbol(outHeadersKey)]: [Object] },
     _currentUrl: 'https://sheets.googleapis.com/v4/spreadsheets/xxx?includeGridData=true' },
  response: undefined }

[miv]

I'm seeing same bug. Using request library to make same https request instead of axios works 10 times as fast.

[miv]

So, with some digging i've found this pile of horse manure in axios code

        stream.on('data', function handleStreamData(chunk) {
          responseBuffer.push(chunk);

          // make sure the content length is not over the maxContentLength if specified
          if (config.maxContentLength > -1 && Buffer.concat(responseBuffer).length > config.maxContentLength) {
            reject(createError('maxContentLength size of ' + config.maxContentLength + ' exceeded',
              config, null, lastRequest));
          }
        });

Who in his right mind could have made buffer allocation in a tight loop just to check it's length is escaping me.

@ovaris @JustinBeckwith just add maxContentLength: -1, to params of your request to api and it will work fast. Like so:

    const data =  await sheets.spreadsheets.get(
        {
            spreadsheetId,
            includeGridData: true,
            ranges: "MAIN"
        },
        {
            maxContentLength: -1,
            params: {
                fields:
                    "sheets(data(rowData(values(userEnteredFormat/backgroundColor,userEnteredValue)),startColumn,startRow))"
            }
        }
    );

[JustinBeckwith]

Greetings folks! I'm like 99% sure this was fixed in v37 :) If you're still running into issues - let me know!