Nodejs-storage: Running into connection resets after 60s when streaming to GCS, `timeout` setting has no effect

[fspoettel]

Environment details

• OS: Ubuntu 20.04
• Node.js version: 14.5.0
• npm version: using yarn
• @google-cloud/storage version: 5.1.2

Steps to reproduce

I'm trying to stream large files to GCS via createWriteStream(). The files are streamed from a multi-part file upload in a POST handler through ffmpeg to createWriteStream(). When an upload stream takes longer than 60 seconds to complete, the connection resets and the upload never completes. I can't seem to catch an error in that case. I tried to use the timeout option in createWriteStream() but it does not seem to do anything (although it is picked up in reqOpts).

EDIT: The issue is limited to piping the original Express request req through busboy to GCS. If I write the upload to disk before streaming it to GCS, all is well and the upload finishes as expected. So this might not actually be an issue with GCS. Feel free to close this issue in that case :slightly_smiling_face:

My upload code looks like this:

const storageClient = new Storage(credentials);

function gcsUploadStream(key) {
  const pass = PassThrough();

  const bucket = storageClient.bucket(GOOGLE_BUCKET);
  const blob = bucket.file(key);

  const googleCloudStorageStream = blob.createWriteStream({
    resumable: false,
    validation: false,
    timeout: 1000 * 60 * 60,
  });

  pass.pipe(googleCloudStorageStream);

  const promise = new Promise((resolve, reject) => {
    googleCloudStorageStream.on('error', err => {
      googleCloudStorageStream.end();
      reject(err);
    });

    googleCloudStorageStream.on('finish', () => {
      googleCloudStorageStream.end();
      resolve(blob.name);
    });
  });

  return { writeStream: pass, promise };
}

The pipeline can be boiled down to:

app.post('/', (req, res) => {
busboy.on('file', async (fieldname, file, filename) => {
  const { writeStream, promise } = gcsUploadStream(`xo.flac`);
  flacEncoder(fileStream).pipe(writeStream);
});

busboy.on('finish', async () => {
  const gcsUri = await upload.promise;
  debug(`Finished upload for file: ${gcsUri}`);
});

req.pipe(busboy);
});

Any help would be greatly appreciated, thank you very much for looking!

[Eywek]

I was having the same issue, after debugging I've found that the nodejs header timeout is the cause (cf. https://nodejs.org/api/http.html#http_server_headerstimeout), you can try to disable it with:

server.headersTimeout = 0

[fspoettel]

Thanks for chiming in @Eywek! :raised_hands: Will give that a try

[patrickmichalina]

I am seeing this same thing and cannot figure out what is happening here.

[patrickmichalina]

@Eywek can confirm bumping headersTimeout much higher fixes the issue. Thanks for sharing.

[jimmylee]

This issue saved my life, thank you so much. Just in case someone else needs it:

  const listenServer = server.listen(Environment.PORT, (e) => {
    if (e) throw e;

    console.log(`${Environment.PORT}`);
  });

  listenServer.headersTimeout = 0;

In case it wasn't obvious that you couldn't set headersTimeout = 0 on some arbitrary server object.

[shaffeeullah]

Closing because this seems to be resolved. Please reopen if this is still an issue.