Google-api-nodejs-client: drive.files.create with ReadableStream creates a file with 0 bytes

[cyrusmg]

Environment details

• OS: Arch Linux (latest)
• Node.js version: v12.16.1
• npm version: 6.13.4
• yarn version: 1.22.4
• googleapis version: 49.0.0
• aws-sdk version: ^2.115.0

Description

There seems to be an issue introduced in v49 (it works as expected in v48) with Stream upload from S3 to Google Drive. On v48 I am able to get a ReadStream from S3 and pass it directly to Google Drive via the media.body argument. On v49 I get 0 bytes in Google Drive (and no errors).

Reproduction:

1) Get S3 stream via s3.getObject(key).createReadStream()
2) Pass it to google apis: drive.files.create({ requestBody: { name, mimeType }, { media: { mimeType, body: stream } })
3) Check response - it says 0 bytes and there is an empty file on google drive

What I have tried:
1) Stream the data to older version googleapis - it works on v48, I get non-zero bytes
2) Stream the data to fs.createWriteStream - works
3) Wrap the stream in PassThrough as suggested in other issues - no luck

[sofisl]

Hi @cyrusmg,

This issue sounds very similar to this issue we were having before. Essentially, when users were trying to upload streams in the browser context, they weren't able to. We fixed the problem in the newest version of the library. Now, the library only supports streams in non-browser-upload contexts. In Node contexts, it can be whichever.

In that issue, we recommended either the user download it to memory and then upload it for browser contexts, or switch the upload to the Node context. Would that solution work for you, or are you looking for a different functionality?

[cyrusmg]

Thank you for the response! This happens on the backend - so plain Node process context

I specifically want to avoid buffering the stream locally as we handle large files if that's possible. Was the change you mention done within v48-v49 of the googleapis library?

I will check the linked Issue to see if there's anything I can try to fix it.

[JustinBeckwith]

Greeting @cyrusmg! This is a head scratcher. We have a sample (with integration tests) over here that's creating a readable stream from a file on disk, and then using that in the media body to upload the file:
https://github.com/googleapis/google-api-nodejs-client/blob/master/samples/drive/upload.js

I'm suffering a little "it works on my machine" ™️ Taking a look at the v49 release notes:
https://github.com/googleapis/google-api-nodejs-client/releases/tag/v49.0.0

The only interesting change there is the release of googleapis-common@4. And the 4.x release there is also exceptionally boring:
https://github.com/googleapis/nodejs-googleapis-common/releases/tag/v4.0.0

I don't have an AWS account handy (heh, googler) so it's a bit difficult for me to test that specific case. Could I trouble you to try this out with a local readable stream, and see if that works for you? I just want to level set against what I'm seeing.

[cyrusmg]

I have tried to set up a test case that I though would be sufficient, but it has not failed on it. I will dig more to see what's the real minimal test case.

Note: I can still reproduce this with our full application.

This is working correctly:

{
  "name": "gdrive-test",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "dependencies": {
    "aws-sdk": "2.522.0",
    "googleapis": "49.0.0"
  }
}

const AWS = require("aws-sdk");
const { google } = require("googleapis");

(async () => {
  const s3 = new AWS.S3();
  const request = s3.getObject({ Bucket: "www.bibtex.com", Key: "index.html" });
  const readStream = request.createReadStream();

  const oauth2Client = new google.auth.OAuth2(
    process.env.GOOGLE_CLIENT_ID,
    process.env.GOOGLE_CLIENT_SECRET
  );
  oauth2Client.setCredentials({ refresh_token: process.env.REFRESH_TOKEN });

  const drive = google.drive({ version: "v3", auth: oauth2Client });
  await drive.files.create({
    resource: {
      name: "index.html",
      mimeType: "text/html",
    },
    media: {
      mimeType: "text/html",
      body: readStream,
    },
  });
})();

[cyrusmg]

I have narrowed it down to require('internal-legacy-library') which does all kinds of magic internally and is used from both Chrome browser and Node backend.

Note: I could also reproduce it with fs.createFileStream, so it was not related to AWS code and also happens on v51.0.0 so the changes implemented w.r.t. browser-context did not affect it.

I will close this down for now and keep on investigating how the legacy library could be interacting with the code in googleapis and/or with default stream behaviour and reopen it only if I think there's anything to improve in googleapis library.

Thank you for your time!

[JustinBeckwith]

Awesome! Let me know if there's anything we can do of course.