Google-api-nodejs-client: Drive API: partial upload of images?

[persona0591]

Hi all,

I'm trying to upload images from my server to Google Drive. However, occasionally images are uploaded only partially, incomplete. I'm not sure if this issue relates to the Node library or to something else. Can you help me?

Environment details

1. OS: Debian
2. Node.js version: 10.13.0
3. npm version: 6.4.1
4. googleapis version: 35.0.0

Steps to reproduce

1. Create file "upload.js" and paste the code underneath. The code is based on the examples (https://github.com/googleapis/google-api-nodejs-client/blob/b00bf4200d254b87e3c5b81ba867abe7ab9343ad/README.md#media-uploads)
2. Add your Google client ID, Google client secret and refresh token to "upload.js"
3. Download the (public) PNG example file https://drive.google.com/open?id=1w1uTL81Oknimz0-RBMjn36mB8RRptYsl. Save as "test01.png" in the same directory as "upload.js"
4. Run file: node upload.js. The output (if Google client ID, Google client secret and refresh token are correct) contains some metadata about the file
5. Go to your Google Drive and find image "testimage.png". Expected result: the image you've downloaded in step 3. Actual result: a partial image (see https://drive.google.com/open?id=1dM8G4CJ7AX4vKhXRqd485t2VH7nox0Og)

The code underneath uses onUploadProgress for tracking upload progress. It says that 180391 bytes are read - identical to the size of the image. However, the image on Google Drive has a size of 65536 bytes.

Code for file "upload.js":

'use strict';

const fs = require('fs');
const { google } = require('googleapis');

const googleClientId = // Your client ID
const googleClientSecret = // Your client secret
const googleRefreshToken = // Your refresh token

const oauth2Client = new google.auth.OAuth2(googleClientId, googleClientSecret, 'http://localhost');
oauth2Client.setCredentials({ refresh_token: googleRefreshToken });

const drive = google.drive({
    version: 'v3',
    auth: oauth2Client
});

async function main() {

    const res = await drive.files.create(
        {
            requestBody: {
            name: 'testimage.png',
            mimeType: 'image/png'
            },
            media: {
            mimeType: 'image/png',
            body: fs.createReadStream('test01.png')
            }
        },
        {
            onUploadProgress: (e) => console.log(e)
        }
    );
    console.log(res.data);
};

main().catch(console.error);

[AVaksman]

@persona0591
I have tested files.create() function multiple times in a couple of different scenarios, all seemed to work properly.
Could you please provide the full working example that you are experiencing issue with.
Also please show how you are obtaining a RefreshToken.

[persona0591]

Hi @AVaksman - thanks for looking into this! I'll provide my working example as soon as possible.

[persona0591]

Hi @AVaksman,

I've created a full working example - see underneath.

Are you able to reproduce my issue using this?

---

Steps to reproduce

1. Create a directory, e.g. google-drive-test
2. Create file upload.js in google-drive-test and paste the code underneath
3. Add your Google client ID and Google client secret to upload.js (constants googleClientId and googleClientSecret)
4. Download the (public) PNG example files:
5. https://drive.google.com/open?id=1w1uTL81Oknimz0-RBMjn36mB8RRptYsl. Save as test01.png in directory google-drive-test
6. https://drive.google.com/open?id=1h7P0NoLgBD_g2-YBc8xtJdXlzQW7wS7-. Save as test02.png in directory google-drive-test
7. Create file package.json in directory google-drive-test and paste the settings underneath
8. Install the required packages: npm install
9. Run file: node upload.js test01.png. This will open a browser and ask permission. Grant permission. File tokens.json will then be created in directory google-drive-test (containing e.g. the refresh_token, access_token and expiry_date). Next, a file upload to Google Drive will start and file test01.png will appear in your Drive (with a file size of some 176 KB)
10. Run file: node upload.js test01.png. This will not open a browser since the required tokens for communicating with Google are available in file tokens.json. A file upload to Google Drive will start and file test01.png will appear in your Drive (with a file size of some 176 KB)

So far, so good. However, this changes if you wait an hour or so (or set the expiry_date in file tokens.json manually to some prior date):

1. Run file: node upload.js test01.png. A file upload to Google Drive will start and file test01.png will appear in your Drive. The size of the file will be some 64 KB - not 176 KB, as was the case in steps 7 and 8
2. Run file: node upload.js test01.png. A file upload to Google Drive will start and file test01.png will appear in your Drive (with a file size of some 176 KB)

In short: file test01.png is only partially uploaded to Drive in step 9 - however, without giving errors of a failed upload. But in step 10, the file is uploaded fully again. Oddly, if I repeat steps 7 to 10 using file test02.png as input, the file in all instances is uploaded fully (with a file size of some 349 KB).

Do you know what causes the partial upload in step 9 with file test01.png?

File: upload.js

'use strict';

const destroyer = require('server-destroy');
const fs = require('fs');
const { google } = require('googleapis');
const http = require('http');
const opn = require('opn');
const path = require('path');
const url = require('url');

const googleClientId = ''; // Your client ID
const googleClientSecret = ''; // Your client secret

class Upload {

    constructor() {
        this.oAuth2Client = new google.auth.OAuth2(googleClientId, googleClientSecret, 'http://localhost:3000/oauth2callback');
    }

    async readCredentialsFromFile() {
        return new Promise((resolve, reject) => {
            fs.readFile('tokens.json', (err, data) => {
                if (err) {
                    return reject(err);
                }
                const tokens = JSON.parse(data);
                resolve(tokens);
            });
        });
    }

    async writeCredentialsToFile(tokens) {
        return new Promise((resolve, reject) => {
            fs.writeFile('tokens.json', JSON.stringify(tokens), (err) => {
                if (err) {
                    return reject(err);
                }
                resolve();
            });
        });
    }

    async authenticate() {
        return new Promise((resolve, reject) => {
            const authorizeUrl = this.oAuth2Client.generateAuthUrl({
                access_type: 'offline',
                scope: 'https://www.googleapis.com/auth/drive.file'
            });

            const server = http
                .createServer(async (req, res) => {
                    try {
                        if (req.url.indexOf('/oauth2callback') > -1) {
                            const qs = new url.URL('http://localhost:3000' + req.url).searchParams;
                            res.end('Authentication successful! Please return to the console.');
                            server.destroy();
                            const { tokens } = await this.oAuth2Client.getToken(qs.get('code'));
                            await this.writeCredentialsToFile(tokens);
                            resolve(tokens);
                        }
                    }
                    catch (err) {
                        return reject(err);
                    }
                })
                .listen(3000, () => {
                    opn(authorizeUrl, { wait: false }).then(cp => cp.unref());
                });
                destroyer(server);
        });
    }

    async run(fileName) {
        let tokens;
        try {
            tokens = await this.readCredentialsFromFile();
        }
        catch (err) {
            tokens = await this.authenticate();
        }

        this.oAuth2Client.setCredentials(tokens);

        this.oAuth2Client.on('tokens', async (newTokens) => {
            const updatedTokens = Object.assign(tokens, newTokens);
            await this.writeCredentialsToFile(updatedTokens);
        });

        const drive = google.drive({
            version: 'v3',
            auth: this.oAuth2Client
        });

        const res = await drive.files.create(
            {
                requestBody: {
                    name: path.basename(fileName),
                    mimeType: 'image/png'
                },
                media: {
                    mimeType: 'image/png',
                    body: fs.createReadStream(fileName)
                }
            },
            {
                onUploadProgress: (e) => console.log(e)
            }
        );
        console.log(res.data);
    }
};

new Upload().run(process.argv[2]).catch(console.error);

File: package.json

{
  "name": "googleapis-upload",
  "description": "googleapis-upload",
  "engines": {
    "node": ">=8"
  },
  "dependencies": {
    "googleapis": "36.0.0",
    "opn": "5.3.0",
    "server-destroy": "1.0.1"
  }
}

[AVaksman]

Hi @persona0591 ,

Yes I am able to reproduce the same issue. I also found a problem that causes such behavior. I will submit a Pull request shortly with a fix.

[persona0591]

Hi @AVaksman, thank you!

[AVaksman]

:+1: It will be available with the next release shortly.

[persona0591]

Hi @AVaksman Do you know when the next release will be available? For example: in a week or a month? We're eager to start using your fix :)

[AVaksman]

@persona0591 there was another fix for a related issue googleapis/nodejs-googleapis-common/pull/67. I am assuming there will be next release of nodejs-googleapis-common shortly followed by updating dependencies here.

[l-barbosa]

Hi @AVaksman!

As far as I can see, nodejs-googleapis-common has been updated with the fix with 0.7.1.
Do you know how long it would take for google-api-nodejs-client (googleapis) to be updated to include this? I see an update 3 days ago but it was to include version 0.7.0.
I have new features ready to be rolled out in my project but they are currently significantly impacted by this issue so I would like to understand how often dependencies get updated or how the process works.

Thanks in advance for your time!

[JustinBeckwith]

We're actively working on cutting a release 👍