Google-api-nodejs-client: Gmail performance issue getting messages with attachments since v28

[albert-gonzalez]

Hi!

Environment details

• OS: Ubuntu 18.04.1
• Node.js version: 8.12.0
• npm version: 6.2.0
• googleapis version: 34.0.0

Steps to reproduce

I am testing the method gmail.users.messages.get, and I think that there is a performance issue since v28. I get a message with a 7MB attachment using this code

gmail.users.messages.get({
        auth: client,
        userId: 'me',
        id: id,
        format: 'raw'
    });

With v24 the time is around 3s. The time output of a call is:

real    0m2,849s
user    0m0,501s
sys     0m0,052s

With v27 the time is around 4s. Time output:

real    0m3,629s
user    0m1,457s
sys     0m0,924s

Since v28, the time is around 16-20s. Time output:

real    0m15,368s
user    0m10,911s
sys     0m5,026s

Can you help me, please? If i am doing something wrong or you need more info, please tell me.

Thank you!

[albert-gonzalez]

Hi again,

it seems that gzip is causing the big performance issue, if I remove the line:

https://github.com/googleapis/nodejs-googleapis-common/blob/master/src/apirequest.ts#L230

The time is 2-3 seconds slower than v27, but a lot faster than v34. Probably there are other things that make it slower.

Best Regards!

[grant]

@albert-gonzalez Are you looking to disable gzip in the Gmail API request header?

[albert-gonzalez]

Hi,

I was trying to figure out what was the performance issue. Then, I found that disabling the gzip the performance improved a lot and I told you because maybe it would help. But I opened the issue because the performance of old versions is a lot better than the new ones. The performance problem started at v27 or v28.

Thank you!

[remidenele]

Hi,

I also have performance concerns when using gmail.users.messages.get and gmail.users.messages.attachments.get.

By doing some profiling, I noticed that most of the time consumed comes from a Buffer.concat operation in axios http adapter.

Below are the elapsed times to retieve a 10MB attachment from a Gmail email with different methods:

| using [email protected] | using request-promise-native | using [email protected] but commenting lines 183 to 186 in [email protected] http adapter |
| --- | --- | --- |
| 102141.546ms | 3541.013ms | 4042.996ms

It seems to be related to the introduction of the axios dependency in [email protected] to change the transport used for HTTP requests from request to axios

[bradvogel]

Bump on this. We're hitting it too!

[uberism]

Hi,

I also have performance concerns when using gmail.users.messages.get and gmail.users.messages.attachments.get.

By doing some profiling, I noticed that most of the time consumed comes from a Buffer.concat operation in axios http adapter.

Below are the elapsed times to retieve a 10MB attachment from a Gmail email with different methods:
using [email protected] using request-promise-native using [email protected] but commenting lines 183 to 186 in [email protected] http adapter
102141.546ms 3541.013ms 4042.996ms

It seems to be related to the introduction of the axios dependency in [email protected] to change the transport used for HTTP requests from request to axios

I can confirm that this is the case as well. I suppose the workaround is to use request-promise-native.

[JustinBeckwith]

👋 We are actively looking into this by using node-fetch under the hood instead of axios. More coming soon :)

[JustinBeckwith]

oooookay. The change to node-fetch (along with a lot of other stuff... ) has hit master, but we haven't cut a release yet. Would anyone be willing to give this a go with master and see what the perf impact looks like? I really appreciate the help and the patience folks!

[albert-gonzalez]

Hi! I have just tested it with an email with an attachment of 13 MB. With master the result is:

real    0m7,633s
user    0m2,766s
sys     0m0,908s

And with v36 the result is:

real    1m58,053s
user    1m41,921s
sys     1m5,620s

The performance improvement is really amazing!

Great work and thank you!

[JustinBeckwith]

SWEET. I'm going to go ahead and close this out. If y'all keep running into issues - please do let us know!

[bradvogel]

Thanks all! Just to be clear - what google-api-nodejs-client release is this performance regression fixed in? Is the performance back to the baseline measurement before the v28 regression?

[JustinBeckwith]

This should be in the next release (37), which will come out this week. It's a very big release (webpack support!) So it'll take a bit of verification.