Google-api-nodejs-client: Batch Request Update

[jamesmhaley]

As mentioned in Issue #290 batch requests were removed with the idea to introduce a new updated way of performing batch requests.

Any update on this? If not, what is the alternative? I potentially need to do 100s of calls via a service account to retrieve the activities of all the user's in our system.

[edwardysun]

+1

[wearhere]

+1

[bradvogel]

+1

[ryanseys]

Hi all, does anyone have a suggestion as to how they would like batch requests to function in this library? One of the big issues I've had with batch is how the API would look and function for users so that it's not overly confusing.

[wearhere]

I never used the old batch request API, but it looks wonderfully concise. I really like its principle that API calls return request objects that could (theoretically, again I never used it) be executed singly or be added to the batch request object–i.e. the batch API is very similar to the non-batch API.

Failing that, I think almost any API would be alright so long as there were code example(s) using the client library. I find Google's HTTP batch request documentation (example) very abstract and have difficulty mapping it to working code.

[robertrossmann]

I was thinking the same way - every API call could in the absence of a callback (or explicitly) return an object that fully describes the action to be taken on the remote endpoint - this includes headers, params, message body etc. These objects could then be sent to an aggregator function that takes an array of such requests, generates the batch request, sends it, parses the response back into individual pieces and delivers the results, as an array.

[sm-github]

An API like the front end batch requests would be great!

gapi client batch

[jamesmhaley]

I agree with the above points with regards to batching requests. An array based system, with headers/a request id on return would be grand!

But, also, I guess the dream for me would be the ability to "join" requests as well. You could potentially solve both problems with one nifty solution.

A use case would be when pulling in Activities: list. An Activity object doesn't always contain all the required data related to that activity. So, I might need to do a separate request for People: listByActivity.

So a request to Activities: list and then a way to "join" other requests to that request.

Something like:

{
    "join":{
        "api":"/people/", // relative API/absolute would both work
        "output":"relatedPeople",
        ...
    }
}

Might return something like this:

{
    "kind": "plus#activity",
    ...
    "relatedPeople":{
        "kind": "plus#peopleFeed",
        ...
    }
}

[ryanseys]

@sparkyfied cool idea, this library is meant to be a generic solution for all Google APIs so it will be a little difficult to create those connections with the metadata we are given about the APIs. An array-based system seems like the most do-able here.

Throwing a code sample out there:

var google = require('googleapis');
var drive = google.drive('v2');
var batcher = google.batcher();
var aReq1 = drive.files.list({ /* options */ }); // no callback = don't fire request
var aReq2 = drive.files.list({ /* options */ });
var aReq3 = drive.files.list({ /* options */ });
batcher.add(aReq1);
batcher.add([ aReq2, aReq3 ]);
batcher.run(function(err, responses) {
  // err: [ error1, null, null ], responses: [ null, response1, response2 ]
});

[wearhere]

Looks good @ryanseys!

[robertrossmann]

Maybe we could even do away without a custom batcher object and instead use arrays directly, then provide that array to some batch request handler:

var google = require('googleapis')
  , drive = google.drive('v2')
  , reqs = []   // Just an array of requests

// Push individual requests into the array
reqs.push(drive.files.list({ /* options */ })) // no callback = don't fire request
reqs.push(drive.files.list({ /* options */ }))
reqs.push(drive.files.list({ /* options */ }))

// Batch it!
google.batch(reqs, function (err, responses) { /* magic inside */ })

Feels more natural to me. Also, this way we would not force people to store the requests in a custom object.

[ryanseys]

@Alaneor I love it!

[ryanseys]

I cannot find any documentation on how to create a batch request manually. Can someone link me to some resources?

[robertrossmann]

@wearhere posted a link that contained pretty detailed HTTP examples here...

[jamesmhaley]

+1 all! What are we looking at timeframe wise @ryanseys?

[ryanseys]

Hey @sparkyfied unfortunately I can't promise anything. I'm a full-time student that works on this in my off-hours so I don't see myself having the time to create this anytime soon (next 3 months), but I'd definitely review and merge PRs that add this functionality. :)

[jamesmhaley]

Cheers @ryanseys!

[jamesmhaley]

@ryanseys Not advanced enough with my Node skills to rewrite your API, so needing this feature I created my own module: https://github.com/wapisasa/Batchelor. Everyone feel free to use and get involved if you need extra features. Obviously interested in using an official batching system when available, but this is a good stop-gap in the mean time!

[ryanseys]

@sparkyfied This is great! Thanks for doing this.

[jamesmhaley]

@ryanseys :+1: No worries, feel free to make use of it in anyway you require :) And any changes you think it needs, go for it :)

[pradeep-mishra]

till it gets implemented in sdk itself try this https://www.npmjs.com/package/google-batch