Superagent: using with paginated api requests

[bonesoul]

So I'm basically using node-huntergatherer (https://github.com/erictj/node-huntergatherer) for one of my projects for my api pagination needs but as the library is not updated any more, i'm considering to switch superagent.

But what I couldn't find out yet is how to support paginated api calls with it;

To make it more clear, node-huntergatherer supports them natively. Here is the explanation from the project;

It's ideally suited to the types of calls you see via APIs. Here's one example from the Etsy API (http://developer.etsy.com/docs) that you need to make if you want to gather all of a users' active listings:

http://openapi.etsy.com/v2/listings/active?limit=50&offset=0 http://openapi.etsy.com/v2/listings/active?limit=50&offset=50 http://openapi.etsy.com/v2/listings/active?limit=50&offset=100 http://openapi.etsy.com/v2/listings/active?limit=50&offset=150 http://openapi.etsy.com/v2/listings/active?limit=50&offset=200 http://openapi.etsy.com/v2/listings/active?limit=50&offset=250

Traditionally, you'd loop through each call, process the results of that call, and then call the next offset, process, ad nauseum.

Not only is this super inefficient, it's slow and a pain in the ass to re-code all the time. Especially when this pattern occurs in many APIs, and all that changes between them for the most part are the URL and how you process the results. So let's abstract out everything else!

And lastly, with HunterGatherer each of these calls happen simultaneously, up to the maximum number of simultaneous connections allowed based on the remote API terms of usage. Non-blocking FTW!

Is this possible with superagent and I'll be really glad if i can see a working sample.

[seb-thomas]

@bonesoul Hey I'm doing a similar thing, what did you go with in the end? ty

[kornelski]

Superagent by itself doesn't have such specific functionality. Our recommendation is to wrap creation of Request into something higher-level, e.g.

function _requestForOffset(path, offset) {
     return superagent.get(`https://api.example.com/${path}`).query({offset});
}

async function getAllPages() {
    const limit = 50;
    let offset = 0, total;
    do {
        const response = await _requestForOffset("listings/active", offset);
        total = response.body.total;
        offset += limit;
    } 
    while(offset < total);
}