Async: Question: how to paginate with token? (or recursively call async function and pass through parameters)

[tommedema]

AWS API calls typically respond with a pagination token. When present, another call to the same API should be made with that token, until a token is no longer supplied. This is typically done using recursion.

I would like to use async (or another library) to simplify this approach. I had imagined the following psuedo code:

  await paginate(async (lastToken, recurse, page) => {
    const { certificates, token } = await listCertificates({nextToken: lastToken})
    handleCertificates(certificates)
    token && recurse(token)
  })

This illustrates how something as complicated as token based pagination could be accomplished with a few lines of code. I've looked into async and saw some relevant functions (e.g. until), but have not found a way to recursively pass the output of one call to the next (i.e. pass in the token to the next call).

Is this possible with async? Or is there a library that's more suitable for this?

[tommedema]

After scanning through the documentation once again, I hoped this would do it.

async.forever(
    (next, lastToken) => {
        const { certificates, token } = await listCertificates({ nextToken: lastToken })
        handleCertificates(certificates)
        token && next(null, token)
    }
)

However, unfortunately it appears that a call like next(null, token) does NOT cause the token to be accessible in the callback function (next, lastToken) => {}, and only next is defined (lastToken is undefined).

Is there an alternative method or library that can help?

[tommedema]

Ultimately this worked but it's a little bit ugly. What do you think?

import promisify from 'pify'
import foreverSync from 'async.forever'

const forever = promisify(foreverSync, { errorFirst: false })

let lastToken
const result = await forever(async next => {
    const { certificates, token } = await listCertificates({nextToken: lastToken})
    handleCertificates(certificates)
    lastToken = token
    next(lastToken ? null : new Error('no more pages'))
})

[Seojiwoong]

How about making recursive function?
I think that it doesn't need async.js....

[tommedema]

@Seojiwoong this would result in quite some boilerplate, whereas the above is cleaner with forever

[aearly]

doWhilst is good for this kind of thing.

let results = []
let nextToken
doWhilst(
  (next) => {
    getPage(url, nextToken, (err, body) => {
      if (err) return next(err);
      nextToken= body.nextToken
      results = results.concat(body.objects)
      next()
    })
  },
  () => nextToken,
  (err) => {
    // results now has all pages
  })

Although, with async/await, it's easier to use a plain do/while:

let results = []
let nextToken
do {
  const body = await getPage(url, opts, nextToken)
  nextToken = body.nextToken
  results = results.concat(body.objects)
} while (nextToken)
// results now has everything

[tommedema]

@aearly I like how async/await makes something like do/while so simple to read again; thanks!