Aws-sdk-go: Example to use QueryPages

[alwindoss]

Version of Go and AWS SDK: Latest

Questions:

1. Are there any examples on how to use QueryPages(Primary Question)
2. In NodeJS we have to write the recursive query based on whether LastEvaluatedKey is set or no, is QueryPages an easier way to do that?

[jasdel]

Hi @alwindoss are you able to use the DynamoDB client's QueryPages pagination method to paginate the responses? The SDK should automatically be using the LastEvaluatedKey fields to paginate the query response.

[alwindoss]

@jasdel The documentation for QueryPages says that I need to return a boolean flag from the callback depending on whether I want to continue. I would only know if I have to continue based on LastEvaluatedKey right? In this case how will QueryPages take care of this automatically?

[jasdel]

For the SDK's paginators that Boolean is only if you want your application to continue within the scope of the available pages. The QueryPages will automatically stop paging when it knows it reached the last page.

The returned bool is only if your application wants to stop paging early. This is a confusion that we are looking to correct in the v2 sdk. It uses an iterator pattern instead of callbacks for pagination.

[alwindoss]

@jasdel in that case we need to update the documentation to clearly state this and it would help if we add an example for this case.
I will attempt to do this if you too think it should be done and I will raise a PR.

[alwindoss]

@jasdel In fact I had another question along the same lines. Let's say the QueryPages iterates multiple times. Would the QueryOutput object content be replaced or updated?

[jasdel]

Thanks for the feedback on the documentation of the pagination. We'll look into cleaning that up.

Each iteration of QueryPages will return a new QueryOutput. It should be safe to store and mutate this value as needed without worry of the paginator mutating QueryOutput also.

[lekki]

@jasdel is it a bug or an expected behaviour? Limit param seems to be ignored on paging.

[diehlaws]

Hi @lekki, apologies for the long delay in response from our end on this. The Limit parameter will limit each Query call to evaluate (not return) at most the given number of items. When combined with the paginator for this call, the Limit parameter will evaluate at most the given number of items per page rather than evaluating at most the given number of items across all pages. As a result, if you want to evaluate for example 200 keys across 10 pages you'd want to use a combination of Limit and maximum page values that totals out to 200 (e.g. limit=20 and pages=10, limit=50 and pages=4, limit=100 and pages=2, etc).

[alwindoss]

I just noticed a tag was added which reads closing-soon-if-no-response
Is the documentation available (which is the primary reason this issue was created)

[diehlaws]

Thanks for the reply @alwindoss. Can you provide clarification on what you wanted included in the docs for QueryPages? Was it just a note stating that returning false on its function isn't required if you want to iterate through all available pages?

[alwindoss]

This issue is 2 part:

1. We need to add an example of how to use QueryPages, see the example in Query
2. We need to change the documentation in Query to say "if you want to paginate over the result set if Last Evaluated key has data then use QueryPages". Today the documentation for Query says I need to write the logic to paginate(this is how it's done in nodejs) however since query documentation does not point to QueryPages for pagination it is confusing. Hence I raised this issue.

I hope this clarifies.

[diehlaws]

Thanks for the clarification @alwindoss! Please keep in mind that the description for the Query API call (the portion between the function statement and the example) is provided to the SDKs from the service's API reference which is why it suggests that pagination must be done manually - in order to change the content of the API call's description in the SDK's docs the DynamoDB team would have to update their service's API reference to include a mention of the paginator, which would result in inaccurate documentation from the service API's perspective since the paginator is not a part of the service's API.

Paginators for given service operations are not built into the service's API but are instead customizations for each AWS SDK, which is why the Query API call's documentation does not make a mention of using the QueryPages function. The documentation for QueryPages sits directly below Query within the SDK's docs and provides an example of how to use this paginator. Since this paginator is a customization for the AWS SDK for Go we can update the description for this function in the docs, however any changes made to the Query description from our end would be undone on the following release due to the way in which the code is generated surrounding the API calls and their documentation.

[alwindoss]

@diehlaws I understood about the part that the generic documentation cannot be modified. However not sure if I understood the latter where you say even if we make the changes in the SDK API documentation it will be undone in the following release.

I thought we add documentation in the Go source files and the documentation is picked from here.

One other request was to add an example. I think this should be possible. Right?

[diehlaws]

Thanks for the reply @alwindoss! The documentation you see for Query in the AWS SDK for Go API Reference comes from lines 3044-3123 of service/dynamodb/api.go, which are generated from the same source as DynamoDB's Query API Reference page. As a result, making changes in api.go to the documentation for this call from our end would result in the changes being overwritten by the source for the service's API reference.

By contrast, the source for the QueryPages documentation is controlled by our team, since this paginator is a customization for the AWS SDK for Go. As a result, documentation for this paginator is best left in a separate entry from the API call itself so the description and code example for the paginator sit directly below the documentation for the API call.

With that in mind, we can add a note in the documentation for QueryPages stating that, in order to iterate through all available pages, the function that is part of the input for the paginator should always return true; we can also add a second code example within the QueryPages documentation showing this scenario in addition to the existing example that shows how to iterate through a maximum of 3 pages.

[alwindoss]

@diehlaws your suggestion makes sense. I wasn't aware that the documentation for Query was pulled from the reference document.