Aws-sdk-net: When using DynamoDBContext.FromScanAsync the return type AsyncSearch<T> does not expose the pagination token

[bittercoder]

Expected Behavior

Currently unless using the Document mid-level API directly there is no way to implement pagination during a search.

I expect that the PaginationToken should be available as a property on the AsyncSearch type, so that I can use it page through results returned from the Scan.

Current Behavior

The difference in behavior would be to expose PaginationToken as a public property on this class:
https://github.com/aws/aws-sdk-net/blob/master/sdk/src/Services/DynamoDBv2/Custom/DataModel/AsyncSearch.cs#L40

Possible Solution

The change could be implemented in this way:

    public string PaginationToken 
    {
        get { return DocumentSearch.PaginationToken; }
    }

Context

This issue is impacting me because I would prefer not to have to take a dependency in my classes on the Document API if I can just use the DynamoDBContext by itself.

Your Environment

• Service assembly and version used: AWSSDK.DynamoDBv2 nuget package v3.3.4.13
• Operating System and version: Windows 10
• Visual Studio version: VS 2017
• Targeted .NET platform: .Net Core 1.1

.NET Core Info

• .NET Core version used for development: 1.1
• .NET Core version installed in the environment where application runs: 1.1
• Output of dotnet --info:

.NET Command Line Tools (1.0.4)

Product Information:
Version: 1.0.4
Commit SHA-1 hash: af1e6684fd

Runtime Environment:
OS Name: Windows
OS Version: 10.0.15063
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\1.0.4

• Contents of project.json/project.csproj: not relevant

[PavelSafronov]

This is by design. DynamoDBContext abstracts away a fair bit of the service to make object-oriented operations easier. So the Scan operation returns IEnumerable<T> which automatically performs the pagination for you.

That being said, you can paginate using the Document model, then use the DynamoDBContext to convert the retrieved Document objects to your POCOs using FromDocument or FromDocuments, as illustrated here:

using (var client = new AmazonDynamoDBClient(pavelCreds, usEast1Region))
using (var context = new DynamoDBContext(client))
{
    var table = context.GetTargetTable<Actor>();

    string paginationToken = "{}";
    do
    {
        var result = table.Scan(new ScanOperationConfig
        {
            Limit = 1,
            PaginationToken = paginationToken
        });
        var items = result.GetNextSet();
        paginationToken = result.PaginationToken;

        var actors = context.FromDocuments<Actor>(items);
        foreach (var actor in actors)
        {
            Console.WriteLine(actor.Name);
        }
    }
    while (!string.Equals(paginationToken, "{}", StringComparison.Ordinal));
}

[bittercoder]

Thanks for the feedback @PavelSafronov - I had already implemented your above solution to work around the issue before you even posted :) In my implementation I start with a null paginationToken, rather then an empty JSON array, but it still seems to work.

I do still wonder if the abstraction is a little leaky on DynamoDBContext then because the method:

context.FromScanAsync<T>(new ScanOperationConfig{ ... }, ...)   -> AsyncSearch<T>

Takes a ScanOperationConfig, which allows specifying a PaginationToken - which doesn't really work in that instance because you never had a chance to grab one (unless you construct the pagination token yourself) - which is what led me to think the inability to get back the pagination token didn't make much sense.

Also looking at the result type of FromScanAsync<T> which is AsyncSearch<T> it does not implement automatic paging via IEnumerable<T> either - this is the API it exposes (notice the comments too mentioning PaginationToken - even more confusing!)

public class AsyncSearch<T>
{
    //
    // Summary:
    //     Flag that, if true, indicates that the search is done
    public bool IsDone { get; }

    //
    // Summary:
    //     Initiates the asynchronous execution to get the next set of results from DynamoDB.
    //     If there are more items in the Scan/Query, PaginationToken will be set and can
    //     be consumed in a new Scan/Query operation to resume retrieving items from this
    //     point.
    //
    // Parameters:
    //   cancellationToken:
    //     Token which can be used to cancel the task.
    //
    // Returns:
    //     A Task that can be used to poll or wait for results, or both. Results will include
    //     the next set of result items from DynamoDB.
    public Task<List<T>> GetNextSetAsync(CancellationToken cancellationToken = default(CancellationToken));
    //
    // Summary:
    //     Initiates the asynchronous execution to get all the remaining results from DynamoDB.
    //
    // Parameters:
    //   cancellationToken:
    //     Token which can be used to cancel the task.
    //
    // Returns:
    //     A Task that can be used to poll or wait for results, or both. Results will include
    //     the remaining result items from DynamoDB.
    public Task<List<T>> GetRemainingAsync(CancellationToken cancellationToken = default(CancellationToken));
}

Cheers,

Alex

[PavelSafronov]

You're right, the AsyncSearch object is leaky. This was our attempt to support an async version of IEnumerable<T> Scan<T>(...) operation. For that reason, I'm not very comfortable adding PaginationToken to AsyncSearch: suddenly there's this big difference between the sync and async calls, where one is able to handle pagination and the other isn't.

Not sure what the best approach here is, will keep thinking about this.

[firenero]

@PavelSafronov So now there are no ways to implement a pagination using DynamoDBContext with async queries? Using GetNextSetAsync() is not an option because I can't get and set any tokens to share search state between my API calls.

The only way I see now is to manage search state manually on the application level saving AsyncSearch instances somewhere but it's not a really good option from the architecture point of view. Or am I missing something?

[kgrvr]

Any updates on this? There's any way we can get Pagination Token from QueryAsync?

[dietermijle]

Would like some solution for this too. +1

[BrunoPace96]

Any update about this?

[niraj-bpsoftware]

Any update on this?

[rlisnoff]

My team and I would also really like to see a solution to this, right now if you want pagination _and_ the ability to easily deserialize results from a query, you need to use an IAmazonDynamoDB client to make the async query with a LastEvaluatedKey, and then you need to construct a DynamoDBContext using that same client to use its FromDocument() function. From the literature I've read, it seems you don't want to be creating this context class too often as it's apparently sort of heavy.

I'd really like to see either more pagination control given to the DDB Context, or something more handy for converting results from the lower level client into the entity type we've stored.

[Doolali]

Are there any updates on this as a feature? Would be great to have working!

[trevore-q4]

Why is using the IDynamoDBContext the recommended approach if it doesn't support pagination across requests? This is not a small feature that is missing.

[github-actions[bot]]

We have noticed this issue has not recieved attention in a year. We will close this issue for now. If you think this is in error, please feel free to comment and reopen the issue.