Aws-sdk-ios: Retrieve list of more than 1,000 S3 objects

[dougboberg]

When calling

[[[AmazonClientManager s3] listObjects: req] continueWithSuccessBlock:^id(BFTask *task) {
  AWSS3ListObjectsOutput *listoutput = task.result;
}];

and the [listoutput.isTruncated boolValue] returns true the listoutput.nextMarker is nil.

This seems to be similar to the behaviour someone reported for the go SDK: https://github.com/aws/aws-sdk-go/issues/152

In iOS Obj-C, what is the correct way to retrieve more than 1,000 S3 items if nextMarker is not available? Does the iOS SDK have pagination (I cannot find reference to it)? I don't need to download the objects, just simply list them.

Further, can an example be added to the iOS SDK that shows the correct way to recursively setup BFTask (or AWSTask) to retrieve all S3 entries?

[yosuke-matsuda]

According to Amazon S3 API Reference:

NextMarker: This element is returned only if you have delimiter request parameter specified. If response does not include the NextMaker and it is truncated, you can use the value of the last Key in the response as the marker in the subsequent request to get the next set of object keys.

You can refer to Executing Multiple Tasks in AWS Mobile SDK for iOS Developer Guide for finding out how to execute multiple tasks in sequence.

[dougboberg]

Thank you for the assistance. I tried with the delimiter before my original post but apparently I was using it incorrectly. Now I have arbitrarily decided to use "\" as the delimiter and the nextMarker is populated as expected. Thanks again.

With regard to the Executing Multiple Tasks in AWS Mobile SDK, I have read through this multiple times and cannot figure out how to make it work using AWSTask and the nextMarker pattern.

I understand that with a set of known tasks to perform I could bundle them and call taskForCompletionOfAllTasks. However I think the nextMarker lends itself to some sort of do-while loop. This is demonstrated in the Java documentation at Listing Keys Using the AWS SDK for Java. I need to retrieve the first 1,000 batch before I get the nextMarker which will let me construct the second listObjects request. (Correct?)

Below is the code I am using currently. I want it to retrieve and return the ENTIRE list of objects. A nice big array of thousands of objects.

By reading it I know that my solution won't work... and it surely doesn't. But I don't know what to change. I am recursively calling back to the same function with the updated nextMarker but am not sure how to construct the result to gather all of the records with each pass.

Ideally the function would not need a "helper" property on the instance just to store it's results as they are built. I hope I could construct it in a way to be self-contained. This is why I thought a demo would help.

- (BFTask* ) getFiles: (NSString*) prefix withOffsetMarker:(NSString*)marker
{
    AWSS3ListObjectsRequest *req = [[AWSS3ListObjectsRequest alloc] init];
    [req setBucket: kAMAZON_ORGBUCKET];

    req.prefix = prefix;
    req.delimiter = @"\\";

    if (nil != marker) {
        [req setMarker:marker];
    }

    return [[[AmazonClientManager s3] listObjects: req] continueWithSuccessBlock:^id(BFTask *task) {

        AWSS3ListObjectsOutput *listoutput = task.result;

        NSLog(@"OBJECTS: %d    AT MARKER: %@", (int)[listoutput.contents count], listoutput.nextMarker);

        // isTruncated means we only got a partial list
        if ([listoutput.isTruncated boolValue]) {

            // call the current function again to get more results
            BFTask *anothertask = [self getFiles:listoutput.prefix withOffsetMarker:listoutput.nextMarker];
            AWSS3ListObjectsOutput *moreoutput = anothertask.result;

            // merge our current output with more output
            [listoutput setContents:[listoutput.contents arrayByAddingObjectsFromArray:moreoutput.contents]];
        }

        return task;
    }];
}

[yosuke-matsuda]

taskForCompletionOfAllTasks is used to execute multiple tasks in parallel. You should read the "In Sequence" section and adopt the pattern.

[dougboberg]

Yup, I understood that I should use the "In Sequence" section.

I don't see a way to adapt the for (int32_t i = 0; i < 100; i++) shown in the "In Sequence" section to a do-while loop of nextMarker and isTruncated params for the listObjects call.

Closing issue without resolving how to retrieve the entire list of objects in a single returned array.

[mjtedwards]

I know this is an old one - dougboberg I think your issue was not using an out-of-band array to store the results in. For each successive call, you are reading the results from the task just created (anothertask.result), which may not have completed yet. This means that 'moreoutput' is likely empty. The dispatch of a new task is an async operation.

I solved this by having a mutable array passed through recursive calls to my own version of getFiles: withOffsetMarker:. Each iteration, within the scope of continueWithSuccessBlock, I append the results to it. Once I reached the last page (isTruncated=false), I then signal that the paging operations have completed.