Aws-sdk-cpp: TransferManager: Signal directory download for a nested directory structure

[bawejakunal]

In the aws-cpp-sdk-transfer-tests the example code triggers directoryDownloadSignal once 3 file downloads have started.

How should I do the same for a unknown directory structure in which the number of files (including nested files) is unkown ?

[singku]

If you don't know the structure of the directory, there is no way to determine when to send out the signal. You should use list object method to find out (possibly recursion needed) the dir structure first if you want achieve similar objection.

[bawejakunal]

I am counting the number of keys to download as below:

uint64_t countFiles(const Aws::String &bucketName,
                    const Aws::String &prefix,
                    const std::shared_ptr<Aws::S3::S3Client> s3Client) {
    // List files request
    Aws::S3::Model::ListObjectsV2Request listObjectsRequest;
    listObjectsRequest = listObjectsRequest.WithBucket(bucketName).WithPrefix(prefix);

    Aws::S3::Model::ListObjectsV2Result listObjectsResult;
    uint64_t keyCount = 0;
    do {
        auto listObjectsOutCome = s3Client->ListObjectsV2(listObjectsRequest);
        if (listObjectsOutCome.IsSuccess() == false) {
            std::cout << "Error listing objects from S3 Bucket" << std::endl;
            return 0;   // error listing objects
        }
        listObjectsResult = listObjectsOutCome.GetResult();
        keyCount += listObjectsResult.GetKeyCount();    // increment count
        if (listObjectsResult.GetIsTruncated()) {
            listObjectsRequest = listObjectsRequest.\
                WithContinuationToken(listObjectsResult.GetNextContinuationToken());
        }
    } while (listObjectsResult.GetIsTruncated());
    return keyCount;    // total objects found for downloading
}

uint64_t keyCount = countFiles();

and then trigger the signal as below:

if (directoryDownloads.size() == keyCount) {
                    directoryDownloadSignal.notify_one();
 }

It gets stuck after creating 1000 directories on my local disk. No files are written and no further directories are created

[bawejakunal]

I updated my code as below but still stuck at the same problem. Only 1000 directories are created and then it gets stuck after that. No files are downloaded.

transferConfig.transferStatusUpdatedCallback = 
        [&](const TransferManager* transferManager, const TransferHandle& handle)
{
    std::lock_guard<std::mutex> m(semaphoreLock);
    // if file download succeeds
    if (handle.GetStatus() == TransferStatus::COMPLETED) {
        bytesTransferred += handle.GetBytesTransferred();
        // all files downloaded
        if (--keyCount == 0)
            directoryDownloadSignal.notify_one();
    }
};

// specify transferInitCallback
transferConfig.transferInitiatedCallback = [&](const TransferManager* transferManager,
                                      const std::shared_ptr<TransferHandle>& handle) {};

TransferManager transferManager(transferConfig);
transferManager.DownloadToDirectory(localDir, bucketName, prefix);

{
     std::unique_lock<std::mutex> locker(semaphoreLock);
     directoryDownloadSignal.wait(locker);
}

As directory creation is handled within the DownloadToDirectory() call and not more than 1000 are being created could it be a bug in SDK ?

[singku]

Did you reset your callback function there the second time assignment? Could you please attach a trace level log? Regard the 1000 directories, did they contain sub directories? How about the total size of these 1000 directories?

[bawejakunal]

Yes I have changed my transferInitiatedCallback function to an empty lambda function.

Yes the 1000 directories that I mentioned about do contain varying levels of sub directories, but those sublevels have not been created on my local disk.

I think the following segments of log should be helpful which can be seen several times in the trace log file:

[DEBUG] 2017-06-28 21:09:15 AWSAuthV4Signer [11312] Canonical Request String: GET
/
continuation-token=&delimiter=%2F&list-type=2&prefix=012f%2F
---------------------------------------------

[TRACE] 2017-06-28 21:09:15 WinHttpSyncHttpClient [10384] Making GET request to uri https://bucket-name-here.s3.amazonaws.com?continuation-token=&delimiter=%2F&list-type=2&prefix=0147%2F

---------------------------------------------
[DEBUG] 2017-06-28 21:09:19 AWSClient [13060] Request returned error. Attempting to generate appropriate error codes from response
[TRACE] 2017-06-28 21:09:19 AWSErrorMarshaller [13060] Error response is <?xml version="1.0"?>
<?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>InvalidArgument</Code>
    <Message>The continuation token provided is incorrect</Message>
    <ArgumentName>continuation-token</ArgumentName>
    <RequestId>HEXADECIMALREQUESTID</RequestId>
    <HostId>BASE64HOSTID</HostId>
</Error>

[WARN] 2017-06-28 21:09:19 AWSErrorMarshaller [13060] Encountered Unknown AWSError
InvalidArgument
The continuation token provided is incorrect:
[WARN] 2017-06-28 21:09:19 AWSClient [13060] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
[DEBUG] 2017-06-28 21:09:19 AWSClient [13060] Server time is Thu, 29 Jun 2017 01:09:18 GMT, while client time is Thu, 29 Jun 2017 01:09:19 GMT

As you can see the continuation token is empty from requests originating from recursive calls within the DownloadToDirectory method and hence the incorrect continutation token. So I guess that's a bug ?

[singku]

https://github.com/aws/aws-sdk-cpp/issues/176 This is exactly what you want

[bawejakunal]

@singku I guess my problem originates within sdk here: https://github.com/aws/aws-sdk-cpp/blob/31e26620e9844abbe160b7bbbd4cc7b5f65d4062/aws-cpp-sdk-transfer/source/transfer/TransferManager.cpp#L793

As you can see in the countFiles() method mentioned above I have used the ListObjectsRequest with continutation token only to count the number of object keys to be fetched, and that works fine. The trouble begins after the transferManager.DownloadToDirectory() call is made.

[bawejakunal]

I also defined errorCallBack

// errorCallBack
transferConfig.errorCallback = [&](const TransferManager *transferManger,
    const TransferHandle &handle, const Aws::Client::AWSError<Aws::S3::S3Errors> &error) {
    std::lock_guard<std::mutex> m(semaphoreLock);
    std::cout << error.GetMessage() << std::endl;
    std::cout << error.GetExceptionName() << std::endl;
};

The output is as below:

Unable to parse ExceptionName: InvalidArgument Message: The continuation token provided is incorrect
InvalidArgument

This is on v1.0.62 from NuGet package manager on VS2015.

[singku]

I tested with multiple files upload/download. I met the similar problem, uploaded 1017 files, then failed. I logged the error msg then found out that a file can't be opened. But the file is there. So I realized I might have a limitation on my system to open over 1024 files at the same time. It is! Then I changed the limit by ulimit, then it worked. There might be a bug, I am looking at it.

[bawejakunal]

@singku did you also try downloading over a 1000 files via DownloadDirectory. Can you confirm if you are also facing the continuation token error ?

Thanks

[singku]

I am doing uploading. Can you log the token string from result?

On Thu, Jun 29, 2017 at 7:21 PM Kunal Baweja [e-mail]
wrote:

@singku https://github.com/singku did you also downloading over a 1000
files via DownloadDirectory. Can you confirm if you are also facing the continuation
token error ?

Thanks

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-sdk-cpp/issues/593#issuecomment-312157450,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADTsCHbp7haDgpo4ZWXSQ3zg0FHsMP7jks5sJFu-gaJpZM4OGwBM
.

>

Best Regards!

Liang.

Software Engineer
Game Developer

[bawejakunal]

@singku I am facing the issue in DownloadToDirectory as mentioned above.
In the trace logs I can see an empty continuation token is being sent from the sdk to aws s3 api.

[singku]

Can you double check your open files limitation to help us narrow down the
problem? Also log the token from result, instead of request.

On Fri, Jun 30, 2017 at 7:00 AM Kunal Baweja [e-mail]
wrote:

@singku https://github.com/singku I am facing the issue in
DownloadToDirectory as mentioned above.
In the trace logs I can see an empty continuation token
https://github.com/aws/aws-sdk-cpp/issues/593#issuecomment-311837837 is
being sent from the sdk to aws s3 api.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-sdk-cpp/issues/593#issuecomment-312274963,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADTsCN9UNIBF13myReHxSKLIZeEWpb5Oks5sJP-OgaJpZM4OGwBM
.

>

Best Regards!

Liang.

Software Engineer
Game Developer

[bawejakunal]

The SDK code within DownloadToDirectory is not logging the token from response, only logging for requests.

But the logs within my self written countFiles look like below:

[DEBUG] 2017-06-30 12:20:04 SAMPLE::DEBUG [10404] continuation-token-received: 1s6tVcc5ZXsfRCXdz2pqi/jBLcw2ZCchKDsYy1ukNdu2cdy/0J5ijpOAMSAnUvsCJtNr0vJUKgeM=

----------------------------------------------------------------

[TRACE] 2017-06-30 12:20:04 WinHttpSyncHttpClient [10404] Making GET request to uri https://bucketname.s3.amazonaws.com?continuation-token=1s6tVcc5ZXsfRCXdz2pqi%2FjBLcw2ZCchKDsYy1ukNdu2cdy%2F0J5ijpOAMSAnUvsCJtNr0vJUKgeM%3D&list-type=2&prefix=

In the above case tokens are same but I guess that is not how it happens within the SDK implemented function TransferManager.DownloadToDirectory, somehow it ends up sending an empty continuationToken.

[singku]

So you are certain you received the token from reponse, but when making
request, it doesn't appear in the request made(from your previous post
above ), but appeared in your own log. Is that so?

On Fri, Jun 30, 2017 at 9:35 AM Kunal Baweja [e-mail]
wrote:

The SDK code within DownloadToDirectory is not logging the token from
response, only logging for requests.

But the logs within my self written countFiles look like below:

[DEBUG] 2017-06-30 12:20:04 SAMPLE::DEBUG [10404] continuation-token-received: 1s6tVcc5ZXsfRCXdz2pqi/jBLcw2ZCchKDsYy1ukNdu2cdy/0J5ijpOAMSAnUvsCJtNr0vJUKgeM=

---

[TRACE] 2017-06-30 12:20:04 WinHttpSyncHttpClient [10404] Making GET request to uri https://1010-leo-hashed.s3.amazonaws.com?continuation-token=1s6tVcc5ZXsfRCXdz2pqi%2FjBLcw2ZCchKDsYy1ukNdu2cdy%2F0J5ijpOAMSAnUvsCJtNr0vJUKgeM%3D&list-type=2&prefix=

In the above case tokens are same but I guess that is not how it happens
within the SDK implemented function TransferManager.DownloadToDirectory,
somehow it ends up sending an empty continuationToken.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-sdk-cpp/issues/593#issuecomment-312314775,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADTsCGVXglfHzmCrz_tTZuvlQ87bzFzGks5sJSO9gaJpZM4OGwBM
.

>

Best Regards!

Liang.

Software Engineer
Game Developer

[bawejakunal]

Yes
Maximum open files per session limit on my windows machine is 16384 so I guess file handles aren't a problem here as of now.

[singku]

@bawejakunal I got a similar problem when testing over 1000 files when downloading, even I increased the fd limitation, it didn't work. I have a solution for uploading, and now I am working on fixing downloading.

[singku]

@bawejakunal I am able to run my program with a minor fix. But basically, you can follow DownloadToDirectory method to download a nested directory. This method has a bug with setting Token, it should be NextContinuationToken instead of ContinuationToken

[bawejakunal]

@singku any updates on this fix and probable timeline for nuget release ?

[bawejakunal]

@singku additionally I noticed a performance issue in java sdk which I guess might come up in C++ sdk too, so should I open up a new issue for that ?

[singku]

@bawejakunal we are doing code review for this change

[bawejakunal]

@singku any updates on this one ? I see that the code has been updated to use GetNextContinuationToken() but the performance bottleneck (similar to java sdk) remains. Is that being sorted out ?

[singku]

@bawejakunal Sorry for the delayed update. We are working on fixing this now.

[singku]

@bawejakunal Fixed at https://github.com/aws/aws-sdk-cpp/commit/22944324f7da1bc361e1ef988bd4db5159f5ee32