Aws-sdk-php: Default credential provider failing to resolve

[ArthurGuy]

I am using the defaultProvider to sign an elastic search request but a recent update to the SDK or guzzle has caused it to to start failing.

I have isolated the problem to the chaining going on in the default provider, it seems that if the last item in the chain fails the whole request fails with the following error.

The promise was rejected with reason: Invoking the wait callback did not resolve the promise

I am using the role based credentials (instanceProfile) but if that option appears last as it does with the default provider the request fails, if I reverse the last two options the promise resolves

This Fails

$provider = CredentialProvider::chain(CredentialProvider::env(), CredentialProvider::ini(), CredentialProvider::instanceProfile(), CredentialProvider::ecsCredentials());
call_user_func($provider)->wait();

This returns a credentials object

$provider = CredentialProvider::chain(CredentialProvider::env(), CredentialProvider::ini(), CredentialProvider::ecsCredentials(), CredentialProvider::instanceProfile());
call_user_func($provider)->wait();

Has anyone encountered this or knows why it's happening?

[cjyclaire]

@acinader Taking a look, working on the fix.

[cjyclaire]

@acinader A fix has been merged into master, could you try it to see if that fixes the issue? SDK cannot change the resolving order in the default credential provider because it's unified across all SDKs.
Yet this bug might be caused by ECS credential failure fails to be wrapped inside the promise to make it through the chain correctly. If the fix doesn't work, please let me know.

[ArthurGuy]

I have updated to use the master branch but this hasn't made any changes, I am still getting the same error as before.

[cjyclaire]

@ArthurGuy Hmm, interesting, could you provide you php version, OS information then I could see if I could reproduce it there? Also may I ask which version of SDK start causing this break for you?

Edited, to clarify the problem, are you suggesting that with current chain, using assume role always fails? (InstanceCredentialProvider cannot be resolved)?

[ArthurGuy]

The sdk went from 3.19.28 to 3.19.31 and the guzzle promise library went from 1.2.0 to 1.3.0, this was the update for us that caused the failure.

We are running on Ubuntu 16.04 LTS and php 7.0.8.

The library is being pulled in as part of the laravel framework but for the previous test I did explicitly pull in the master branch of the sdk.

[cjyclaire]

@ArthurGuy So for sdk went from 3.19.28 to 3.19.31, there is no changes happen in Credentials. I'll take a look at Guzzle to see if something happen there. Thanks for the information, I'll see if I could reproduce. Just a quick workaround, if you are using instanceCredentials only, you could try use instanceProfile directly.

[ArthurGuy]

I have placed user credentials onto the server which has gotten around this, if a solution isn't readily available I will switch to the instanceProfile method.

[cjyclaire]

@ArthurGuy I looked at guzzle, I don't think there is any related changes neither. I couldn't reproduce this issue locally. Could you provide more information? Is using instanceProfile working for you?

[ArthurGuy]

instanceProfile works, its just that combination in the initial message that's behaving incorrectly.

I have tried a few combinations and its definitly the guzzlehttp/promises library, if I pull it back to 1.2.0 everything works but if I leave it to go to 1.3.0 everything fails.

[cjyclaire]

@ArthurGuy Interesting, thank you for your patience on this and appreciated that, could you open an issue in guzzle repo as well? I'll try again with guzzle 1.3.0 to see If I could reproduce

[dittto]

@cjyclaire I've encountered an issue with this as well. We've tried running master with Guzzle 1.3.0 but still get the same error raised above.

I broke apart the defaultProvider() and there seems to be an issue with chaining and memoizing together:

$provider = Aws\Credentials\CredentialProvider::defaultProvider();
call_user_func( $provider )->wait(); // fails with "The promise was rejected with reason: Invoking the wait callback did not resolve the promise"

$env_creds = Aws\Credentials\CredentialProvider::env();
call_user_func( $env_creds )->wait(); // fails with "Could not find environment variable credentials in AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY"
$ini_creds = Aws\Credentials\CredentialProvider::ini();
call_user_func( $ini_creds )->wait(); // fails with "Cannot read credentials from /.aws/credentials"
$ecs_creds = Aws\Credentials\CredentialProvider::ecsCredentials();
call_user_func( $ecs_creds )->wait(); // fails with "Error retrieving credential from ECS (cURL error 28: Connection timed out after 1001 milliseconds..."
$instance_creds = Aws\Credentials\CredentialProvider::instanceProfile();
call_user_func( $instance_creds )->wait(); // returns valid credentials

$chain = Aws\Credentials\CredentialProvider::chain( $env_creds, $ini_creds, $ecs_creds, $instance_creds );
call_user_func( $chain )->wait(); // returns valid credentials

$simple_memo = Aws\Credentials\CredentialProvider::memoize( $instance_creds );
call_user_func( $simple_memo )->wait(); // returns valid credentials

$chain_memo = Aws\Credentials\CredentialProvider::memoize( $chain );
call_user_func( $chain_memo )->wait(); // fails with "The promise was rejected with reason: Invoking the wait callback did not resolve the promise"

[cjyclaire]

@ArthurGuy Interestingly, I still couldn't reproduce the issue with a fresh launched ec2 instance:
Ubuntu Server 16.04 LTS (HVM), SSD Volume Type - ami-a9d276c9 with uname -a:

uname -a
Linux ip-172-31-39-144 4.4.0-45-generic #66-Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

My dependencies (via composer)

aws/aws-sdk-php        3.19.33 AWS SDK for PHP - Use Amazon Web Services in your PHP...
guzzlehttp/guzzle      6.2.2   Guzzle is a PHP HTTP client library
guzzlehttp/promises    1.3.0   Guzzle promises library
guzzlehttp/psr7        1.3.1   PSR-7 message implementation
mtdowling/jmespath.php 2.3.0   Declaratively specify how to extract elements from a ...
psr/http-message       1.0.1   Common interface for HTTP messages

My php version

php -v
PHP 7.0.8-0ubuntu0.16.04.3 (cli) ( NTS )
Copyright (c) 1997-2016 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2016 Zend Technologies
    with Zend OPcache v7.0.8-0ubuntu0.16.04.3, Copyright (c) 1999-2016, by Zend Technologies

@dittto Thanks for the info., may I ask are you also finding this failing since guzzle 1.3.0? What's you php version and OS environment?

[dittto]

@cjyclaire we're running the following:
CentOS Linux release 7.2.1511 (Core) with:

Linux i-7d8f51bd 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

The dependencies are the same version numbers as those you've listed, apart from my test was using aws-sdk-php on master.

The PHP version is:

PHP 5.6.25 (cli) (built: Aug 26 2016 15:39:31)
Copyright (c) 1997-2016 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
    with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies

If I get some time free then I may try creating a little project with just this bug in it, as currently it's part of a giant Wordpress-based project.

[ArthurGuy]

@cjyclaire stupid question but you were trying to get credentials from a role and there was nothing else in place to take precedence?

[cjyclaire]

@ArthurGuy Yeah, it is using the instance metadata credentials, also since it's a newly launch instance with an assume role, I didn't do any customization either.

What I did is require straight forward. I launched a new ec2 instance with assume role that enabled with s3 access. I installed PHP 7 and SDK with all dependencies with composer. And I'm able to perform S3 operations with successful response. Also, given some time, tracking trace, the metadata credential is cached and correctly fetched.

[llernestal]

Hi, I'm getting this issue when creating a pre-signed URL with Aws\S3\S3Client->createPresignedRequest()

Uncaught GuzzleHttp\Promise\RejectionException: The promise was rejected with reason: Invoking the wait callback did not resolve the promise in /vendor/guzzlehttp/promises/src/functions.php:112
Stack trace:
#0 /vendor/guzzlehttp/promises/src/Promise.php(75): GuzzleHttp\Promise\exception_for('Invoking the wa...')
#1 /vendor/aws/aws-sdk-php/src/S3/S3Client.php(313): GuzzleHttp\Promise\Promise->wait()

aws/aws-sdk-php: 3.20.1
guzzlehttp/promises: 1.3.0
php: 7.0.13

uname -a
Linux 48d6274ffe2f 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

I'm using CredentialProvider::defaultProvider(). When I explicitly set guzzlehttp/promises: 1.2.0 it works without a problem.

[stevenscg]

@ArthurGuy @llernestal I'm having the same issue with createPresignedRequest() with almost the same setup as Lars. My aws/aws-sdk-php is 3.19.30.

[cjyclaire]

Thanks for the following inputs, I'll take a second look at Guzzle changes.
Also, may I ask your php extensions loaded? Just curious, is xdebug enabled? Is PHP SDK used directly or integrated with other framework? May I gather those data from your guys?

[llernestal]

@cjyclaire I use the SDK directly, no xdebug enabled.

Loaded PHP modules

[PHP Modules]
calendar
Core
ctype
curl
date
dom
exif
fileinfo
filter
ftp
gettext
hash
iconv
json
libxml
memcache
mysqli
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
pdo_sqlite
Phar
posix
readline
Reflection
session
shmop
SimpleXML
sockets
SPL
sqlite3
standard
sysvmsg
sysvsem
sysvshm
tokenizer
wddx
xml
xmlreader
xmlwriter
xsl
Zend OPcache
zip
zlib

[Zend Modules]
Zend OPcache

[Nizari]

We've had this issue as well! It still worked on version 3.19.21, but somewhere after that all our new servers crashed. As of today we still have this issue when we use the CredentialProvider::defaultProvider()

[cjyclaire]

@Nizari Quick question, does this start since guzzle version upgraded to 1.3.0 ?

[Nizari]

Hmmm, can't say for sure. What guzzle requirements does version 3.19.21 have in composer? If it's below 1.3.0, this could be related

[stevenscg]

I specifically downgraded from guzzlehttp/promises 1.3.0 to 1.2.0 with AWS SDK 3.19.30 and the issue was no longer present.

[sagikazarmark]

I was able to reproduce the issue: https://github.com/reproduce/guzzle/tree/issue1668

After a quick look in the code: the way how coroutines are handled has been changed in 1.3.0, which might be the guilty one.

[Nizari]

Is anyone on this gullze issue?

[ArthurGuy]

It looks like it's nearly been solved.
https://github.com/guzzle/promises/pull/55

[sagikazarmark]

Hi guys!

We are waiting for a final review before merging and tagging 1.3.1 with a fix. I will keep you posted.

[sagikazarmark]

https://github.com/guzzle/promises/releases/tag/v1.3.1 is out containing a fix (confirmed in the reproduction environment)

[cjyclaire]

Closing, thank you all for the efforts in this, appreciate that.

[saxenasam]

Upgraded to guzzlehttp/promises 1.3.1 but still facing this issue when using createPresignedRequest() with iam roles (default provider chain). Please help

PHP Fatal error: Uncaught GuzzleHttp\Promise\RejectionException: The promise was rejected with reason: Invoking the wait callback did not resolve the promise

0 library/AwsV3/vendor/guzzlehttp/promises/src/Promise.php(76): GuzzleHttp\Promise\exception_for('Invoking the wa...')

1 application/business/apps/test.php(15): GuzzleHttp\Promise\Promise->wait()

2 {main}

thrown in application/vendor/guzzlehttp/promises/src/functions.php on line 112

[saxenasam]

please ignore.. I have solved the issue.