Aws-sdk-js: Misleading error messages

[LinusU]

It seems that a common pattern in this library is to replace errors with more generic ones when traversing the callback chain backwards.

For example, I had the following problem today: TimeoutError: Missing credentials in config

First of all, it being a TimeoutError seems really weird, especially since it happened right away. But okay, missing credentials in config; that should tell me what it is! So I open up my ~/.aws/credentials and have a look at it. Nope, that looks alright.

Following the first stack trace I open the file aws-sdk/lib/http/node.js and have a look around line 56. It calls in to AWS.util.error which seems to be some kind of global wrapper function which wraps all the errors. I put some console.logs into the error function and I see that the error gets passed thru it three times. Finally creating the frankenstien error which is actually a merge of both the timeout error and the actual error.

[Error: Connection timed out after 1000ms]

{ [TimeoutError: Connection timed out after 1000ms]
  message: 'Connection timed out after 1000ms',
  code: 'TimeoutError',
  time: Fri Dec 18 2015 23:02:26 GMT+0100 (CET) }

{ [TimeoutError: Could not load credentials from any providers]
  message: 'Could not load credentials from any providers',
  code: 'CredentialsError',
  time: Fri Dec 18 2015 23:02:26 GMT+0100 (CET),
  originalError: 
   { message: 'Connection timed out after 1000ms',
     code: 'TimeoutError',
     time: Fri Dec 18 2015 23:02:26 GMT+0100 (CET) } }

{ [TimeoutError: Missing credentials in config]
  message: 'Missing credentials in config',
  code: 'CredentialsError',
  time: Fri Dec 18 2015 23:02:26 GMT+0100 (CET),
  originalError: 
   { message: 'Could not load credentials from any providers',
     code: 'CredentialsError',
     time: Fri Dec 18 2015 23:02:26 GMT+0100 (CET),
     originalError: 
      { message: 'Connection timed out after 1000ms',
        code: 'TimeoutError',
        time: Fri Dec 18 2015 23:02:26 GMT+0100 (CET) } } }

So, somehow CredentialsError: Could not load credentials from any providers together with TimeoutError: Connection timed out after 1000ms got merged into TimeoutError: Missing credentials in config. Great.

At least now I have some more info, Could not load credentials from any providers. But why?

After a lot of digging into how credential loading works inside of this sdk, I finally found the relevant class; SharedIniFileCredentials. The relevant function seems to be refresh whose body is wrapped in a huge try/catch. Hmm, might this be it? I set a console.log inside of the catch block and then start my script again.

[Error: Credentials not set in /Users/linus/.aws/credentials using profile default]

BINGO! :confetti_ball:

I look at my credentials file again and sure enough, the section isn't called [default], but rather the name of the company I work for. Hmm, I think, I guess that's kind of smart. It requires me to be specific before starting something from my company account instead of my private, smart.

Now to the billion dollar question; why couldn't the error message have been kept all the way up so that I'd know what was wrong??

[chrisradek]

@LinusU
Thanks for the feedback!
If I'm understanding correctly, you were eventually able to see the error when debugging:
[Error: Credentials not set in /Users/linus/.aws/credentials using profile default]
but it did not show up in the error that was returned, right?

The order of the error messages in the error that was returned doesn't seem to be the issue so much as the original error being overwritten? At least, from my point of view, Missing credentials in config seems more descriptive than Connection timed out after 1000ms.

Out of curiosity, when you seeing this issue, were you specifying in the SDK how to retrieve credentials?

[LinusU]

Yes, the error presented to me was TimeoutError: Missing credentials in config when it really ought to be CredentialsError: Could not load credentials from any providers.

I was not specifying where to retrieve credentials at all, rather relying on the default lookup.

[matthiasdg]

Maybe this should be a separate issue, but I also find some of the error messages quite misleading.
I'm using S3 in Frankfurt (because closest to where I live) and since a lot of examples don't specify the region for the aws.S3 object, I didn't do this either (it's also not mentioned in the 'get started fast' at https://aws.amazon.com/sdk-for-node-js/ ). When things didn't work (error: not supported authorization mechanism) I found somewhere to add signatureVersion: 'v4'. Now I could perform putObject and getObject without any problem!!

However, when I tried headObject the only message was code: 'BadRequest'. On top of that, headObject works for buckets in Ireland without specifying a region (but only without the signatureVersion parameter).

I then tried using the cli and when I tried get-object for Frankfurt I finally got an error message about adding a region parameter... (head-object's response is again just bad request). Adding this fixed everything, but better error messages (or more consistent behaviour) could have saved me some time.

[LiuJoyceC]

Hi @LinusU
The reason you got a TimeoutError is because the default credential provider chain attempted to source credentials from the EC2MetadataCredentials provider, since it failed to source credentials from the providers before it in the chain. Even though the problem is that the credentials weren't found in the ~/.aws/credentials file, there is no way for the SDK to know that your intention was to source credentials from that file. If the SDK fails to source credentials from that file, it will continue on to the next credentials provider. Eventually it gets to the EC2MetadataCredentials provider, which returns a TimeoutError because it attempts to source credentials from the EC2 Metadata Service, and times out after not getting any response (which will happen if you are not running on an EC2 instance or don't have an IAM role specified in the EC2 instance). Since it is the last credentials provider in the chain, the error surfaces as a TimeoutError. We do then overwrite the error code to be CredentialsError, as you can see in the code property of the error you logged, but the error name did not get changed, and it shows up as [TimeoutError: ...]. I will mark this as a bug, since the error name is intended to change to CredentialsError.

[thomaslwk]

Hi @LiuJoyceC
So how do I fix the credentials error?

[jeskew]

The fix described in https://github.com/aws/aws-sdk-js/issues/849#issuecomment-247725567 has been merged to master and will go out with the next release.

[davidmorabito]

@LinusU I followed all your steps and found this:
[Error: ENOENT: no such file or directory, open '/root/.aws/credentials']

It was driving me crazy, thanks for the help! For some reason AWS looks for the credentials at /root probably because I'm running sudo. Not sure. But, thanks for the troubleshooting tips.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.