Aws-sdk-js: new issue - InvalidSignatureException ?

[fprivitera]

in the last few days I am having this error. Can it be related to the new version of the module?

"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details." - code:InvalidSignatureException

[lsegal]

Can you provide more details on where this error is coming from? What service are you using? Code to reproduce this behavior would be helpful.

[fprivitera]

Right.
We are using the the module version 1.7.0 ,

var AWS = require('aws-sdk');
AWS.config.update({
accessKeyId: configApp().aws.accessKeyId,
secretAccessKey: configApp().aws.secretAccessKey
});
AWS.config.update({
region: configApp().aws.region
});
var dynamodb = new AWS.DynamoDB();
var params= ....
var r = dynamodb.getItem(params, callBack);

[fprivitera]

btw, If I restart the server, it disappears for a while (some hours).

[lsegal]

Are you using IAM roles for EC2 instances, or are you actually specifying credentials with the following as you showed in the sample?

AWS.config.update({
accessKeyId: configApp().aws.accessKeyId,
secretAccessKey: configApp().aws.secretAccessKey
});

If you are hardcoding credentials like the above, you shouldn't be having any expiration issues (since hardcoded credentials should not expire).

[fprivitera]

hardcoded, in a configuration file

[lsegal]

Are the credentials in the configuration file being rotated by any chance? If not, I'm not sure what the issue could be. We did change some semantics around credential management but nothing related to statically loaded credentials. Are you sure this was not happening before?

[fprivitera]

no we didn't at all. The credentials are valid infact if I restart the work again. configApp() it is a stupid function that return an object where variable are hardcoded, and there is any kind of reload/update logic.

[lsegal]

Note that we recently had a forum post about this issue but it was related to IAM roles for EC2 instances, not hardcoded credentials.

Unfortunately I'm not sure how to narrow this down any further. If you can provide more information about the version of Node.js you are running, the server software, etc., that might help.

[mhart]

Could it be that the date/times are out, messing up the signature? Make sure you've got your times sync'ed correctly on your system.

[fprivitera]

I checked and date time was ok

Il giorno sabato 5 ottobre 2013, Michael Hart ha scritto:

Could it be that the date/times are out, messing up the signature? Make
sure you've got your times sync'ed correctly on your system.

—
Reply to this email directly or view it on GitHubhttps://github.com/aws/aws-sdk-js/issues/168#issuecomment-25746501
.

Confidential information may be contained in this message. If you are not
the address indicated in this message, please do not copy or deliver this
message to anyone. In such case, you should notify the sender immediately
and delete the original message.

[gianluca-sabena]

Hi, I'm working with fprivitera, so I'm talking about the same issue.

We have two micro server on EC2 Irland with:

= Ubuntu precise (12.04.2 LTS)
= Node js v0.10.13
= npm 1.3.2
= aws-sdk 1.7.0

I need to better investigate logs, but this error could manifest a first time just after midnight when date change. We keep server time updated with ntpdate pool.ntp.org so normally we have less then 3 sec of delay.

Error example

"err": {
    "message": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'POST\n/\n\ncontent-length:122\ncontent-type:application/x-amz-json-1.0\ndate:20131005T000005Z\nhost:dynamodb.eu-west-1.amazonaws.com\nuser-agent:aws-sdk-nodejs/1.7.0 linux/v0.10.13\nx-amz-date:20131005T000005Z\nx-amz-target:DynamoDB_20120810.GetItem\n\ncontent-length;content-type;date;host;user-agent;x-amz-date;x-amz-target\ne615b36***********************************************'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20131005T000005Z\n20131005/eu-west-1/dynamodb/aws4_request\na800*************************************'\n",
    "code": "InvalidSignatureException",
    "name": "InvalidSignatureException",
    "statusCode": 400,
    "retryable": false
}

[gianluca-sabena]

Switch back to aws-sdk-js 1.6.0 solved this problem

[mhart]

This looks like a likely culprit: https://github.com/aws/aws-sdk-js/commit/0f2e7232b92d15b6c4577e6cc3e5715387b4aa3f

[mhart]

(the datetime.substr(0, 8) call specifically - if that's cached, that will fail when the date changes - ie, at midnight)

[lsegal]

Good catch Michael! That is definitely the issue. We started coaching SigV4 requests for performance but are incorrectly caching the date. The fix for this is pretty easy. I will take a look in a bit.

[mhart]

Yeah - I think it's pretty straightforward - #169 should fix it

[lsegal]

Thanks for reporting this @fprivitera and @gsabena. I just pushed out a patchlevel release of aws-sdk v1.7.1 which you should be able to npm install. We've added regression tests above for this issue and will be much more vigilant about tracking changes like these in the future so that we don't introduce other similar bugs.

Also thanks to @mhart for the quick patch!

[zkimmel]

Thanks for posting this issue. Our dynamo connection died at midnight UTC and we were baffled as to why.

[lsegal]

@zkimmel can you confirm that 1.7.1 and/or 1.8.0 resolved the issue for you? We have a similar issue (#171) being reported and it would be helpful to rule out some of the culprits. Thanks!

[zkimmel]

@lsegal we just applied patch 1.8.0, will wait for the next midnight switchover to find out if it works or not.

[lsegal]

Re-opening because this is still an issue. We have a fix in the works that should correctly resolve.

[lsegal]

I just pushed [email protected] which should resolve this issue. The root cause can be found in the above commit. In short, our performance optimization patches (which improved request performance by ~2x on subsequent calls) added caching, which was not correctly being invalidated when the expired access keys were changed. An earlier patch this issue fixed invalidation on date, but we missed invalidation on the access key ID.

[zkimmel]

Seems to be working -- we haven't re-encountered the issue. Thanks much!

[colinmathews]

I think this issue might still be outstanding.

I'm using the 1.12 version of the aws-sdk package. When I try to make calls on OpsWorks for one set of credentials I get an InvalidSignatureException, but when I use another set it succeeds. Both sets of credentials are IAM users using the same IAM policy. There doesn't seem to be any structural different between the credential secrets except that the set that works has a '+' in it and the one that fails doesn't.

Here's a snippet of the exception details:

ERROR: InvalidSignatureException: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

The Canonical String for this request should have been
'POST
/

content-length:2
host:opsworks.us-east-1.amazonaws.com
x-amz-date:20131109T160149Z
x-amz-target:OpsWorks_20130218.DescribeStacks

content-length;host;x-amz-date;x-amz-target
[REMOVED]'

The String-to-Sign should have been
'AWS4-HMAC-SHA256
20131109T160149Z
20131109/us-east-1/opsworks/aws4_request
[REMOVED]'

UPDATE: Today I tested the credentials that weren't working and they worked successfully. Maybe it was a combination of the current date and the credentials (like some character wasn't escaped properly?). I was really methodical in my testing, but I of course could be going crazy...

[bennett000]

Using 1.8.1, and one of the newer 2.x-rc series (I can dig up the version after I cool off) triggers this problem _if_ there is a + character in the secret key. Big thanks to @colinmathews for mentioning that, I would have gone insane otherwise. It just took me a few minutes to generate a key without a +.

The initial key I was using works fine with SES, but not through the AWS sdk with Dynamo. I have _not_ tried other services.

I was initially using the newest version that npm install gives me, but I downgraded to 1.8.1 as it was referenced in this thread as a known valid version.

[lsegal]

I have tested this with a secret key that has + in it and cannot reproduce the issue on the latest version of the SDK. Any more information about this issue (minimal code that reproduces) would help.

[colinmathews]

No problem, @bennett000!
I don't have my test code at this moment, but I know that I later saw success with a + (see my update above). I was always on 1.12 and didn't alter my code, but a failed signature started working again. My comment above mentioned how maybe it is time-based, but I guess it could also be if some AWS servers are running different code and validating the signature slightly differently.

[mhart]

@bennett000 how are you instantiating your config? It's not a shell escaping issue is it? (ie, doing AWS_SECRET_ACCESS_KEY=asdf+asdf instead of AWS_SECRET_ACCESS_KEY="asdf+asdf")

[bennett000]

@mhart good question. AWS.config.loadFromPath

@colinmathews I'm running ntpd on my instance, as my own systems are time sensitive. ntpq -p looked reasonable. I can paste output if need be.

@lsegal I will update back to npm's current library, and copy some snippets in. I should have a reply in the next few hours, if not sooner, just tying up some other loose ends!

[bennett000]

@lsegal here's more detail

• Region: us-east-1
• Endpoint: dynamodb.us-east-1.amazonaws.com
• sslEnabled: true
• Instance Type: Micro
• AWS SDK Version: 2.0.0-rc12 (also tried 1.8.1)

config.json:

{
    "endpoint": "dynamodb.us-east-1.amazonaws.com",
    "sslEnabled": true,
    "accessKeyId": "akid",
    "secretAccessKey": "secret",
    "region": "us-east-1"
}

In the _working_ case akid is an alphanumeric string, and secret is also an alphanumeric string

In the _failing_ case akid is an alphanumeric string, and secret is an alphanumeric string with _two (2)_ + characters that are _not_ adjacent to each other, and are neither the first, nor the last characters of the string.

Both sets of credentials belong to the _same_ IAM user. Using credentials that do not have two pluses, the AWS library generates a valid hash.

Connection code:

AWS.config.loadFromPath('./etc/config.json');

db = new AWS.DynamoDB();

Query Function:

function query(dataParams) {
    'use strict';
    var d = Q.defer();

    db.query(dataParams, function(err, data) {
       if(err) {
           d.reject(err);
       } else {
           d.resolve(data);
       }
    });

    return vow.promise;
}

ntpd status:

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+atiniui.nhm.org 173.255.215.209  3 u  509 1024  377   67.674   -1.951   1.108
-ntp2.h.vporn.co 184.22.153.11    2 u  916 1024  377   29.971   -8.843   0.580
 rev-18-85-44-59 18.26.4.105      2 u  740 1024    0   13.959    3.718   0.000
*mail.ggong.info 164.67.62.194    2 u  838 1024  377   73.517    0.370   2.259
+europium.canoni 140.203.204.77   2 u  451 1024  377   76.987   -0.367   0.434

_note_ I have removed a private ntp server from the above output

Please let me know if I can provide more detail, or attempt to reproduce this error with other code.

[mhart]

Just to see if it's an issue with aws-sdk or something in your setup, try using the aws4 lib as a sanity check.

Try this (after an npm install aws4):

var http = require('http'),
    aws4 = require('aws4')

// If you don't already have them in env variables:
var credentials = {accessKeyId: '<access key id>', secretAccessKey: '<secret key>'}

var opts = aws4.sign({
  service: 'dynamodb',
  headers: {
    'Content-Type': 'application/x-amz-json-1.0',
    'X-Amz-Target': 'DynamoDB_20120810.ListTables',
  },
  body: '{}'
}, credentials)

http.request(opts, function(res) { res.pipe(process.stdout) }).end(opts.body)

Should result in something like:

{"TableNames":[]}

(you can add a region param in the options you pass to aws4.sign if you want to use something other than us-east-1)

[lsegal]

I ran this to try to reproduce, but the request was successful:

var AWS = require('aws-sdk');

var db = new AWS.DynamoDB({credentials: new AWS.TemporaryCredentials()});

function findSuitableKeys() {
  db.config.credentials.refresh(function(err) {
    if (!err && db.config.credentials.secretAccessKey.match(/^.+\+.+\+.+$/)) {
      console.log("Found secret key:", db.config.credentials.secretAccessKey.replace(/[^+]/g, '*'));
      db.listTables(function(err, data) {
        console.log(err, data);
      });
    } else {
      setTimeout(findSuitableKeys, 1000);
    }
  });
}

console.log("Searching for suitable keys...");
findSuitableKeys();

Output:

Searching for suitable keys...
Found secret key: **+******************+******************
null { TableNames: [ 'sessions', 'test' ] }

[bennett000]

@mhart thanks. That failed for me... however @lsegal 's script found a key, and worked perfectly.

I'm going to assume that either I'm an idiot, and messed up somewhere else with my 'bad' credentials, or that it's an obscure edge case. I would investigate further, but I really need to wrap up my work tonight. If anyone is still interested in me trying more with my 'bad' credentials I can set aside an hour tomorrow, and some time this week-end.

Otherwise, thanks for the quick replies, and sorry for wasting your respective times.

[mhart]

@bennett000 I suspect if it's happening with both the aws-sdk lib and the aws4 lib, then it's very likely to be something to do with your setup (which I guess you already suspect)...

If you ever figure out what it was, let us know!

[manprash]

I am getting this issue again. I am using aws-sdk though and not in js. My aws.rb in initializers looks like
require 'aws-sdk'
AWS.config({
access_key_id: 'key',
secret_access_key: 'secret'
})

and model looks like
class User < AWS::Record::HashModel
....
end

I get the same exact error when I do the following in the controller
@users = User.all
@users.each

I would appreciate any help, thanks.

[lsegal]

@manprash I would recommend opening an issue against aws/aws-sdk-ruby if you have an issue with the Ruby SDK. That said, if you've gotten the same issue across multiple SDKs, that should probably rule out this specific issue, since this was implementation specific to the JS SDK.

[satpal82bhandari]

I found this issue today with node.js aws-sdk.

it was working fine till night. suddenly it shows same error as in the discussion.

message: 'The request signature we calculated does not match the signature you
provided. Check your AWS Secret Access Key and signing method. Consult the serv
ice documentation for details.nnThe Canonical String for this request should h
ave beenn\'GETn/2012-09-25/jobs/%7BId%7Dnncontent-length:0nhost:elastictran
scoder.us-west-2.amazonaws.comnx-amz-date:20140412T003626Znncontent-length;ho
st;x-amz-datene3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\
'nnThe String-to-Sign should have beenn\'AWS4-HMAC-SHA256n20140412T003626Zn
20140412/us-west-2/elastictranscoder/aws4_requestn5aeff672d178552b07051ec152b44
be5f98fef63a623a79c341bc36c24a13f19\'n',
code: 'InvalidSignatureException',
time: Sat Apr 12 2014 06:06:27 GMT+0530 (India Standard Time),
statusCode: 403,
retryable: false,
_willRetry: false } 'InvalidSignatureException: The request signature we calcu
lated does not match the signature you provided. Check your AWS Secret Access Ke
y and signing method. Consult the service documentation for details.nnThe Cano
nical String for this request should have beenn\'GETn/2012-09-25/jobs/%7BId%7D
nncontent-length:0nhost:elastictranscoder.us-west-2.amazonaws.comnx-amz-date
:20140412T003626Znncontent-length;host;x-amz-datene3b0c44298fc1c149afbf4c8996

[lsegal]

@satpal82bhandari can you provide more information about this issue? Are you experiencing this on EC2? Are you using IAM roles for credential configuration? What version of Node.js are you using? What version of the SDK?

[brettswift]

Same thing here. Brand new to this API.. amazon forced my hand as my synology NAS is unable to delete my vaults and I'm getting billed.

code:

var AWS = require('aws-sdk');

AWS.config.loadFromPath('./config.json');

var glacier = new AWS.Glacier();
var accountId  = '8456-8938-4714';
var params = {
  accountId: accountId,
  limit: '10',
  marker: '0',
};

glacier.listVaults(params, function(err, data) {
  if (err) {
    console.log(err,err.stack);
    return;
  }
  console.log(data);
});

{
  "accessKeyId": "~~",
  "secretAccessKey": "~~", 
  "region": "us-east-1"
}

aws-sdk versions tried: 2.0.0-rc13 and 1.18.0

error:

~~
InvalidSignatureException: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method.
~~
code: 'InvalidSignatureException',
  time: Mon Apr 21 2014 09:01:52 GMT-0600 (MDT),
  statusCode: 403,
  retryable: false
~~

[sabrinaluo]

same issue, [email protected] [email protected]

InvalidSignatureException: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

The Canonical String for this request should have been
'POST
/2015-03-31/functions/usercity_getRegionalTableByLocale/invocations
Qualifier=PROD
host:lambda.ap-northeast-1.amazonaws.com
x-amz-content-sha256:dcf4f50cca02685575a3c860c453eb09c66000b481359c9870e737cce54ad222
x-amz-date:20151204T084257Z

host;x-amz-content-sha256;x-amz-date
dcf4f50cca02685575a3c860c453eb09c66000b481359c9870e737cce54ad222'

The String-to-Sign should have been
'AWS4-HMAC-SHA256
20151204T084257Z
20151204/ap-northeast-1/lambda/aws4_request
8fa220f683adb6517b2df8f5558075060e023aabc1bb0308963977e75f161dda'

[compwright]

Same here with version 2.102.0:

{ InvalidSignatureException: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

The Canonical String for this request should have been
'POST
/

host:logs.us-east-1.amazonaws.com
x-amz-content-sha256:bc57f9a1b9550796843253f0df0271345bffa818f3b067bbed3402377acb0fd3
x-amz-date:20170827T072221Z
x-amz-target:Logs_20140328.DescribeLogStreams

host;x-amz-content-sha256;x-amz-date;x-amz-target
bc57f9a1b9550796843253f0df0271345bffa818f3b067bbed3402377acb0fd3'

The String-to-Sign should have been
'AWS4-HMAC-SHA256
20170827T072221Z
20170827/us-east-1/logs/aws4_request
422964b9aec83cb77088a89f80dea065183aae2ecddbe79c194748cd5db7ab48'

    at Request.extractError (/usr/lib/nodejs/node-v6.11.2-linux-x64/lib/node_modules/bbva_email/node_modules/aws-sdk/lib/protocol/json.js:48:27)
    at Request.callListeners (/usr/lib/nodejs/node-v6.11.2-linux-x64/lib/node_modules/bbva_email/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
    at Request.emit (/usr/lib/nodejs/node-v6.11.2-linux-x64/lib/node_modules/bbva_email/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/usr/lib/nodejs/node-v6.11.2-linux-x64/lib/node_modules/bbva_email/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/usr/lib/nodejs/node-v6.11.2-linux-x64/lib/node_modules/bbva_email/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/usr/lib/nodejs/node-v6.11.2-linux-x64/lib/node_modules/bbva_email/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /usr/lib/nodejs/node-v6.11.2-linux-x64/lib/node_modules/bbva_email/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/usr/lib/nodejs/node-v6.11.2-linux-x64/lib/node_modules/bbva_email/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/usr/lib/nodejs/node-v6.11.2-linux-x64/lib/node_modules/bbva_email/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/usr/lib/nodejs/node-v6.11.2-linux-x64/lib/node_modules/bbva_email/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
    at Request.emit (/usr/lib/nodejs/node-v6.11.2-linux-x64/lib/node_modules/bbva_email/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/usr/lib/nodejs/node-v6.11.2-linux-x64/lib/node_modules/bbva_email/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/usr/lib/nodejs/node-v6.11.2-linux-x64/lib/node_modules/bbva_email/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/usr/lib/nodejs/node-v6.11.2-linux-x64/lib/node_modules/bbva_email/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /usr/lib/nodejs/node-v6.11.2-linux-x64/lib/node_modules/bbva_email/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/usr/lib/nodejs/node-v6.11.2-linux-x64/lib/node_modules/bbva_email/node_modules/aws-sdk/lib/request.js:38:9)

[TylerSustare]

I'm getting the same issue using version 2.5.2

[TylerSustare]

I need to update my comment, I had a typo in my keys when I got this error. I am not getting this error after using the correct keys. 👈 shame :(

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.