Aws-sdk-js: SignatureDoesNotMatch: Signature expired

[pvsrivathsa]

Node.js version 0.10.25

AWS SDK Version latest - 2.0.23

I have an app that is continuously listening to a Queue (SQS) and if there are messages posted in that queue the app will read the message and process it and save some data to S3. When I start the app after about 20 minutes I am getting the following error continuously. Potentially unhandled rejection [160] the number in the bracket indicates the number of times the error occurs. It goes on till like 20000 till I stop the app and restart it.

Potentially unhandled rejection [160] SignatureDoesNotMatch: Signature expired: 20141104T062952Z is now earlier than 20141104T062952Z (20141104T064452Z - 15 min.)
at Request.extractError (/myproject/node_modules/aws-sdk/lib/protocol/query.js:39:29)
at Request.callListeners (/myproject/node_modules/aws-sdk/lib/sequential_executor.js:100:18)
at Request.emit (/myproject/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (/myproject/node_modules/aws-sdk/lib/request.js:604:14)
at Request.transition (/myproject/node_modules/aws-sdk/lib/request.js:21:12)
at AcceptorStateMachine.runTo (/myproject/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /myproject/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/myproject/node_modules/aws-sdk/lib/request.js:22:9)
at Request.<anonymous> (/myproject/node_modules/aws-sdk/lib/request.js:606:12)
at Request.callListeners (/myproject/node_modules/aws-sdk/lib/sequential_executor.js:104:18)

It is not an issue with my system time. My system time is in sync with the time of my EC2 instance. Why am I getting this error? Is it related to SQS or S3?

[lsegal]

Can you print the retryCount property on the response? i.e.,

sqs.theOperation(params, function(err, data) {
  console.log('retry count:', this.retryCount);
});

The full err object would also be helpful.

[pvsrivathsa]

The err object is

{ [SignatureDoesNotMatch: Signature expired: 20141105T071117Z is now earlier than     20141105T071121Z (20141105T072621Z - 15 min.)]
  message: 'Signature expired: 20141105T071117Z is now earlier than 20141105T071121Z (20141105T072621Z - 15 min.)',
  code: 'SignatureDoesNotMatch',
  time: Wed Nov 05 2014 12:56:21 GMT+0530 (IST),
  statusCode: 403,
  retryable: false }

retryCount is 0

[lsegal]

Ahh, it looks like the SDK is not retrying these signature errors. You can workaround this in the SDK for now by adding the following global event:

AWS.events.on('retry', function(resp) {
  if (resp.error.code === 'SignatureDoesNotMatch') {
    resp.error.retryable = true;
  }
});

Let me know if this works for you.

I'll keep this open to track fixing this bug in the SDK. At the very least, the SDK should be retrying expired signature errors, which are occasionally hard to detect, but doable.

[pvsrivathsa]

Is this a bug introduced in 2.0.23 ? Will a downgrade help? If so, to what version?

[lsegal]

@pvsrivathsa no, this is not a regression. The SDK has never retried 4xx responses unless explicitly whitelisted. We never whitelisted signature errors because for the most part retrying them will not help. This is a special case where the signature error can be retried. Some services have a special error class for expired signatures (in which case we whitelist them), but this one is not, so it was never added. I'm marking it as a bug because it _should_ work, but it's also somewhat of a feature addition too, since the SDK never did this before.

[lsegal]

Thanks for reporting. The above workaround should not be needed after the next release.

[binoculars]

It's a system time issue http://stackoverflow.com/questions/18295185/invalidsignatureexception-signature-expired-timestamp-is-now-earlier-than-t

[ghost]

I experienced this issue today, and fortunately, the AWS NodeJS SDK now has a config option to fix the clock offset by itself:

http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Config.html#correctClockSkew-property

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.