Aws-sdk-js: Easier access to request ids

[rclark]

Most of the time that I interact with AWS support it involves trying to troubleshoot the cause of failure for particular requests. The support team always wants to know request ids, but these are not trivial to access.

Two ways that this could be easier would be:

• if error objects had x-amz-request-id and x-amz-id-2 values attached for the failed request
• if "logger" output included these values

The first option would be excellent, because then any time that we encountered an error, we would have the fuel for better support interaction, without having to do anything special in our applications or log details for every single SDK request that is made.

[chrisradek]

@rclark
As of version 2.2.10 of the SDK, the requestId of the response object is populated on errors, whereas previously it was only populated when an operation succeeded. The response object can be accessed by referencing this in a callback function to an operation.

For example:

s3.getObject({Bucket: 'bucket', Key: 'invalidKey'}, function (err, data) {
  if (err) {
    console.log('RequestId: ' + this.requestId);
  }
});

Is this the method you were already using to get the request ids?

[rclark]

I didn't realize that the response was bound to my callback, but this solution still presents the requirement that your application perform that console.log _in addition to_ whatever it is that is done to handle the error itself (which may lose this context).

This value corresponds to the x-amz-request-id header, correct? Does the response also provide easy access to the x-amz-id-2 header value?

[chrisradek]

@rclark
The response doesn't currently have as direct of a way of accessing the x-amz-id-2 header value like it does for the request id. It is possible to look at the headers on the response to grab this header, though I can understand the benefit to having easier access to this information.

I'm marking this as an enhancement to expose the 2nd header value on the response. May also look into adding this information on the error itself.

[markstos]

I'm using 2.2.16, and the syntax of this.requestId returns undefined. I tried this:

var s3 = new AWS.S3();

s3.getObject({Bucket: 'bucket', Key: 'invalidKey'}, function (err, data) {
  if (err) {
    console.log('RequestId: ' + this.requestId);
    console.log('Err:  %j', err);
    console.log('Data:  %j', data);
  }
});

Easy access to the requestId is important, as the official docs state the request ID is key to getting help from Amazon support when requests fail.

[chrisradek]

@markstos
Using the example you provided, this should refer to a response object, and requestId should be populated. I tested using your example and was able to print the request id. Can you share what this is when you run your above code?

[markstos]

@chrisradek Thanks for the response. Here's the output of the command above. I've added a line to dump out this with console.log.

{ request: 
   { domain: null,
     service: { config: [Object], isGlobalEndpoint: false, endpoint: [Object] },
     operation: 'getObject',
     params: { Bucket: 'bucket', Key: 'invalidKey' },
     httpRequest: 
      { method: 'POST',
        path: '/',
        headers: [Object],
        body: '',
        endpoint: [Object],
        region: 'us-east-1' },
     startTime: Thu Nov 12 2015 17:42:07 GMT-0500 (EST),
     response: [Circular],
     _asm: { currentState: 'complete', states: [Object] },
     _haltHandlersOnError: false,
     _events: 
      { validate: [Object],
        afterBuild: [Object],
        restart: [Object],
        sign: [Object],
        validateResponse: [Object],
        send: [Object],
        httpHeaders: [Object],
        httpData: [Object],
        httpDone: [Object],
        retry: [Object],
        afterRetry: [Object],
        build: [Object],
        extractData: [Object],
        extractError: [Object],
        httpError: [Object],
        beforePresign: [Object],
        complete: [Object] },
     emit: [Function: emit] },
  data: null,
  error: 
   { [TimeoutError: Missing credentials in config]
     message: 'Missing credentials in config',
     code: 'CredentialsError',
     time: Thu Nov 12 2015 17:42:08 GMT-0500 (EST),
     originalError: 
      { message: 'Could not load credentials from any providers',
        code: 'CredentialsError',
        time: Thu Nov 12 2015 17:42:08 GMT-0500 (EST),
        originalError: [Object] } },
  retryCount: 0,
  redirectCount: 0,
  httpResponse: 
   { statusCode: undefined,
     headers: {},
     body: undefined,
     streaming: false,
     stream: null },
  maxRetries: 3,
  maxRedirects: 10 }
RequestId: undefined
Err:  {"message":"Missing credentials in config","code":"CredentialsError","time":"2015-11-12T22:42:08.386Z","originalError":{"message":"Could not load credentials from any providers","code":"CredentialsError","time":"2015-11-12T22:42:08.386Z","originalError":{"message":"Connection timed out after 1000ms","code":"TimeoutError","time":"2015-11-12T22:42:08.385Z"}}}
Data:  null

[chrisradek]

@markstos
Based on the error message, it looks like the SDK can't find your credentials before making your call. If that's the case, then the getObject method will never actually send its request to the service and instead return the error you're seeing. Since requestId is returned by whichever service is being called, and no calls are actually being made to a service, requestId is undefined.

Let me know if that clarifies things. If you are providing credentials and still seeing this issue there may be something else going on we can look at.

[markstos]

@chrisradek,

I was following the exact syntax that had been suggested above:
https://github.com/aws/aws-sdk-js/issues/781#issuecomment-154499642

I updated my code to trigger an "Access Denied" case, and I confirmed I got "this.requestId" back for both a getObject call and a putObject call.

Where is this existence of this requestId documented? I would have expected to find a mention of it any or all of these pages, but a search for "requestId" on any of these pages finds no results.

http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Request.html
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3/ManagedUpload.html

[markstos]

@chrisradek, while requestId is populated for getObject and putObject, I found it is not available in the callback passed to s3.upload. That's the method I actually need to debug.

[rclark]

@markstos' issues pretty clearly demonstrate my original point that access to these headers is obscure in the current API.

In the s3.upload case, for example, you're making a bunch of requests and you don't have direct access to all of the AWS.Request objects, which is where (I guess only in the event of an error?) the requestId attribute is exposed.

If instead, the request ids were attached to error objects, an error triggered by one of the requests in a multi-part upload could "bubble-up" to the caller in a predictable way.

[chrisradek]

@markstos
The existence of requestId is documented here:
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Response.html#requestId-property
and called out in the upgrading docs here:
http://docs.aws.amazon.com/AWSJavaScriptSDK/guide/upgrading.html

That said, this won't help you when using some of the high-level abstractions such as s3.upload. I think in this scenario it would be helpful to have the requestId on the error object as well.

[markstos]

In case it's of interest: I go to the bottom of my "Access Denied" issue by disabling SSL on the requests and sending them through the Charles Debug Proxy so I could see the raw HTTP requests and responses. (I was hoping the "logger" output might show that, but it doesn't).

The root cause that was that I had a Key started with a "/", wich I presumed was correct. However, this resolved in a double-slash in the final URL, which caused prefix-pattern-matching rules to fail.

It seems the question of leading-slash-or-not has caused problems for developers using a variety of frontends. Guidance on this point would be welcome in the JavaScript SDK docs would be welcome as well.

Ref: https://encrypted.google.com/search?hl=en&q=should%20AWS%20S3%20keys%20start%20with%20a%20slash%3F#hl=en&q=AWS+S3+key+%22leading+slash%22

[chrisradek]

I've merged #797 that adds requestId to the error object when the error is caused by the service. For S3, it also adds the extendedRequestId field, which is synonymous to x-amz-id-2.

This change will be available via NPM in the next release, or can be pulled in now by installing the SDK from github.

[yhahn]

@chrisradek can we send you donuts?

[onassar]

Little late to the party @chrisradek but I'm using aws-sdk-2.2.10.min.js and a AWS.S3.ManagedUpload for uploading. How can I access the request ids? Don't seem to be available to me (unfort) via this helper class.

[AllanZhengYP]

Hi @onassar
This thread may help you #1688

[onassar]

Thanks @AllanFly120 I'll give that a go; would those headers be exposed for a successful upload as well (so long as I expose them via the CORS rule)? Seems the easiest way to test it :)

[onassar]

I tried this, but still don't see the headers being available in the scope of this. Here's CORS rules:

[AllanZhengYP]

Oh sorry, you are using upload instead of putObject. Because upload calls a serial of operations under the hood, the requestId is unavailable intentionally. If you want to get access to requestId for each of the operations, the quickest way is to use customizeRequests.

var AWS = require('aws-sdk')
AWS.S3.prototype.customizeRequests(function(request) {
    request.on('extractData', function(response){
        console.log('success operation: ', response.request.operation, 'requestId: ', response.requestId)
    });
    request.on('extractError', function(response){
        console.log('fail operation: ', response.request.operation, 'requestId: ', response.requestId)
    });
})

var s3 = new AWS.S3({region: 'us-west-2'});
s3.upload(uploadParams, function(e, d){
    if(!e) {
        console.log(d);
    } else {
        console.log(e);
    }
})

I hope this will help you

[onassar]

Getting AWS.S3.prototype.customizeRequests is not defined. Which version of the SDK makes the customizeRequests method available?

I'm using aws-sdk-2.2.10.min.js
Also, I don't see that the extractData events are changing the e or d objects. How would those request ids be available via those properties?

[onassar]

Incase anyone is curious, release v2.7.1 is the first release to have this method available:
https://github.com/aws/aws-sdk-js/releases/tag/v2.7.1

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.