Aws-sdk-js: SignatureDoesNotMatch when using results of getSignedUrl

[pcvisit]

Hi,

we keep receiving SignaturDoesNotMatch errors when using urls generated by the getSignedUrl method.

We're using [email protected], node v0.10.33 and our bucket is located in 'eu-central-1'.

Our code (we condensed it down to calling headBucket):

    var s3_config = config.get('s3');
    var s3 = new aws.S3({
        signatureVersion: 'v4',
        region: 'eu-central-1',
        accessKeyId: 'XXX',
        secretAccessKey: 'XXX'
    });

    var params = { Bucket: 'XXX' };
    s3.getSignedUrl('headBucket', params, function(err, data) {
            consumeSignedUrl('GET', null, data, function(err, result) {
                console.log('result', err);
            });
        });
    });

function consumeSignedUrl(method, payload, signedUrl, finishedCb) {
    var parsed = url.parse(signedUrl);
    var body = '';
    var request = http.request({ hostname: parsed.hostname, path: parsed.path, method: method }, function(res) {
        res.on('data', function(chunk) { body += chunk; });
        res.on('end', function() {
            parseString(body, function(err, answer) {
                if (answer["Error"]) {
                    finishedCb(answer["Error"], body);
                } else {
                    finishedCb(null, body);
                }
            });
        });
    });
    if (payload) request.write(payload);
    request.end();
}

It produces this error output:

Code: [ 'SignatureDoesNotMatch' ],
Message: [ 'The request signature we calculated does not match the signature you provided. Check your key and signing method.' ],
AWSAccessKeyId: [ 'XXXXX' ],
StringToSign: [ 'AWS4-HMAC-SHA256\n20150106T145451Z\n20150106/eu-central-1/s3/aws4_request\nb78901e33b473d8ed9b288ed054c4300f60e9db065b5baba9deb9ce81e1407c1' ],
SignatureProvided: [ '09517b4d27b7cc3104e69a3a81251ec05bb459247f029031d8be8de51cc96c42' ],
StringToSignBytes: [ '41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 31 35 30 31 30 36 54 31 34 35 34 35 31 5a 0a 32 30 31 35 30 31 30 36 2f 65 75 2d 63 65 6e 74 72 61 6c 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 62 37 38 39 30 31 65 33 33 62 34 37 33 64 38 65 64 39 62 32 38 38 65 64 30 35 34 63 34 33 30 30 66 36 30 65 39 64 62 30 36 35 62 35 62 61 62 61 39 64 65 62 39 63 65 38 31 65 31 34 30 37 63 31' ],
CanonicalRequest: [ 'GET\n/\nX-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=XXXX%2F20150106%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20150106T145451Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host\nhost:xxxx.s3.eu-central-1.amazonaws.com\n\nhost\nUNSIGNED-PAYLOAD' ],
CanonicalRequestBytes: [ '47 45 54 0a 2f 0a 58 2d 41 6d 7a 2d 41 6c 67 6f 72 69 74 68 6d 3d 41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 26 58 2d 41 6d 7a 2d 43 72 65 64 65 6e 74 69 61 6c 3d 41 4b 49 41 4a 35 57 45 56 49 33 49 46 4e 37 4a 4a 36 52 41 25 32 46 32 30 31 35 30 31 30 36 25 32 46 65 75 2d 63 65 6e 74 72 61 6c 2d 31 25 32 46 73 33 25 32 46 61 77 73 34 5f 72 65 71 75 65 73 74 26 58 2d 41 6d 7a 2d 44 61 74 65 3d 32 30 31 35 30 31 30 36 54 31 34 35 34 35 31 5a 26 58 2d 41 6d 7a 2d 45 78 70 69 72 65 73 3d 39 30 30 26 58 2d 41 6d 7a 2d 53 69 67 6e 65 64 48 65 61 64 65 72 73 3d 68 6f 73 74 0a 68 6f 73 74 3a 70 63 76 69 73 69 74 2d 67 72 61 76 61 74 61 72 73 2d 74 65 73 74 2e 73 33 2e 65 75 2d 63 65 6e 74 72 61 6c 2d 31 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 0a 68 6f 73 74 0a 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44' ],
RequestId: [ 'D3AB0C2FAE7C90AE' ],
HostId: [ 'sSsCUPBSFjkqhJRxLHdyL+NF9y4ln6vq0vgZih5gS6HslMF9FjKMmqdhIauBl0AxAzWOTsavta4=' ]

Are we doing anything wrong?

Thanks
Ruediger

[lsegal]

When you create a signed URL you must make sure to access the URL via the same method that you signed it with. The headBucket command sends a HEAD request, so you will need to use 'HEAD' instead of 'GET'. Note however that 'HEAD' does not return any body response, just headers, so reading the 'data' event from res won't print anything-- but it will fix the signing error.

[pcvisit]

That's great, thanks! Seems we left the 'GET' in there, when we dumbed down our testcase.

Now we're able to make presigned GET and HEAD requests. However when using PUT for putObject we still run into a signing error.

Seems we're still doing something wrong.

Our code looks like this (uses pieces of the code we posted yesterday):

            var content = 'testdata';
            var params = { Bucket: 'XXXX', Key: 'abcde', ACL: 'public-read', ContentLength: content.length };
            s3.getSignedUrl('putObject', params, function(err, url) {
                console.log("url", url);
                consumeSignedUrl('PUT', content, url, function(err, result) {
                    console.log(err);
                });
            });

It gives us the error message below. How can we find out what part of our request isn't matching to the signed version?

Code: [ 'SignatureDoesNotMatch' ],
Message: [ 'The request signature we calculated does not match the signature you provided. Check your key and signing method.' ],
AWSAccessKeyId: [ 'AKIAJ5WEVI3IFN7JJ6RA' ],
StringToSign: [ 'AWS4-HMAC-SHA256\n20150107T190648Z\n20150107/eu-central-1/s3/aws4_request\nf729423eaa0f08c082925491a00ece8e69889fc0e13d826363f283c2e6726cff' ],
SignatureProvided: [ '6fc8a1a40364f5745e30cc6b47dbdae22ff89312185f5353a4e3c3a25bbba887' ],
StringToSignBytes: [ '41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 31 35 30 31 30 37 54 31 39 30 36 34 38 5a 0a 32 30 31 35 30 31 30 37 2f 65 75 2d 63 65 6e 74 72 61 6c 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 66 37 32 39 34 32 33 65 61 61 30 66 30 38 63 30 38 32 39 32 35 34 39 31 61 30 30 65 63 65 38 65 36 39 38 38 39 66 63 30 65 31 33 64 38 32 36 33 36 33 66 32 38 33 63 32 65 36 37 32 36 63 66 66' ],
CanonicalRequest: [ 'PUT\n/abcde\nContent-Length=8&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJ5WEVI3IFN7JJ6RA%2F20150107%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20150107T190648Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host%3Bx-amz-acl&x-amz-acl=public-read\nhost:xxxx.s3.eu-central-1.amazonaws.com\nx-amz-acl:public-read\n\nhost;x-amz-acl\nUNSIGNED-PAYLOAD' ],
CanonicalRequestBytes: [ '50 55 54 0a 2f 61 62 63 64 65 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 3d 38 26 58 2d 41 6d 7a 2d 41 6c 67 6f 72 69 74 68 6d 3d 41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 26 58 2d 41 6d 7a 2d 43 72 65 64 65 6e 74 69 61 6c 3d 41 4b 49 41 4a 35 57 45 56 49 33 49 46 4e 37 4a 4a 36 52 41 25 32 46 32 30 31 35 30 31 30 37 25 32 46 65 75 2d 63 65 6e 74 72 61 6c 2d 31 25 32 46 73 33 25 32 46 61 77 73 34 5f 72 65 71 75 65 73 74 26 58 2d 41 6d 7a 2d 44 61 74 65 3d 32 30 31 35 30 31 30 37 54 31 39 30 36 34 38 5a 26 58 2d 41 6d 7a 2d 45 78 70 69 72 65 73 3d 39 30 30 26 58 2d 41 6d 7a 2d 53 69 67 6e 65 64 48 65 61 64 65 72 73 3d 68 6f 73 74 25 33 42 78 2d 61 6d 7a 2d 61 63 6c 26 78 2d 61 6d 7a 2d 61 63 6c 3d 70 75 62 6c 69 63 2d 72 65 61 64 0a 68 6f 73 74 3a 70 63 76 69 73 69 74 2d 67 72 61 76 61 74 61 72 73 2d 74 65 73 74 2e 73 33 2e 65 75 2d 63 65 6e 74 72 61 6c 2d 31 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 78 2d 61 6d 7a 2d 61 63 6c 3a 70 75 62 6c 69 63 2d 72 65 61 64 0a 0a 68 6f 73 74 3b 78 2d 61 6d 7a 2d 61 63 6c 0a 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44' ],
RequestId: [ '31E153920D4484F8' ],
HostId: [ 'AnjpvOqVEv361VLGKUgTaqysAsDx9gbyh0zVpRIvXMFBG6furdzuxtg+p0KJsGJmo8XToWxhI/8=' ]

Thanks
Ruediger

[lsegal]

In the case of PUT, it's the ACL (and any other headers) that may also need to be passed through in the request itself. You can see this in the response, where S3 has built the CanonicalRequest to include the "x-amz-acl" header: x-amz-acl:public-read. You would need to pass that as a header in your request as well.

Also note that for signing PUT requests you probably do not want to be providing a ContentLength or Body unless you expect the user to be providing a payload of that exact body or content-length.

[pcvisit]

Thank you very much.

I've been setting ContentLength to force node not to use TransferEncoding: chunked as we'd otherwise get a 'A header you provided implies functionality that is not implemented' error.

However, we know the size before signing the request, so it's no problem.

Concerning the signature mismatch, do I understand correctly, that those headers listed under "X-Amz-SignedHeaders" in the error message have to be exactly the same in the put request?

I tried both adding the x-amz-acl header as well as removing the ACL: ... from the call to getSignedURL. No success so far we still get a signature mismatch (here the error message when going without acl):

Code: [ 'SignatureDoesNotMatch' ],
Message: [ 'The request signature we calculated does not match the signature you provided. Check your key and signing method.' ],
AWSAccessKeyId: [ 'AKIAJ5WEVI3IFN7JJ6RA' ],
StringToSign: [ 'AWS4-HMAC-SHA256\n20150108T222615Z\n20150108/eu-central-1/s3/aws4_request\n468b709348b4654eb9f2410dda3e666531ba0c7f9f98faf43a2a0a480fb21afb' ],
SignatureProvided: [ '356e7aed10ef2c5bd9c0780570cb23f0abbc2c19ce481ad02fe0d3ccfea8b7b0' ],
StringToSignBytes: [ '41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 31 35 30 31 30 38 54 32 32 32 36 31 35 5a 0a 32 30 31 35 30 31 30 38 2f 65 75 2d 63 65 6e 74 72 61 6c 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 34 36 38 62 37 30 39 33 34 38 62 34 36 35 34 65 62 39 66 32 34 31 30 64 64 61 33 65 36 36 36 35 33 31 62 61 30 63 37 66 39 66 39 38 66 61 66 34 33 61 32 61 30 61 34 38 30 66 62 32 31 61 66 62' ],
CanonicalRequest: [ 'PUT\n/abcde\nContent-Length=8&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJ5WEVI3IFN7JJ6RA%2F20150108%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20150108T222615Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host\nhost:xxxx.s3.eu-central-1.amazonaws.com\n\nhost\nUNSIGNED-PAYLOAD' ],
CanonicalRequestBytes: [ '50 55 54 0a 2f 61 62 63 64 65 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 3d 38 26 58 2d 41 6d 7a 2d 41 6c 67 6f 72 69 74 68 6d 3d 41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 26 58 2d 41 6d 7a 2d 43 72 65 64 65 6e 74 69 61 6c 3d 41 4b 49 41 4a 35 57 45 56 49 33 49 46 4e 37 4a 4a 36 52 41 25 32 46 32 30 31 35 30 31 30 38 25 32 46 65 75 2d 63 65 6e 74 72 61 6c 2d 31 25 32 46 73 33 25 32 46 61 77 73 34 5f 72 65 71 75 65 73 74 26 58 2d 41 6d 7a 2d 44 61 74 65 3d 32 30 31 35 30 31 30 38 54 32 32 32 36 31 35 5a 26 58 2d 41 6d 7a 2d 45 78 70 69 72 65 73 3d 39 30 30 26 58 2d 41 6d 7a 2d 53 69 67 6e 65 64 48 65 61 64 65 72 73 3d 68 6f 73 74 0a 68 6f 73 74 3a 70 63 76 69 73 69 74 2d 67 72 61 76 61 74 61 72 73 2d 74 65 73 74 2e 73 33 2e 65 75 2d 63 65 6e 74 72 61 6c 2d 31 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 0a 68 6f 73 74 0a 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44' ],
RequestId: [ 'BB80E4565E8FA0C5' ],
HostId: [ '1J/zdpD9Itypwts5JlBVzHqSR2oOOP25oRlGPJY6CDOt42VN/mpwmIum3eAxNLZL9uq48jyCRS0=' ]

Thanks
Ruediger

[lsegal]

I've been setting ContentLength to force node not to use TransferEncoding: chunked as we'd otherwise get a 'A header you provided implies functionality that is not implemented' error.

The SDK doesn't actually send a request in getSignedUrl(), so you should be omitting it there and only providing it in your http.request() call. The issue here is that you probably don't want to sign the request with a specific ContentLength.

For what it's worth, I'm not able to reproduce this when testing with a signed URL through curl. Here's a little snippet to return a signed URL:

// test.js
var AWS = require('aws-sdk');
var s3 = new AWS.S3({params: {Bucket: process.env.BUCKET, Key: process.env.KEY}});
console.log(s3.getSignedUrl('putObject', {ACL: 'public-read'}));

And then I run it successfully with this shell snippet:

$ curl -X PUT -d "hello world" `AWS_REGION=eu-central-1 BUCKET=mybucket KEY=mykey node test.js`
$ curl http://mybucket.s3.amazonaws.com/mykey
hello world

Maybe this issue is specific to the headers Node.js is sending? Can you print the headers that are part of your http.request?

[pcvisit]

Thank you for your quick reply. Removing the ContentLength in both the signing and sending the request did the trick.

Didn't examine the headers sent out by node, but unless we had also some other subtle error in our request headers, I'd suspect that using ContentLength in signed requests might be broken ...

Anyhow .. we are fine with not using ContentLength.

Thanks for your great help!
Ruediger

[mhart]

From what I can tell @pcvisit it's not that it's broken, but rather there's a bug in the way you're calculating ContentLength – it should be in bytes, not characters.

So {ContentLength: content.length} is incorrect and will be buggy.

{ContentLength: Buffer.byteLength(content)} is the correct way to do it if you ever need to.

[mhart]

(of course, that could just be a coincidence and something else was happening :smile_cat:)

[lsegal]

Passing ContentLength in is what is causing this issue. The SDK is not correctly signing with this parameter present. It's probably not something you want to be passing in anyway, since this parameter doesn't actually have any effect in the generated URL (namely we don't sign Content-Length as a header, and having Content-Length as a query string parameter has no effect from S3's end), but it _could_ be considered a bug.

That said, I'm not sure what the fix is-- we either want to provide a warning/error, or silently ignore this parameter, even though it won't do what you want. Thoughts?

[pcvisit]

Well I think returning an error message in this case should be sufficient to keep people like me from chasing one's own tail ...

Silently ignoring this param would have the same effect, but it is unexpected behavior. And it might be confusing if someone is implementing a scenario where the Content-Length has to be signed (don't know whether such a scenario exists, though)

Just my 2 cents

[lsegal]

The biggest concern here would be breaking existing users who have been using ContentLength in production code that will start to throw errors when they upgrade the SDK. Not sure how common that is, but I think we need to keep that in mind.

[lsegal]

Actually, given that this has been failing for both V2 and V4 signed URLs, it doesn't seem like we are breaking anyone by enforcing the parameter validation, so I will go ahead and add code to error out if ContentLength is passed in.

[chenliu0831]

We are sending the presigned url to someone else that does the upload and they wish to make sure the upload is completed (e.g. they open the file in a weird way that truncate the file). If the Content-Length can be signed through the SDK it'll at least give the client an error. If not, is there any way for the client to check if the uploaded file is of the desired size based on the response from s3 upload? Thanks.

[jpsantosbh]

Without ContentLength is not working, because almost every client includes the ContentLength header on his own and then the StringToSign need to include content length value.

An example captured from a 403 error I just got.

PUT

image/jpeg
1507804869
x-amz-security-token:FQoDYXdzEMP//////////wEaDAKgdFD/xRX7K3eU5SLuAekvCkM7nOHWAQ0U61f0qz16SB9BPjotNPQNJWV5BIzfWpi4lI62O9BTKhvrTrU+qmIekv3YeJBcthZDcvMcPEcPcLBLcQ5Kjj8EaCe9ItFzbrasRvCZgbltFXoV5IbL15oXXqeEzaoljSGmoVFarrSC7A2HvWZ65Ga1hTiyJ+jYyJMnmXHmW9d9vy6TVs/sl2acWoYPmNZiOtF0pw520C3pTKNfO57t1rQvsmoXHErfF2iVptVToW6EedlhGE19Nq7/bkOL6b50F7YllMI7q1h7GjLEJi7YFROuuNcqkbg0o3oIiNv/r7j9gGT+Yz8o85OhzQU=
/ugimages/2017/08/31/433bf3b349aa6292352eb2906a6108e42927dac725bcb8336e4c25949de232a9/765199c38ed9a6937b2ae219cf12fbc5e37c1bfae0d3b64ac026590967f0be7a.jpg

[nazreen]

In case someone else is looking at this and is in a similar situation as me, I got a similar SignatureDoesNotMatchError when my s3 bucket's CORS Configuration did not contain <AllowedHeader>*</AllowedHeader>

I ran into this when moving from one bucket to another, copying all the settings except for the CORS Configuration.

[mihaiserban]

@nxmohamad life saver!

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.