Nodejs-storage: getSignedUrl with cname results in SignatureDoesNotMatch error
~Note that this works fine with v2.~ I thought it did, but it actually no longer does.
Environment details
- OS: macOS 10.14.6
- Node.js version: v8.16.1
- npm version: 6.4.1
- yarn version: 1.17.3
@google-cloud/storageversion: 3.2.1
Steps to reproduce v2
```js
file.getSignedUrl({
version: 'v2',
action: 'read',
expires: Date.now() + 10000000,
cname: 'https://storage.customhostname.com'
}, (err, url) => {
})
With `cname`, the URL doesn't work in the [browser](https://storage.dev.bookcreator.com/data%2Fi4rNhrOGJgNGlzELwesBcVrSfnG2%2Fbooks%2Fqle8FcNGSNGXQGd1EElspA%2Fresources%2FQC5Qdp5gQgeCMw7AQ18nmw.m4a?GoogleAccessId=api-service%40bookcreator-dev.iam.gserviceaccount.com&Expires=1568829600&Signature=oDAGX4myOoD%2F1UrWPhKOhl2WCw09fQ%2FatkJk5C8%2Bvk4FV%2F5t5lj2Kr3EUNUkQc2jm7vMg1X1XGu8y23sTIklW525f6VbJvN7MjHSGSnh317FO%2Fv6lTBVSekmN7CfUY4Bds%2BdcWYj%2FgI%2FjPFqXMsjmr%2Bqs4wYGRZJ0P6%2BiGUgX8WueF%2F0LNdsI44OsZiZ7z%2FR3vXR%2BTzCfRfxOMETAYrn8jjjITMJpQ1UO7PVYc5E0aAAT0URh816T%2BSee%2Fu3UNkD0hUMJbQC6XtiqPsOawHb%2Bi%2FCNr0R7yjYuluKnIuCe51oIBVQdG9iNTBZD8oJFsPtgrMufibXgS2e%2BuJcTPbMww%3D%3D).
However, sending the same request using `curl` works:
`curl 'https://storage.dev.bookcreator.com/data%2Fi4rNhrOGJgNGlzELwesBcVrSfnG2%2Fbooks%2Fqle8FcNGSNGXQGd1EElspA%2Fresources%2FQC5Qdp5gQgeCMw7AQ18nmw.m4a?GoogleAccessId=api-service%40bookcreator-dev.iam.gserviceaccount.com&Expires=1568829600&Signature=oDAGX4myOoD%2F1UrWPhKOhl2WCw09fQ%2FatkJk5C8%2Bvk4FV%2F5t5lj2Kr3EUNUkQc2jm7vMg1X1XGu8y23sTIklW525f6VbJvN7MjHSGSnh317FO%2Fv6lTBVSekmN7CfUY4Bds%2BdcWYj%2FgI%2FjPFqXMsjmr%2Bqs4wYGRZJ0P6%2BiGUgX8WueF%2F0LNdsI44OsZiZ7z%2FR3vXR%2BTzCfRfxOMETAYrn8jjjITMJpQ1UO7PVYc5E0aAAT0URh816T%2BSee%2Fu3UNkD0hUMJbQC6XtiqPsOawHb%2Bi%2FCNr0R7yjYuluKnIuCe51oIBVQdG9iNTBZD8oJFsPtgrMufibXgS2e%2BuJcTPbMww%3D%3D' -Lv > /dev/null`
But sending the same user-agent as the browser, causes a 403:
`curl 'https://storage.dev.bookcreator.com/data%2Fi4rNhrOGJgNGlzELwesBcVrSfnG2%2Fbooks%2Fqle8FcNGSNGXQGd1EElspA%2Fresources%2FQC5Qdp5gQgeCMw7AQ18nmw.m4a?GoogleAccessId=api-service%40bookcreator-dev.iam.gserviceaccount.com&Expires=1568829600&Signature=oDAGX4myOoD%2F1UrWPhKOhl2WCw09fQ%2FatkJk5C8%2Bvk4FV%2F5t5lj2Kr3EUNUkQc2jm7vMg1X1XGu8y23sTIklW525f6VbJvN7MjHSGSnh317FO%2Fv6lTBVSekmN7CfUY4Bds%2BdcWYj%2FgI%2FjPFqXMsjmr%2Bqs4wYGRZJ0P6%2BiGUgX8WueF%2F0LNdsI44OsZiZ7z%2FR3vXR%2BTzCfRfxOMETAYrn8jjjITMJpQ1UO7PVYc5E0aAAT0URh816T%2BSee%2Fu3UNkD0hUMJbQC6XtiqPsOawHb%2Bi%2FCNr0R7yjYuluKnIuCe51oIBVQdG9iNTBZD8oJFsPtgrMufibXgS2e%2BuJcTPbMww%3D%3D' -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36' -Lv > /dev/null`
If you change the user-agent to just `user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko)` it also works.
The `v2` signing used to work (~but unsure when it stopped~ it stopped working around 1700 yesterday [GMT]).
#### Steps to reproduce `v4`
I thought that maybe the legacy (as the docs called it) `v2` had been deprecated so tried the `v4` and this doesn't work either.
```js
file.getSignedUrl({
version: 'v4',
action: 'read',
expires: Date.now() + 10000000,
cname: 'https://storage.customhostname.com'
}, (err, url) => {
})
The URL produced does have the correct host, but on going to the URL you get a SignatureDoesNotMatch error.
Removing the cname results in a working URL.
E.g.: with cname:
Results:
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>
The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.
</Message>
<StringToSign>
GOOG4-RSA-SHA256 20190918T143035Z 20190918/auto/storage/goog4_request 75304e5ca664b15a1fbbebca700d83f4fbe48fc7bfe8db70399ce7fbc3ba0b38
</StringToSign>
<CanonicalRequest>
GET /data%2Fi4rNhrOGJgNGlzELwesBcVrSfnG2%2Fbooks%2Fqle8FcNGSNGXQGd1EElspA%2Fresources%2FQC5Qdp5gQgeCMw7AQ18nmw.m4a X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=api-service%40bookcreator-dev.iam.gserviceaccount.com%2F20190918%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20190918T143035Z&X-Goog-Expires=12565&X-Goog-SignedHeaders=host host:storage.dev.bookcreator.com host UNSIGNED-PAYLOAD
</CanonicalRequest>
</Error>
rhodgkins
All 14 comments
Regarding the v2 issue, I've found that removing the encodeURIComponent of the object name from here allows it now to work regardless of the user-agent header...
This seems to be more of a backend issue now, as this was previously working without changing existing code!
rhodgkins
on 19 Sep 2019
@rhodgkins Thank you for the bug report and the detailed reproductions!
The v2 issue does seem like an issue with the service - for this the best course is to file an issue at the issue tracker to get a direct response.
For v4, it seems like the Node.js implementation did not use the cname provided in constructing the canonical request:
https://github.com/googleapis/nodejs-storage/blob/master/src/file.ts#L2590
I'll send a PR to fix this issue.
Thanks for your patience!
jkwlui
on 27 Sep 2019
@jkwlui - thanks for getting back to me.
Regarding the v2 issue, I've already raised a support case in GCP (#20688201) and they came back with the following:
As per Engineering, a security issue came to light recently, and we can not safely support the form of URL you are currently signing (with URL-encoded slashes). You can work around this by not url-escaping the slash between bucket and object in your URL (use "/" instead of "%2F"). Cname-based cases are expected to be restored to functional in coming weeks.
Engineering has advised that URL-encoding slashes is discouraged, and will never be supported again for signed URLs to the default host of "storage.googleapis.com".
(They also raised an issue in the public tracker).
If it helps, I removed the encodeURIComponent from this line:
https://github.com/googleapis/nodejs-storage/blob/ace3b5e3819c7894159dd84f17aac0f21396e407/src/file.ts#L2481
And everything started to work as before (with cname based URLs at least).
Not sure if this will also apply to v4...
rhodgkins
on 1 Oct 2019
@rhodgkins I've tried to reproduce what you're experiencing with the v2 signed url with various user-agent headers, but I haven't gotten 403 back.
Given that Engineering has said that URL-encoded slashes is expected to be restored, can you see if you're still experiencing the issue? This is so that if we go ahead and make a fix to remove encodeURIComponent, we want to make sure it's addressing the issue that you raised.
Thank you for your patience!
jkwlui
on 9 Oct 2019
@jkwlui sorry about the delayed response...!
I just wanted to check:
Engineering has said that URL-encoded slashes is expected to be restored
I was under the impression from the comment, that URL-encoded slashes will never be supported again?
I generated a new signed URL (with v3.3.1 of this package), and trying it in Chrome results in the 403 signature mismatch. I then copied the cURL form of the request from the network tab:
curl 'https://storage.dev.bookcreator.com/data%2Fi4rNhrOGJgNGlzELwesBcVrSfnG2%2Fbooks%2Fqle8FcNGSNGXQGd1EElspA%2Fresources%2FQC5Qdp5gQgeCMw7AQ18nmw.m4a?GoogleAccessId=api-service%40bookcreator-dev.iam.gserviceaccount.com&Expires=1570795200&Signature=WFTKpU%2BOUo%2FqgAckS2vvPkohfbbuX4l4BKZ7s%2BfhJDPBc5gte7UHox8sWWX5%2FDEub%2FgNAkf5QV3l%2FHSgXu8mPwMB%2B6vKx17v9oogug39ReF7oT%2BBcm3Tmrlz92f6%2F3DwnSIf2OthJzEZY31PISFUdnyHiy%2F9Uak52JVk7Ugcowa%2B3B1O%2BC%2BZvwLvwElwXMRuAicQvIvVKopnNolgBARZ6gN%2Fiswe5Ukax7QYPzP5VRglpOUjnTHWYYJbIGNGNY4l7g1R3ZIMpBU45SxJgufUevygvh6nJoTPQDHXWMovUn1ri0bxiWV%2FHNNFEtjbvPpTmwjhUqNzy35n4QonJCeltw%3D%3D' -H 'authority: storage.dev.bookcreator.com' -H 'upgrade-insecure-requests: 1' -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36' -H 'sec-fetch-mode: navigate' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3' -H 'sec-fetch-site: none' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: en-GB,en-US;q=0.9,en;q=0.8' -H 'cookie: _ga=GA1.2.707883014.1570783193; _gid=GA1.2.895096839.1570783193; bc_auth=eyJhbGciOiJSUzI1NiIsImtpZCI6ImQ2YzM5Mzc4YWVmYzA2YzQyYTJlODI1OTA0ZWNlZDMwODg2YTk5MjIiLCJ0eXAiOiJKV1QifQ.eyJuYW1lIjoiUXdlcnR5MTIzIiwiaXNzIjoiaHR0cHM6Ly9zZWN1cmV0b2tlbi5nb29nbGUuY29tL2Jvb2stY3JlYXRvciIsImF1ZCI6ImJvb2stY3JlYXRvciIsImF1dGhfdGltZSI6MTU3MDc4MzU4OCwidXNlcl9pZCI6IlgzQ01yQ1VjVDladnFId2xtVnhlS2RHY09SSjMiLCJzdWIiOiJYM0NNckNVY1Q5WnZxSHdsbVZ4ZUtkR2NPUkozIiwiaWF0IjoxNTcwNzgzNTkxLCJleHAiOjE1NzA3ODcxOTEsImVtYWlsIjoiYmlsYm9AYm9va2NyZWF0b3IuY29tIiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJmaXJlYmFzZSI6eyJpZGVudGl0aWVzIjp7ImVtYWlsIjpbImJpbGJvQGJvb2tjcmVhdG9yLmNvbSJdfSwic2lnbl9pbl9wcm92aWRlciI6InBhc3N3b3JkIn19.nYOYzat7bBxgpPJhLBYevABFX5n1sFVSW0rznTZ0lk1qcQ9EKZoTv66apJGSUdzJ0gaqRkYYuiYC8naP-Gq2IM1QbNZv9W_4YcKcSJEiMhFY2F132MbtM_RQPZqD24rQ24CGDnxGafWYXUYpBBBOZivAkzn-r33O4SNqI8B_GN_N3t8lY3j4nFcwJmnGfPbDGMGYDseEOnXYneoLxEdEnXid0lE18rRX9z1e1p8ZY8m1hbD_tFJ3iqxbYNNsIwv1r3phuET86GrMqxN8tK17TDqlxkFEaap3-vdwIN0OmRg9nM4OfxDQoySPHtQtPR7SCfiyGD0G15cMQp6nsY9u_Q' --compressed -vL
(I added the -vL flags myself).
Let me know if you need any more information!
rhodgkins
on 11 Oct 2019
What was the name of the file?
jkwlui
on 16 Oct 2019
I wanted to double check, since in the response you got from engineering, it's mentioned that "You can work around this by not url-escaping the slash between bucket and object in your URL". We've only url-encoded the object name, not the slash between bucket and object:
I've tried constructing a (v2) signed URLs with and without the encodeURIComponent line removed, and both URLs seem to have worked for me:
These are equivalent, except in the later one the slashes in the object name is not encoded (thus generating a different signature).
jkwlui
on 16 Oct 2019
What was the name of the file?
/data/i4rNhrOGJgNGlzELwesBcVrSfnG2/books/qle8FcNGSNGXQGd1EElspA/resources/QC5Qdp5gQgeCMw7AQ18nmw.m4a
Regarding the second reply, I'm not sure how I can help there. All I've found is that viewing a signed URL with no URL-encoding slashes in the object name component works in the browser, but the same signed URL with encoded slashes does not work in the browser, resulting in a SignatureDoesNotMatch error.
(Note that a plain cURL with both URLs works).
rhodgkins
on 16 Oct 2019
Just to verify, the error occurs only when
v2signature is usedcnameis used- the request is made using the browser, or using equivalent headers via
curl
is that correct?
We released 3.3.1 to address the v4 bug, did it fix your issue?
Thank you for your continued patience!
jkwlui
on 18 Oct 2019
Just to verify, the error occurs only when...
Yep to 2 of these, except for the 1st one (v2 signature is used)!
The v4 bug fixed in that release you mentioned was to do with the missing cname from the calculated signature, so it hasn't fixed this bug. I hadn't actually tried it since that version came out (as we're using v2).
Example v4 URLs with same issue:
rhodgkins
on 21 Oct 2019
@rhodgkins I've spoken to the product team and verified that slashes indeed should not be encoded when generating a signature. I'll implement a fix and update this thread when the fix is released. Thank you for your patience, again, and for your help in reporting the issue.
jkwlui
on 24 Oct 2019
This should be resolved in @google-cloud/[email protected] now. Please let us know if you're still seeing issues with URL signing. Again, thank you for your patience and help with fixing this issue. @rhodgkins
jkwlui
on 31 Oct 2019
@jkwlui yep this has fixed it thanks - though it appears to have exposed another issue. I shall create an issue with more details. Thanks again for this fix!
rhodgkins
on 1 Nov 2019
@jkwlui I was searching before raising the issue and found #682.
It seems that a config simply changing v2 and v4 in a config doesn't work when responseDisposition is provided:
Works:
{
version: 'v2',
action: 'read',
expires: 1573210000000,
responseDisposition: 'inline; filename="QC5Qdp5gQgeCMw7AQ18nmw.m4a"'
}
Doesn't work:
{
version: 'v4',
action: 'read',
expires: 1573210000000,
responseDisposition: 'inline; filename="QC5Qdp5gQgeCMw7AQ18nmw.m4a"'
}
I had a brief look in the docs, and couldn't see anything mentioning the lack of support!
Would it be worth ignoring responseDisposition for v4 instead of it completely failing? Just my 2 pence anyway! Also is there any idea if this will ever be supported?
rhodgkins
on 1 Nov 2019
Related issues
shaffeeullah
路
3Comments
kaygeeklaas
路
4Comments
hstemplewski
路
4Comments
major-mann
路
5Comments
atn
路
6Comments
Most helpful comment
@rhodgkins I've spoken to the product team and verified that slashes indeed should not be encoded when generating a signature. I'll implement a fix and update this thread when the fix is released. Thank you for your patience, again, and for your help in reporting the issue.