Amphtml: update cache return . 403

Created on 27 Sep 2017 · 49Comments · Source: ampproject/amphtml

Hi all

 I had read the "https://developers.google.com/amp/cache/update-cache" and follow the example  verify the data ok

vocka@lalacube:/tmp$ . openssl dgst -sha256 -signature signature2.bin -verify public-key.pem url.txt . Verified OK

Step 1. check the public key

https://blog.lalacube.com/.well-known/amphtml/apikey.pub <- can read the data.

my sample code
`
php
function urlsafe_b64encode($string) {
$data = base64_encode($string);
$data = str_replace(array('+','/','='),array('-','_',''),$data);
return $data;
}
$timestamp=time();
$ampBaseUrl = "https://blog-lalacube-com.cdn.ampproject.org";
//https://blog-lalacube-com.cdn.ampproject.org/c/s/blog.lalacube.com/mei/yahoo_auction_amp.html
$signatureUrl = '/update-cache/c/s/blog.lalacube.com/mei/yahoo_auction_amp.html?amp_action=flush&amp_ts='.$timestamp;
// opening the private key

$pkeyid = openssl_pkey_get_private("file://private-key.pem");
// generating the signature
openssl_sign($signatureUrl, $signature, $pkeyid, OPENSSL_ALGO_SHA256);
openssl_free_key($pkeyid);

$signature = urlsafe_b64encode($signature);
// final url for updating
$ampUrl = $ampBaseUrl.$signatureUrl."&amp_url_signature=".$signature;
echo $ampUrl."n";
`

example : https://blog-lalacube-com.cdn.ampproject.org/update-cache/c/s/blog.lalacube.com/mei/yahoo_auction_amp.html?amp_action=flush&amp_ts=1506496391&amp_url_signature=hYTPVcaUtEWvg0zPEs-DsDDyeWdjz1RirZLFiFkyeG4-0HFbS7PHBqoREHgZpv60xB3hWXLwTr5b0FHcgOYLIK6_Z48nbMvkY3At7FL0dACIKAaovOxCSigbNJ9xt89gRCYzeg10ub-VM7JDxxazOwuu6lR0UB9BMcRn1-aLcbcX56VSqtmG1c84hpvIIfGnMlpp9O9PRB1Ps55RDRR6pVRrBMldL6FS8YtqQbTFohebL_nPFTLs1_Q5hmQMboPSGmewCcZ4YvtLb6jnw_g60mhU352JV6lyy1QwOmo55WY9QHT9wsC-Co7SnhJJVkBoQDGGBIxY3t8NkaGPnq4QMA

step curl 'https://blog-lalacube-com.cdn.ampproject.org/update-cache/c/s/blog.lalacube.com/mei/yahoo_auction_amp.html?amp_action=flush&amp_ts=1506496391&amp_url_signature=hYTPVcaUtEWvg0zPEs-DsDDyeWdjz1RirZLFiFkyeG4-0HFbS7PHBqoREHgZpv60xB3hWXLwTr5b0FHcgOYLIK6_Z48nbMvkY3At7FL0dACIKAaovOxCSigbNJ9xt89gRCYzeg10ub-VM7JDxxazOwuu6lR0UB9BMcRn1-aLcbcX56VSqtmG1c84hpvIIfGnMlpp9O9PRB1Ps55RDRR6pVRrBMldL6FS8YtqQbTFohebL_nPFTLs1_Q5hmQMboPSGmewCcZ4YvtLb6jnw_g60mhU352JV6lyy1QwOmo55WY9QHT9wsC-Co7SnhJJVkBoQDGGBIxY3t8NkaGPnq4QMA'
result is . Error 403 (Forbidden)

I found the https://stackoverflow.com/questions/46176405/amp-update-cache-request-returns-403 , it's similar.

So how can i debug ?
thank
Vocka

caching

Source

vockalimo

All 49 comments

/to @Gregable

dvoytenko on 29 Sep 2017

/cc @codewiz

dvoytenko on 29 Sep 2017

Having similar issue, with no code, but using this app to test:

https://amp-cache-refresh.appspot.com/

Our public key is publicly available, any clue how to debug this issue would be great.

EDIT: we have python code that verifies our signed url, and I can share that if nesc( its also a 403).

advgiarc on 6 Oct 2017

Same here... Update cache gives me a 403: public key is available as plain text, I verified the signature with the public key (SHA256) and server times are synced. I'm using the same code as the issue author above (apart from the additional signature validation).

saitho on 10 Oct 2017

We're adding some debug info to the 403 responses to make debugging these issues easier in the future. Sorry about that.

I traced the above query and the internal error was "Invalid PEM key". The key decodes correctly with "openssl rsa -pubin -in apikey.pub -text -noout".

However, I noticed that blog.lalacube.com returns a 500 Internal Server Error when I try to fetch the key with curl or wget. The same url works with Chrome. So I suspect this is blocking the AMP Cache from fetching the key.

codewiz on 10 Oct 2017

Looks like the server wants an Accept-Encoding header in the request:

curl -H 'Accept-Encoding: gzip' -i https://blog.lalacube.com/.well-known/amphtml/apikey.pub
HTTP/2 200
...

codewiz on 10 Oct 2017

I managed get the key ingested by requesting it directly via its AMP Cache url:

https://blog-lalacube-com.cdn.ampproject.org/r/s/blog.lalacube.com/.well-known/amphtml/apikey.pub

After this, the update-cache query fails due to the expired signature, as expected.

When you get update-cache to work, try purging your key itself to make sure it gets re-ingested.

codewiz on 10 Oct 2017

We're still having the 403 blues, appreciate any help. Here is our public key: https://minutes.advance.net/.well-known/amphtml/apikey.pub

advgiarc on 10 Oct 2017

@advgiarc The PEM key looks correct and the mime-type is text/plain. Please paste a signed update-cache request so I can see where it's being rejected.

Please also provide the code (or pseudocode) you are using to sign the url.

codewiz on 10 Oct 2017

@codewiz

https://minutes-advance-net.cdn.ampproject.org/update-cache/c/s/minutes.advance.net/f31c00-cleveland-indians-at-new-york-yankees-live-score-updates.amp?amp_action=flush&amp_ts=1507663526&amp_url_signature=aI-OEuC1cXN5MLUqGmN0rU225OWHdd0fYT66ZruV0h8fFGuntZVEQVSnJ_AunplYY9j7Gas76dWNjASDc2ysNipLBaAqD3c5KdMKJOV6YWc2O_9YNI7hA5zrCbtdhGEXZFFQUpjRwUt5lQrK6crlCFaRn5TIgYhebJp3-Jei9mYf3GeHeCMCOGEOFqHZvkX2fMzv_a-MKISjiHByL3odVDGhKx3dihzgrJy0GfPhKl7gMSS3NJw-9iVpWkG4_r5GqMe9At-Lq_ogXFK76cH87N1p77VpbLkMUykuDHyVcOKVebBS3CdPcQldoVVV_TmtRAbkLE7Lk8jtajSTeAbKAg==

pastebin with code:

https://dpaste.de/27OM

Thanks for taking a look!

advgiarc on 10 Oct 2017

First of all, I determined that AMP Cache does not handle HTTP refresh correctly for update-cache verification keys: if you issue an update-cache request and then swap /.well-known/amphtml/apikey.pub with a different key, we keep using the old key material indefinitely. To make things worse, 404 responses are also cached forever :-(

I filed an internal bug report, but it might take some time for the fix to roll out to production. Meanwhile, I can flush invalid keys manually. Just send me a private message either on GitHub or on amphtml.slack.com. Apologies for not discovering this sooner.

Specifically for @advgiarc: the code at https://dpaste.de/27OM should include the leading "/" in sign_url. Everything else looks correct to me.

codewiz on 10 Oct 2017

👍1

Great, will post you a private message when back at a computer. Thanks for investigating.

advgiarc on 10 Oct 2017

hi @codewiz

"curl -H 'Accept-Encoding: gzip' -i https://blog.lalacube.com/.well-known/amphtml/apikey.pub
HTTP/2 200" . <- Should i enable all to fetch the apikey.pub ?
I will change another key , thanks ~

vockalimo on 11 Oct 2017

I will change another key , thanks ~ <- cool . fixed this .
thanks

vockalimo on 11 Oct 2017

@codewiz Is the issue with cached verification keys solved? Beacause I still get 403 error when trying to flush cache.
And other question: It is possible to use update-cache request to flush page that has no domain but only IP address (only for testing purposes, eg. https://24-288-88-55.cdn.ampproject.org/c/24.128.88.55/test.html )

jakubszlendak on 6 Dec 2017

I am also getting a 403 error while trying to flush cache

https://gist.github.com/JoyceBabu/5fa508d1ede8f6965601dfce31d1cc8f

JoyceBabu on 6 Dec 2017

Sorry to keep you waiting, @JoyceBabu :

The error reported by AMP Cache in the 404 page is "URL signature verification error."
The cached key matches your key:
- https://www-prokerala-com.cdn.ampproject.org/r/s/www.prokerala.com/.well-known/amphtml/apikey.pub
- https://www.prokerala.com/.well-known/amphtml/apikey.pub
This is the exact string that should have been signed (note the %3f in the middle):
- /update-cache/c/www.prokerala.com/astrology/horoscope/%3fsign=leo&amp=1?amp_action=flush&amp_ts=1512564684

If this doesn't help, please share the code you used to produce the signed url.

codewiz on 20 Dec 2017

👍1

hello,

How can I fix?
404 error:

https://habervaka-com.cdn.ampproject.org/r/s/habervaka.com/.well-known/amphtml/apikey.pub

403 error:

https://www-habervaka-com.cdn.ampproject.org/update-cache/c/s/www.habervaka.com/sureli/mehmet-demirkol-galatasaraya-gelebilir-07-january-2018.html/amp?amp_action=flush&amp_ts=1516171261&amp_url_signature=AlIznL0gKFOjOv5btv8buZhpc5W46uv6zw3YUegrFOx0Q_NdkE-EST7k0gYCYrI5JxEhoti-bWZOMlya9-Uocgv5n9xCSkE4ww5iVSFjvj46uZUtE54ExduxZJy8l4rhvGvlv9_dapcM9Mgb9id9lv7E58_cV3jjfNhduKsZ54EIuEq9w8oTEFmsujUfA1vVmpw3R2q7lk9A1-gPmhqS4tEKcWWODxHIZ5esBDkUzOMynSNXJNsXzJZPx5pK8SlESRcd72Nh-veb-xm8mTgCXrDUEwESjkuAuQxh9hs3nu1yy-uEZBVf-7cYyKj_etilOncmqZ7KTdTDFY6Cxp659A

yumruk on 17 Jan 2018

@yumruk Your website's (https://habervaka.com) SSL certificate was revoked. (SEC_ERROR_REVOKED_CERTIFICATE). This might be the issue. ;)

saitho on 17 Jan 2018

@saitho To fix the problem I'm talking about ı have again uploaded an new SSL certificate. This is the error you mentioned.
How can I fix?
404 error:

https://habervaka-com.cdn.ampproject.org/r/s/habervaka.com/.well-known/amphtml/apikey.pub

403 error:

https://www-habervaka-com.cdn.ampproject.org/update-cache/c/s/www.habervaka.com/sureli/mehmet-demirkol-galatasaraya-gelebilir-07-january-2018.html/amp?amp_action=flush&amp_ts=1516171261&amp_url_signature=AlIznL0gKFOjOv5btv8buZhpc5W46uv6zw3YUegrFOx0Q_NdkE-EST7k0gYCYrI5JxEhoti-bWZOMlya9-Uocgv5n9xCSkE4ww5iVSFjvj46uZUtE54ExduxZJy8l4rhvGvlv9_dapcM9Mgb9id9lv7E58_cV3jjfNhduKsZ54EIuEq9w8oTEFmsujUfA1vVmpw3R2q7lk9A1-gPmhqS4tEKcWWODxHIZ5esBDkUzOMynSNXJNsXzJZPx5pK8SlESRcd72Nh-veb-xm8mTgCXrDUEwESjkuAuQxh9hs3nu1yy-uEZBVf-7cYyKj_etilOncmqZ7KTdTDFY6Cxp659A

yumruk on 17 Jan 2018

@yumruk The 403 error page says: "Invalid PEM key."

The key looks good to me, but there's no Content-Type header in the response. Please configure your webserver to serve the key as "text/plain".

codewiz on 17 Jan 2018

@codewiz thanks

yumruk on 17 Jan 2018

Hi @codewiz
I was trying to implement update cache, encountered few issues and have few questions.

Do we really have to host the public key at the location .well-known/amphtml/apikey.pub ?
The site where we are implementing ampcache is secured. Do we have to do any configuration to work over https

when I tried updating the cache , I got 403 and the response as below (Public Key is still not made public)
HttpResponseProxy { HTTP / 1.1 403 Forbidden[Content - Type: text / html; charset = UTF - 8, X - Content - Type - Options: nosniff, Date: Tue, 23 Jan 2018 20: 09: 27 GMT, Server: sffe, Cache - Control: private, X - XSS - Protection: 1; mode = block, Alt - Svc: hq = ":443"; ma = 2592000; quic = 51303431; quic = 51303339; quic = 51303338; quic = 51303337; quic = 51303335, quic = ":443"; ma = 2592000; v = "41,39,38,37,35", Transfer - Encoding: chunked] }

rnagaraj on 23 Jan 2018

@rnagaraj
(I am not associated with the ampproject)

We are using AMP and I can offer some insight to your questions:

1) Yes, that is the exact location your public key must live under.

2) No, if your SSL certs are all setup correctly, there is nothing else you need to do for HTTPS, as AMP and 'update-cache' ONLY work over HTTPS.

RE: the 403: if your public key is not actually public and at the .wellknown location, things will not work.

advgiarc on 26 Jan 2018

Thank you @advgiarc

rnagaraj on 29 Jan 2018

Hello all and @codewiz,

I'm still getting 403 "Invalid PEM key" error message but I couldn't find any problem. Could you please control our implementation?

https://gist.github.com/ogun/61d8396eb6c58ab3c5ef71d61df89bad

ogun on 26 Mar 2018

Hello @codewiz
same issue. Could you please help flushing my API key?
https://studentloanhero-com.cdn.ampproject.org/r/s/studentloanhero.com/.well-known/amphtml/apikey.pub

I keep getting URL signature verification error.

Attached is generated link:
/update-cache/c/s/studentloanhero.com/featured/fico-auto-score-guide/?amp_action=flush&amp_ts=1523305916&amp_url_signature=aNi4NwXpQJwrLUCqWi-_z9WreySJ19gSvOOR8IC8GcYL2vfzDZiSxtPsTiwRb5I1qRNZrKO_EFhs0eYilM2duXZNqf__uYBvIzAlTpCWT6VESutomI1mbtIKa_1OsJPWWZPF_Madm-ma6aZQdrR2CvUXnWHqmYMve0wfwO-6BsHx0qux3jpsWgGA5t3al6UVWjOKAcaURUJYI8l_ZII_cX_is3EOrW7la3Qhel8-Kto3eEKnny2Up5VivvfhQKufDfe74k91SRR2wys26aPRr7ZsDq8pLCm5CK3f0LYFvplbx5s-OaT-8muL99alZb6z9HBX-KJt2FPES6bdEoh6Jw==

I've checked, it verifies ok with openssl locally.

Dimasmagadan on 9 Apr 2018

I had the same issue. my problem was:

I used phpseclibCryptRSA, and it wasn't signing it in a way that was expected apparently. once I switched to openssl php library, and used OPENSSL_ALGO_SHA256 in openssl_sign(), it finally worked!
also, I was only replacing + and / for web-safe encoding. After reading comments, I added = to it as well, so it looks like this:
$encoded_signature = str_replace(['+','/','='],['-','_',''], base64_encode($signature));
I made sure that I have a leading / in the url request that I sign (the one before update-cache).

Hope that helps!

obeyer on 13 Apr 2018

Hey @obeyer
thanks for an update. I've tried to replace '=' as well, but it didn't work.

The resulting generated link is this one:
/update-cache/c/s/studentloanhero.com/featured/fico-auto-score-guide/?amp_action=flush&amp_ts=1523607919&amp_url_signature=pj_Q-MhePQ2_vA9i-MTe4pBva5dy7FiXbusIBRY4yM0I9csgFG_-oZDxQcB7nx1RgTmoQw6reNJnDzzTKr1rQeK_UnT1e0E7UtshSkK545eMv3wV-PTDiWc4d2L8DW7FbkqoHUsT_gDEMdbuppnGLf7qpK2YTlIAb7_H8iQvvXr6rOaxyY_vE2lQ7nANov8dotjRR8Ao0ZenCwffGRasAnBLCwPmFhhw1aMfmFy6NLOQyXM-Tlv2Au2fhehI0I7TTgdroGW-8JjDbROucxXLnw58pfUWsJYijxnPV37dTPXxH6XgJwTi3Zro9hcT73q0oKOvQ7xRl_GPgz5BFwWQug

Dimasmagadan on 13 Apr 2018

I have published a composer package for clearing AMP cache

https://packagist.org/packages/ennexa/amp-update-cache

JoyceBabu on 25 Apr 2018

👍1

@Dimasmagadan your key gets downloaded, if I click on the link. it should just return the key, like this: http://www.popsugar.com/.well-known/amphtml/apikey.pub

obeyer on 25 Apr 2018

Hello @obeyer thanks for answering.
Actually, I've already changed my code to be run as php script and it started to work.
Earlier I've used a bash script to do exactly the same (the signatures were exact for both versions) and it somehow was unhappy with my signature.

Dimasmagadan on 25 Apr 2018

Have you successfully updated the cache before?

Did you ever changed your public keys before you got the errors?