Amphtml: Invalidating AMP cache returns 403

[anuragphadke]

Please only file reports about bugs in AMP here.

What's the issue?

Trying to invalidate AMP cache using the sample code in amp-update-cache/

How do we reproduce the issue?

1. Check out the code
2. npm install && npm start
3. visit the localhost:3000
4. Try to invalidate the cache by pasting the private key.
5. Returns 403, your client does not have permission.
6. Using Load Balancer on GCP.

This is for the page at https://amp.trysnow.com/amp/products-teeth-whitening-kit.html, public key available at https://amp.trysnow.com/.well-known/amphtml/apikey.pub

Your client does not have permission to get URL /update-cache/c/s/amp.trysnow.com/.....
Invalid public key due to ingestion error: Invalid Content.

[torch2424]

Triaging to @Gregable , feel free to re-assign 😄

[Gregable]

Reported internally as #125911164

[Gregable]

Questions from the internal team:

Is the public key owned by the you or is it a sample public key they're trying to access?

If it's owned by you, the error message is now specific: "Invalid public key due to ingestion error: Invalid Content."

It sounds like whatever key you are trying to use can't be ingested and is invalid.

[anuragphadke]

Is the public key owned by the you or is it a sample public key they're trying to access?
-> key is owned by me.

It sounds like whatever key you are trying to use can't be ingested and is invalid.
-> agree, however, I have tried multitude of ways to make this work [https://www.takaitra.com/amp-cache-update-script/, manually invalidating cache as per documentation and also the above sample code] and keep getting same error.

Any other insights why this might be a problem? We are using LBs on GCP, could it be related to response from LoadBalancer? Possibly related to the underlying Apache we might be using that's not supported?

[Gregable]

Feedback from the internal team:

Has you tried generating and updating the RSA key per the instructions in the documentation? https://developers.google.com/amp/cache/update-cache#rsa-keys

I'm not sure there's much on our side to debug why the keys are invalid. You'll need to look into it on your side.

[anuragphadke]

thx @Gregable
I tried:
https://amp-trysnow-com.cdn.ampproject.org/r/s/amp.trysnow.com/.well-known/amphtml/apikey.pub as per the doc and that gave a 404.
I'll continue debugging on my end.

[cr-solutions]

I have the same Problem, last year same pub-key, code etc. worked!
How to fix it?

1. That’s an error.

Your client does not have permission to get URL /update-cache/c/s/bildagentur.panthermedia.net/landingpage/nl-archiv-20190326-360-grad-bilder-reminder/amp/?amp_action=flush&amp_ts=1556281123&amp_url_signature=JQktV0kxGyHlGEotB7lX7VcXar6pkDVfAlCSiIKH9rovscPjXPMu3Bemo8OXvSclCPU66BS3HulrF5lWJQXnOl3y8lIBpJfdf50aUnNd1eGN-q5ZMmHGfw3eLphIYsFQmF8-YjxVssodCvXaOCa4ryPJGl9k4tm5CbVCF5BW9Eh0ZKE1VNncY9mUEhzEGB9IYZFwJN1WzpBTDslW9ew1103MjBoQRcFou-VFq0XPfSA107iXkpC4JevHEhRREQfKuduBsMNi1yYJEYf7nIjiZH3WKldW90fOlj57YlJcVRaEzt4-05zCZ5sTg-bhraFObTnc46tgyzxLJLRf59tQXg from this server. (Client IP address: 89.245.xxx.xxx)

Invalid public key due to ingestion error: Invalid Content That’s all we know.

[gaborszita]

I have the same Problem, last year same pub-key, code etc. worked!
How to fix it?

1. That’s an error.

Your client does not have permission to get URL /update-cache/c/s/bildagentur.panthermedia.net/landingpage/nl-archiv-20190326-360-grad-bilder-reminder/amp/?amp_action=flush&amp_ts=1556281123&amp_url_signature=JQktV0kxGyHlGEotB7lX7VcXar6pkDVfAlCSiIKH9rovscPjXPMu3Bemo8OXvSclCPU66BS3HulrF5lWJQXnOl3y8lIBpJfdf50aUnNd1eGN-q5ZMmHGfw3eLphIYsFQmF8-YjxVssodCvXaOCa4ryPJGl9k4tm5CbVCF5BW9Eh0ZKE1VNncY9mUEhzEGB9IYZFwJN1WzpBTDslW9ew1103MjBoQRcFou-VFq0XPfSA107iXkpC4JevHEhRREQfKuduBsMNi1yYJEYf7nIjiZH3WKldW90fOlj57YlJcVRaEzt4-05zCZ5sTg-bhraFObTnc46tgyzxLJLRf59tQXg from this server. (Client IP address: 89.245.xxx.xxx)

Invalid public key due to ingestion error: Invalid Content That’s all we know.

Didn't the public key expire?

[firmanpolyrific]

does anyone here has solved this issue yet? I got the same issue

and also where we generate the rsa key, on our local or online server?

[gaborszita]

does anyone here has solved this issue yet? I got the same issue

and also where we generate the rsa key, on our local or online server?

@firmanpolyrific which rsa key are you asking? public or private? The public key is at yourwebsite.net/.well-known/amphtml/apikey.pub You need to keep the private key locally on your own machine and don't show it to ANYONE. And also make sure the apikey.pub has content type "text/plain". You can check this by entering yourwebsite.net/.well-known/amphtml/apikey.pub in chrome. If chrome will want to download this file, then it isn't text/plain. If it displays the file content (the key) it is text/plain. Check this and pls reply to me. I'm happy to help you out.

[firmanpolyrific]

hi @gaborszita

I checked it and it's text/plain.
however, I'm a bit confused about the steps here:
https://developers.google.com/amp/cache/update-cache#rsa-keys

I understand the step no. 1 but when I go to step 2, where is apikey.pub comes from?
is it when I'm on my web server, then generate private-key.pem and public-key.pem and just directly rename public-key.pem to apikey.pub ?

[gaborszita]

Yes, exactly. Rename public-key.pem to apikey.pub and upload it to yourwebsite.net/.well-known/amphtml/apikey.pub Then generate the link and update the cache. If you don't have an automated link generator, you can use myn, which is at https://github.com/gaborszita/amp-c-cache-update

[firmanpolyrific]

@gaborszita
wait did you say upload? so do i have to generate private and public key from our local instead on my webserver?

i use this for link generator
https://github.com/enteresanlikk/amp-cache-update

[gaborszita]

I don't know how generating on webserver works, but I generated the keys locally and uploaded the public key to /.well-known/amphtml/apikey.pub, and it worked. So... Can you please tell me how you generated the keys on the webserver? Isn't it the same thing? By the way can you tell me your website (so I can investigate, isn't there a problem.)

[gaborszita]

Hey, I think I found the problem. I found that your robots.txt file blocks indexing the apikey.pub You have to allow the crawlers to crawl it, then it'll work.

[firmanpolyrific]

@gaborszita

is this will make crawlers do their job?

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Allow: /.well-known/apikey.pub

[gaborszita]

Well, there is one problem: The apikey.pub isn't in /.well-known/apikey.pub, it's in .well-known/amphtml/apikey.pub. So modify the line to:
Allow: /.well-known/amphtml/apikey.pub

[gaborszita]

@firmanpolyrific Are you trying to solve the problem? I just still didn't get a reply. If you don't understand something feel free to reply. I'm here to help.

[firmanpolyrific]

@gaborszita sorry for the late reply, I got another task from my work, well, I have changed the robots.txt to allow the apikey.pub.

but I can't say the amp cache update works yet, it seems, there is a caching problem that makes any changes not affected when I access the robots.txt, I still figuring out what happened.

anyway, thank you so much for your help, really appreciate that :)

[gaborszita]

If it isn't affected, it's likely because "update-cache only ensures that the content is updated within its max-age" (stated in the Update AMP Content Documentation.) If you have a header e.g. Cache-Control: max-age=3000, if you request to update the cache, it only updates the cached page if it is older than 3000 seconds. To fix this problem modify the header to Cache-Control: max-age=0

But if you want to allow the user's device to cache the page, but still update the AMP cache with the link, there's also a solution: The s-maxage caching header. The s-maxage sets the time while the cached content is considered stale in a CDN, NOT on any device, and the AMP cache is a CDN. So you can add s-maxage and modify the header to: Cache-Control: max-age=3000, s-maxage=0 I've never tried s-maxage out in this situation, so if it doesn't work, modify the max-age header. (but in theory, it should work)

[AndreBaumeier]

Hey, I am trying to flush amp caches for our domain. I am using the following script to do so: https://github.com/sizaki30/google-amp-update-cache - this should be covering signature generation etc. - I also put in the time to debug all the steps and could verify that this script should be doing everything as required by the documentation. Our public key is available at https://www.daskochrezept.de/.well-known/amphtml/apikey.pub and mimetype is set to text/plain (is the UTF-8 causing problems here?). https://github.com/ampproject/amphtml/issues/11455#issuecomment-335597990 mentions problems regarding caching of the publickey - is this still present? can someone invalidate the cache for our key? or is something else causing this problem?

curl -Lvso /dev/null https://www.daskochrezept.de/.well-known/amphtml/apikey.pub
*   Trying 104.20.132.6...
* TCP_NODELAY set
* Connected to www.daskochrezept.de (104.20.132.6) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
* Server certificate: ssl515004.cloudflaressl.com
* Server certificate: COMODO ECC Domain Validation Secure Server CA 2
* Server certificate: COMODO ECC Certification Authority
> GET /.well-known/amphtml/apikey.pub HTTP/1.1
> Host: www.daskochrezept.de
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Tue, 03 Dec 2019 12:42:41 GMT
< Content-Type: text/plain; charset=UTF-8
< Content-Length: 451
< Connection: keep-alive
< Set-Cookie: __cfduid=da20525401ca1dde0251bfe95e5eef8a51575376961; expires=Thu, 02-Jan-20 12:42:41 GMT; path=/; domain=.daskochrezept.de; HttpOnly
< X-Content-Type-Options: nosniff
< Last-Modified: Tue, 03 Dec 2019 10:19:26 GMT
< ETag: "1c3-598ca07e5d727"
< Cache-Control: max-age=1209600
< Expires: Tue, 17 Dec 2019 11:31:07 GMT
< x-storage: dkrweb
< X-Varnish: 137266889 135918674
< Age: 4294
< Via: 1.1 varnish (Varnish/6.1)
< X-Cache: HIT
< X-Cache-Hits: 10
< Accept-Ranges: bytes
< CF-Cache-Status: DYNAMIC
< Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< Server: cloudflare
< CF-RAY: 53f59f391ac7d6d1-FRA
<
{ [451 bytes data]
* Connection #0 to host www.daskochrezept.de left intact

calling ```/update-cache/c/s/www.daskochrezept.de/magazin/gewinnspiele/adventskalender.html?amp_action=flush&amp_ts=1575376880&amp_url_signature=I3Tiuzy-oxO5t3BK86HagXBbBitUNaHRHAC0sGAemJA13MdWmQpoInNhLrgQdUjxaXLUeCKKauOiLbOehfa9f4iw8cc-E3DBR4GAlK-xfZRG67upeeRhAscoaQLW99AOEwSoqbMBXuiuhz42HgGZqqlHKFeg8yv1nAUePpIJbDssheegzxBUL1HhEXej2C6Zu3dw_OR0LjQv9_HDO5NbJkHAwpDHj_ofW4YKOtzAwafbxokqEgzZpLzodRQj8NyBZB7dyaMLUJAu2IBshneRNvxGchOIySJDa8oW6PsoUCXlUtuXr8napYd50jidAxXAJouf-uxALcWnhjd9uj9O8A````

Results in 403, Invalid public key due to ingestion error: Invalid Content.

[gaborszita]

@AndreBaumeier I'm here to help you

First of all I've seen you havn't allowed robots to index the .well-known/amphtml/apikey.pub file. According to the documentation the file must be roboted. So please allow all search bots to index the file by adding:
User-agent:*
Allow: /.well-known/amphtml/apikey.pub
Try flushing the cache then, see if it works. If it doesn't, reply to this issue.
If it works, try restricting the access only to Googlebot (if needed cause I've seen in the robots.txt only specific crawlers can index your webpage)
User-agent: Googlebot
Allow: /.well-known/amphtml/apikey.pub

[crazygo]

@AndreBaumeier I'm here to help you

First of all I've seen you havn't allowed robots to index the .well-known/amphtml/apikey.pub file. According to the documentation the file must be roboted. So please allow all search bots to index the file by adding:
User-agent:*
Allow: /.well-known/amphtml/apikey.pub
Try flushing the cache then, see if it works. If it doesn't, reply to this issue.
If it works, try restricting the access only to Googlebot (if needed cause I've seen in the robots.txt only specific crawlers can index your webpage)
User-agent: Googlebot
Allow: /.well-known/amphtml/apikey.pub

Hello, document says "The public key must not be roboted." , Does this means Allow or Disallow ?
https://developers.google.com/amp/cache/update-cache#rsa-keys

[gaborszita]

It means that it should be allowed to be indexed by the robots.txt file.

[davidwhthomas]

I've just successfully implemented AMP cache clearing for articles and would like to note the following things to ensure for others:

1. The /.well-known/amphtml/apikey.pub file is accessible to both mobile and desktop user agents (e.g no redirect for non-mobile as the AMP cache client may redirect)

2. The public key is not excluded in robots.txt (i.e it is allowed in robots.txt) e.g:

    User-agent: *
    Allow: /.well-known/amphtml/apikey.pub

1. The public key response has the expected headers (i.e: content-type: text/plain):

    curl -I https://amp.example.com/.well-known/amphtml/apikey.pub

    HTTP/2 200 
    date: Sun, 26 Jul 2020 23:48:55 GMT
    content-type: text/plain
    vary: Accept-Encoding
    etag: W/"1c3-173478a8840"
    last-modified: Sun, 26 Jul 2020 23:48:55 GMT

With those things in place, I get an "OK" success response from the AMP cache clear endpoint