Amphtml: Discrepancy between CDN & non-CDN version of AMP ads (CORS)

[davisshaver]

What's the issue?

AMP template validates and serves Ad Sense units successfully here: https://www.onwardstate.com/2018/02/27/movin-on-older-mysterious/amp/

But in the CDN version, we see JS errors and the unit does not load: https://www.google.com/amp/s/www.onwardstate.com/2018/02/27/movin-on-older-mysterious/amp/

Here is example console output:

[QES]

This looks similar to the discussion in #13659 which has ended up discussion the CORS differences between the origin and the google cache. in both cases the curiosity to me is that the errors you (and I in #13659 shows a discrepancy between the Cache URL www.google.com/amp/s/ and the cache orgin cdn.ampproject.org

[davisshaver]

@QES Interesting... We did just change the amp-ads implementation, do you think this is a content (page source) mismatch? Or it's CORS headers?

I am just confused bc our DFP implementation w/ amp-ads was working fine, this only cropped up when I switched to AdSense type.

[davisshaver]

Still confused what's goin on here...

@QES are you still struggling with it?

It would be nice if there were a CORS validator in the same way there is a markup validator.

[QES]

@davisshaver Yes I'm still struggling with this. I agree having some way to validate CORS would be nice but better still documentation that gave some real world examples would be useful.

To me there are discrepancies in the way the documentation talks about the cache and the way it works, it took me a long time to understand how the CORS stuff worked locally with my own domains and amp-list and amp-access (and I'm never certain I'm right but it works).

However once you get into the Google AMP cache this seems to break down in ways that I don't understand and hopefully someone who has a better understanding of this will make in make sense to the rest of us :)

[davisshaver]

@QES Thanks... totally agreed! :)

Did you end up adding any headers to resolve your issue?

[QES]

@davisshaver no it seems that if I want to be able to use the unit on my own origin I need:

access-control-allow-origin: https://amp.sportsmole.co.uk

But if I want it to work on the Google AMP Cache it needs to be:

access-control-allow-origin: https://amp-sportsmole-co-uk.cdn.ampproject.org

But it is the same data and I can't see if there is way to have them both active at the same time (plus probably the cloudflare one and any other caches that appear in the future)

[choumx]

Hey folks, I assume you've seen these docs? There are specifics and example implementations there.

[davisshaver]

@choumx I have read those docs but to be honest I feel like everything is setup right. I am coming at this as a contributor to the AMP WordPress project.

If you would be able to analyze a sample setup as mentioned at the top (a generic WordPress instance backed by Cloudflare), I could make upstream contributions to AMP WordPress documenting the best practice setup for CORS. This issue has come up in that project too and as soon as I understand the issue myself I will document the solution.

cc @amedina

[QES]

Hi @choumx I had seen https://github.com/ampproject/amphtml/blob/master/spec/amp-cors-requests.md but the explanation isn't backed with how you actually do it for example:

Verify the CORS Origin header
CORS endpoints receive the requesting origin via the Origin HTTP header. _Endpoints should restrict requests to allow only the following origins:_
• Google AMP Cache subdomain: https://.cdn.ampproject.org (for example, https://nytimes-com.cdn.ampproject.org)
• Google AMP Cache (legacy): https://cdn.ampproject.org
• Cloudflare AMP Cache: https://.amp.cloudflare.com
• The Publisher’s own origins
For information on AMP Cache URL formats, see these resources:
• Google AMP Cache Overview
• Cloudflare AMP Cache

The question is not that we don't understand what is required but we are not sure how to get it to work - once something is in one of the caches I have no control over the code, the JSON endpoints loaded in the code work for my origin but if I change the element to work for the cache then it doesn't work for my end point and I can't see how you can get the page headers to say all of the above is fine.

There is probably a very simple way to do this but it hasn't been obvious to me since I started paying with this last year and took a lot of playing to find the correct incantation in the first place.

I posted something elsewhere that the documentation really needs to guide people through the possible options and coding for different situations so that it is more obvious what is required the CORS stuff is forever causing issues for me and I'm sure that is because I don't understand it :)

[QES]

Apologies I just noticed that https://github.com/ampproject/amphtml/blob/master/spec/amp-cors-requests.md has been updated taking into account some of my comments from last year.

If I read it correctly the CORS HEADERS delivered need to change based on the ORIGIN but only so if the ORIGIN is one of the allowed origins?

Looking into the example and my code I think the issue is that:

From LOCAL

HTTP/2 200
access-control-allow-headers: Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token
access-control-allow-credentials: true
access-control-allow-origin: https://ampbyexample.com
amp-access-control-allow-source-origin: https://ampbyexample.com
access-control-allow-methods: POST, GET, OPTIONS
access-control-expose-headers: AMP-Access-Control-Allow-Source-Origin

FROM CACHE

HTTP/2 200
access-control-allow-headers: Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token
access-control-allow-credentials: true
access-control-allow-origin: https://ampbyexample-com.cdn.ampproject.org
amp-access-control-allow-source-origin: https://ampbyexample.com
access-control-allow-methods: POST, GET, OPTIONS
access-control-expose-headers: AMP-Access-Control-Allow-Source-Origin

I think I miss understood and was changing the wrong element "amp-access-control..." and not "access-control-allow...

I"m investigating this and will reply with my findings later.

[QES]

So on further investigation the issue seems to be that my amp-access and amp-list elements inside of the cached object are being called and rejected because the file is not changing the header to allow the origin to be from the cache (I think).

But what I don't know is how I make my script know that it is coming from the cache and act accordingly I am already checking and thought I was doing that but I don't seem to be.

In the example shown there is some code that seems to say:

var sourceOrigin = req.query.__amp_source_origin;

Look for the __amp_source_origin and set that to be the origin in the header IF it is one of the approved domains.

I do that

res.setHeader('AMP-Access-Control-Allow-Source-Origin', sourceOrigin);

but when I look at the code from the cache in developer mode the error and the called URL seems to be:

Failed to load https://amp.sportsmole.co.uk/amp_ping/82adlfDUPAWVBF7dAF0JXu7hPOhd8yjH9UV4EbGEvImp6oD518g2xlnlRTNs2eIB0.159305393002102671/?rid=82adlfDUPAWVBF7dAF0JXu7hPOhd8yjH9UV4EbGEvImp6oD518g2xlnlRTNs2eIB&uri=https%3A%2F%2Famp.sportsmole.co.uk%2Ffootball%2Fchelsea%2Ftransfer-talk%2Fnews%2Fdortmund-preparing-to-miss-out-on-batshuayi_319745.html&pass=PIK&_=0.219336&host=amp.sportsmole.co.uk&ht=https:&FB=OK&ref=https%3A%2F%2Famp.sportsmole.co.uk%2Ffootball%2Fchelsea%2Ftransfer-talk%2Fnews%2Fdortmund-preparing-to-miss-out-on-batshuayi_319745.html&dynamic&__amp_source_origin=https%3A%2F%2Famp.sportsmole.co.uk

&__amp_source_origin=https%3A%2F%2Famp.sportsmole.co.uk

: The 'Access-Control-Allow-Origin' header has a value 'https://amp.sportsmole.co.uk' that is not equal to the supplied origin. Origin 'https://amp-sportsmole-co-uk.cdn.ampproject.org' is therefore not allowed access. Have the server send the header with a valid value, or, if an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

So it seems the __amp_source_origin is still my source and not the cache source?

[choumx]

Correct, "source origin" is the origin of the published AMP page on your domain.

[QES]

So I'm still confused.

I can get this to work on my domain

https://amp.sportsmole.co.uk and the CORS elements all work perfectly.

However when this sits in the Google Cache it no longer works.

What I don't know is have I got the config the wrong way round or is the Google Cache not doing what it should?

I have looked at this over the last day or so and checked that I am doing what I think I should which is.

There are TWO elements to be played with:

access-control-allow-origin: https://amp-sportsmole-co-uk.cdn.ampproject.org
amp-access-control-allow-source-origin: https://amp.sportsmole.co.uk

__amp_source_origin is inserted by the AMP JS code onto any links and need to be embedded in the JSON CORS response in the

access-control-allow-origin:

Which is what my code does:

header("access-control-allow-origin: $ORIGIN");
header("AMP-Access-Control-Allow-Source-Origin: $HOST");

Where ORIGIN is what has come from __amp_source_origin and HOST is the actual host of the request.

That all seems fine and makes sense but when I look at the CODE inside the Google AMP Cache I see that the __amp_source_origin has not been set to the https://amp-sportsmole-co-uk.cdn.ampproject.org

Failed to load https://amp.sportsmole.co.uk/amp_ping/82adlfDUPAWVBF7dAF0JXu7hPOhd8yjH9UV4EbGEvImp6oD518g2xlnlRTNs2eIB0.87923133212845752/?rid=82adlfDUPAWVBF7dAF0JXu7hPOhd8yjH9UV4EbGEvImp6oD518g2xlnlRTNs2eIB&uri=https%3A%2F%2Famp.sportsmole.co.uk%2Ffootball%2Fchelsea%2Ftransfer-talk%2F&pass=PING&_=0.7456899862965962&host=amp.sportsmole.co.uk&ht=https:&sport=news&art=news&team=news&dynamic&__amp_source_origin=https%3A%2F%2Famp.sportsmole.co.uk

and the error says:

: The 'Access-Control-Allow-Origin' header has a value 'https://amp.sportsmole.co.uk' that is not equal to the supplied origin. Origin 'https://amp-sportsmole-co-uk.cdn.ampproject.org' is therefore not allowed access. Have the server send the header with a valid value, or, if an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled

As you can see the end of the URL

&__amp_source_origin=https%3A%2F%2Famp.sportsmole.co.uk:

Is pointing at amp.sportsmole.c.uk

However if I use a test site and check the netowrk transaction on another page I see correctly that the __amp_source_orgin changes to the source server.

Now am I correctly interpreting how I tell it has come from the Google Cache or is there something I am missing or should the Google Cache change this to be the cache URL?

When I look at the same transaction on the Guardian's web site I can see that the URL enlcloses the guardian URL but the Aollow-Origin is set to the ampproject cdn.

So the question is how do I tell for a JSON object being loaded form the Google Cache that it has come from the Google Cache?

[QES]

Well after a great deal of playing I think I have it working but I'm not sure which solution is working not sure which is the correct one or which is actually working but it seems that the

Allow Origin needs to be set FROM one of the "Request Headers" either Origin or the domain part of the Referer. Then checked against the valid list.

[lannka]

@QES Glad it finally worked. Any part of the doc you felt confusing? You can certainly help us improving the doc.

[QES]

Thanks @lannka I think

https://github.com/ampproject/amphtml/blob/master/spec/amp-cors-requests.md

Could be improved the issue is that

amp-access-control-allow-source-origin must be YOUR origin the publisher domain

access-control-allow-origin must be the location of the file is loaded from (and that must be from an approved list of domains)

However the complication is working out how the cache tells the server that it is this origin, you can check the HEADERS either ORIGIN or REFERER depending on what is set, to me that isn't made clear enough but once you explain it, it becomes obvious :)

I'm not sure that it fully explains that. I'm not sure how best to explain it but I think the confusing thing is that early on:

The AMP CORS security protocol consists of three components:

The CORS Origin header
The AMP-Same-Origin custom header
Source origin restrictions via __amp_source_origin

BUT the CORS origin header is the one marked "Allow-Origin" while the Source is the one marked AMP.
In first reading this I mistakenly thought that it was the AMP one and the Origin one publishers server.

At this point maybe having:

The CORS Origin header [Access-Control-Allow-Origin - the cache or location that the file is being loaded from]
The AMP-Same-Origin custom header
Source origin restrictions via __amp_source_origin

Note: Access-Control-Allow-Origin is the location of the loaded file be that the ampproject cache (or publisher domain) and should be set using the Origin/Referer Header - it must be one of the Goolge/Cloudflare cache domains or one of the publisher domains only. AMP-Access-Control-Allow-Source-Origin should be the publisher domain (as set in the __amp_source_origin) and checked that it is the publishers domain).

[choumx]

/cc @bpaduch for docs improvement.

[calebcordry]

@bpaduch Has the docs feedback here been taken into consideration? Shall I close this issue?

[bpaduch]

No, this feedback appears to have occurred after the major update to the CORs guide. This feedback should be analyzed and used to improve the doc.

[davisshaver]

@bpaduch I checked out the docs. I'm still not totally sure what to do though. @QES did you ever figure out the solution here?