Amphtml: Question: Blocking Bots and Scrappers on reading amp pages from third party cache

[jalajc]

(Sorry for not sticking to issue guideline as this was generic concern)

Context:
I am impressed the way AMP contents is delivered, However, as a niche content creator, I am little concerned over the scrapping door the amp caches open up. I don't want scrappers and bots (except a few ) to crawl the site or store and _re-use content_.

Question:

1. Is there a documentation/reference/case where I can find how _Google AMP cache_ or third party cache validates humans vs bots (like reCaptcha).
   
   
   
   • based on this I can choose to allow only the legit third party cache to crawl my origin server for AMP cache
   
   
   • help propagate standards to third-party cache
   
   

Feature Request:

1. A way I can configure rate limiting or any other access features on reading third party AMP cache? Sort of amp-manifest embedded in amp-pages which indicates rate limits, humans only access control logic, bots control (like robots.txt) etc...
   
   
   
   • allow publishers to be in control of cache access.
   
   

As a standard practice third party cache providers should also document how they delineate bots and scrappers to dose-off publishers' concern of the their content.

PS:

I have gone through amp-access docs and found that currently only browser visibility based option is available and server side access control option is still contemplated.

[jpettitt]

I can't answer for google but I can tell the the cdn.ampproject.org rate limits becasue we hit it they other day loading our own pages (we were forcing updates by reading expired pages).

[jridgewell]

Whoo, don't know who to assign this to so @adelinamart gets it.

[lannka]

@dvoytenko @rudygalfi is something amp-access handles?

[adelinamart]

@jalajc the amp team is planning to add support for recaptcha in AMP as per #2273
Let me know if that answers your question. Also, you can read more about how AMP cache works at https://github.com/ampproject/amphtml/blob/master/spec/amp-cache-guidelines.md

[dvoytenko]

It sounds like the use case here is somewhat different. The desire is to protect from bot scrapping which is not what access/reCaptcha do - they do client-side obfuscation or user action guarding.

I believe all AMP Caches are required to fully respect robots.txt and thus have some protection from scrapping. /to @vitaliybl to clarify.

[jalajc]

@jalajc the amp team is planning to add support for recaptcha in AMP as per #2273
Let me know if that answers your question. Also, you can read more about how AMP cache works at

Not completely. scrappers and bots are not practically affected by component visibility on UI. Javascript is always 'optional'. Good browsers and people will mostly allow scripts. But certainly scrappers find ways to fetch content without javascripts.
@dvoytenko's comment is worth notable in this regard.

So workable solution is a _intermediate service_ (from cache provider) which first validates legitimate calls and then only allow contents to be fetched rather than content is fetched from cache and browser(viewer) validating if a legitimate call?Not sure how reCaptcha will work in AMP.

Also server side rate limits are also desired if in case scrapper abuse faster than before availability of content.
Note that in case scrapper/bot read amp-caches _programmatically_, publisher cannot even find a trace (as javascript based analytics won't work, and server logs are not available at publishers end.). This is scary. However, If scrapper/bot reads from my server I can always introduce rate limits and block access at the webserver (such as apache, ngix) level, because I have _visibility_ and _control_ on who is visiting my server.

@dvoytenko problem with robot.txt is, except the top few, I have seldom found bots to respect robot.txt. And in many cases you won't know if it is bot/scrapper as UA can be faked.
But yes, if strict enforcement comes from AMP cache implementations along with rate limits, this could work.

[jpettitt]

Note that in case scrapper/bot read amp-caches programmatically, publisher cannot even find a trace (as javascript based analytics won't work, and server logs are not available at publishers end.). This is scary.

This is a hard one to solve, particularly as amp caches proliferate. If I were making an evil bot I'd distribute it's reads across all the amp caches since they can all serve the same content. This makes it really hard to prevent scraping if the entity doing it has a good browser emulation.

Adding black/white lists would be a nightmare to manage, and be very brittle and error prone.

For security reasons amp caches can't report page requests to publishers. Doing so would give the publisher visibility into their page being pre-rendered in a search result when the user hasn't yet requested to view it. That would defeat one of the main reasons the cache exists.

[vitaliybl]

@jalajc there are a lot of issues here, I'll try to unpack.

1. Google AMP Cache is closed from well-behaved bots via its robots.txt
2. Today, we don't do anything special to distinguish between bots and real users.
3. Google AMP Cache is trying to respect user privacy and does not use cookies. That makes it near-impossible to persist user/bot classification, and performing classification on every request via captcha or alike is obviously not possible.
4. AMP access server-side, by design, would mask the information that you'd want to see to make access decisions (you'll just see Google UA in your logs).
5. Finally, you can authenticate Google AMP Cache just like you can authenticate Googlebot, by performing reverse-DNS lookup on the connecting IP.

I will try to find time to talk to folks who fight scrapers in Google Search, and see if they have any ideas that could be implemented within the product constraints (particularly, 3).

[vitaliybl]

One more thing before I forget: @dvoytenko , Google AMP Cache itself does not obey robots.txt because it acts on behalf of a user.

[ampprojectbot]

This issue seems to be in Pending Triage for awhile. @vitaliybl Please triage this to an appropriate milestone.

[ampprojectbot]

This issue seems to be in Pending Triage for awhile. @vitaliybl Please triage this to an appropriate milestone.

[sunco007]

I was looking for the same.. I tend to use an ajax call after N seconds. Now I see you can do something using triggers - trackPageview - visibilitySpec - https://ampbyexample.com/components/amp-analytics/ (the cat sample)

[Gregable]

Closing this as it's an AMP Cache issue, not AMP Project related, and there is no actionable recommendation for what to implement.