Amphtml: Percy and Chrome 41 / IE11 support

[aghassemi]

WRS (Web Rendering Service) that crawlera use to render a website for indexing uses Chrome 41. We should have Percy take at least a few screenshots with Chrome 41 in case we ship code that breaks rendering in that old version. We don't really need to run every test in Chrome 41, just a few with images and text would do.

Currently I am investigating a breackage in https://github.com/ampproject/amphtml/issues/20675 , we should really catch these much sooner with visual diffing.

/To @rsimha @danielrozenberg

[rsimha]

VIsual diffs do a two step rendering:

1. Use latest chrome locally to run JS and generate a DOM snapshot
2. Use Chrome and Firefox on the Percy side to render the DOM snapshot and generate an image to compare with the baseline

@aghassemi Which of these steps do you think breaks on Chrome 41? If it's step 1, I'd imagine we can use an older version of chrome with puppeteer. If it's step 2, we'd have to reach out fo Percy to see if this is an option.

Assigning to @danielrozenberg for further comment.

[aghassemi]

@rsimha I'd say in majority of cases it would be step 1 as breakages are likely to be JS related. It is possible to be step 2 as well if we use some newer CSS stuff, but much less likely.

After I figure out what happened in #20675 , I will report back.

[aghassemi]

Well, #20675 turned out to be neither, and not an AMP bug at all :) Anyway, I think if we start with (1) for now, it would be great.

[rsimha]

I did some reading up, and puppeteer, which we use to run our visual tests, is available only from Chrome 59. It doesn't look like we can use Chrome 41 with it. See https://stackoverflow.com/questions/49176356/puppeteer-with-old-chrome-versions

In addition, Travis's chrome addon supports just the latest stable or beta version of Chrome, and doesn't support multiple versions or the use of a particular version number, both of which are needed to run against Chrome 41 on Travis. See https://docs.travis-ci.com/user/chrome#selecting-a-chrome-version

Not sure what else we can do here. @danielrozenberg any ideas?

[ampprojectbot]

This issue doesn't have a category which makes it harder for us to keep track of it. @danielrozenberg Please add an appropriate category.

[danielrozenberg]

I don't think there's anything reasonable we can do here without jumping through too many hoops. I'm leaning towards closing this, unless you think it's something we _really_ need?

[aghassemi]

@danielrozenberg we need something if not visual tests. We used to run tests in Chrome 45 (under a---oldchrome) which seems to be gone now as well (oldchrome tests were there for WRS and crawlers) . Maybe we can run couple of the simple integration tests on old Chrome on Travis instead of saucelabs?

[rsimha]

Might be possible to reinstate chrome 45 on sauce labs. @estherkim is going to look into this soon.

[aghassemi]

@danielrozenberg how about IE 11? Pretty much same situation as Chrome 41, just need a few sanity tests to run. Could have helped with https://github.com/ampproject/amphtml/issues/21144 P0

(updated the title of this issue accordingly)
/cc @choumx

[danielrozenberg]

This is related to #24516. We asked Percy to support beta and legacy browsers and they said they'll consider it. Until then, this is blocked...