Amphtml: Integration tests on Safari run in parallel instead of in series

[estherkim]

This is a bug because every test's iframe is created in a single Karma window. It affects common assertions like element position, and fails to ensure silo'ed environments per test.

Note that this is seen in Safari only, and not Firefox or Chrome.

Thanks @zhouyx for the find!

[estherkim]

Assigning @rsimha to investigate, since we switched to karma-safarinative-launcher when we made the move to GH Actions

[rsimha]

Interesting. Thanks for filing. Will take a look. (I was under the impression that Karma did this for all OSs, but maybe there's something special going on with Safari.)

Meanwhile, if someone can post a quick example (say, a pair of tests that interact badly), it will help investigate.

[estherkim]

If you run integration tests in Safari and Firefox, you'll see how Firefox reloads a window while Safari appends iframes to the same one.

gulp integration --nobuild --compiled --safari
gulp integration --nobuild --compiled --firefox

[zhouyx]

https://github.com/ampproject/amphtml/blob/master/test/integration/test-amp-ad-3p.js

is a test that fail when interact with other tests.

[rsimha]

Just did a little investigating. The reason we had to stop using karma-safari-launcher was because of https://github.com/karma-runner/karma-safari-launcher/issues/29. With current versions of Safari, opening a local HTML file (as Karma does) causes Safari to pop up a modal dialog that interrupts the test runner.

The issue comments suggest workarounds like karma-safarinative-launcher, karma-safari-private-launcher, and karma-safari-applescript-launcher. I can try the others to see which one works best.

[rsimha]

Figured out the problem. There is code in _init_tests.js to clean up iframes between tests, but this was disabled for Safari because it didn't work well with the old karma-safari-launcher. I enabled it with the current karma-safarinative-launcher and noticed that all tests' iframes were at 0,0. Will send out a PR shortly.