Amphtml: Fix broken test 'should be protected from recursive-submission' in `test-integration-form.js`

Created on 11 Dec 2018  路  8Comments  路  Source: ampproject/amphtml

This test broke starting with Chrome 71, and was disabled in #19782. Please triage and fix

Disabled test: https://github.com/ampproject/amphtml/pull/19782/files#diff-342622682fdb4e2fbb7ff8b589e7770bR138

High Priority Flaky Tests Bug

Most helpful comment

OK, I was able to make it fail consistently in Sauce Labs. The test makes a request to localhost:38612, which hosts the test request server in app.js. For some reason, this port refuses any connections in Windows + Chrome 71, but when the test runs in Windows + Chrome 70, the request returns a response and the test succeeds. I verified this with the test server running on my local machine, and Sauce Labs VMs connecting through sc.

Description | Image
--|--
Chrome 70 request|screen shot 2018-12-11 at 17 17 13
Chrome 70 localhost:31862|screen shot 2018-12-11 at 17 20 38
Chrome 71 request|screen shot 2018-12-11 at 17 19 03
Chrome 71 localhost:31862|screen shot 2018-12-11 at 17 19 50

I believe this is the root cause. This explains why the test succeeds on Chrome 71 locally, but not on Sauce Labs. Would Karma or Sauce Labs configure the server endpoint differently for different versions of Chrome?

All 8 comments

@cvializ let's quickly investigate to see if this is actually a code breakage issue in Chrome 71 or purely test related.

sounds good, I'll take a look

@aghassemi @danielrozenberg The test consistently succeeds for me in Chrome 70.0.3538.110 (long-running Chrome window FTW), Chrome 71.0.3578.80 and Chrome 73.0.3637.0, so I'm not sure what was causing it to break on Travis. Was there some condition that causes it to fail other than not being skipped? Was it non-deterministically failing?

It was failing only some of the times (hence flaky). The logs didn't say anything save for a standard test timeout message

OK, I was able to make it fail consistently in Sauce Labs. The test makes a request to localhost:38612, which hosts the test request server in app.js. For some reason, this port refuses any connections in Windows + Chrome 71, but when the test runs in Windows + Chrome 70, the request returns a response and the test succeeds. I verified this with the test server running on my local machine, and Sauce Labs VMs connecting through sc.

Description | Image
--|--
Chrome 70 request|screen shot 2018-12-11 at 17 17 13
Chrome 70 localhost:31862|screen shot 2018-12-11 at 17 20 38
Chrome 71 request|screen shot 2018-12-11 at 17 19 03
Chrome 71 localhost:31862|screen shot 2018-12-11 at 17 19 50

I believe this is the root cause. This explains why the test succeeds on Chrome 71 locally, but not on Sauce Labs. Would Karma or Sauce Labs configure the server endpoint differently for different versions of Chrome?

/to @rsimha + @estherkim for Carlos's question

Very nice sleuthing and analysis, @cvializ!

I poked around the Sauce labs documentation to see if accessing localhost via a port on Sauce VMs is a known problem. I found this year-old article about older Safari browsers, which may offer a hint about what's going on.

There are some port related questions in the Sauce Connect FAQs page. This one, titled "Can I Access Applications on localhost?" seemed like it might lead us to a fix.

Maybe we could try changing the port number from 38612 to one of the known proxied ports?

I can give this a try, let me change the port to 8081 (since 8080 is warned not to work on Android, and although we don't test it now, we might in the future) and I'll report my findings

Was this page helpful?
0 / 5 - 0 ratings

Related issues

jasti picture jasti  路  50Comments

zhouyx picture zhouyx  路  103Comments

sebastianbenz picture sebastianbenz  路  43Comments

ericlindley-g picture ericlindley-g  路  117Comments

vockalimo picture vockalimo  路  49Comments