This test broke starting with Chrome 71, and was disabled in #19782. Please triage and fix
Disabled test: https://github.com/ampproject/amphtml/pull/19782/files#diff-342622682fdb4e2fbb7ff8b589e7770bR138
@cvializ let's quickly investigate to see if this is actually a code breakage issue in Chrome 71 or purely test related.
sounds good, I'll take a look
@aghassemi @danielrozenberg The test consistently succeeds for me in Chrome 70.0.3538.110 (long-running Chrome window FTW), Chrome 71.0.3578.80 and Chrome 73.0.3637.0, so I'm not sure what was causing it to break on Travis. Was there some condition that causes it to fail other than not being skipped? Was it non-deterministically failing?
It was failing only some of the times (hence flaky). The logs didn't say anything save for a standard test timeout message
OK, I was able to make it fail consistently in Sauce Labs. The test makes a request to localhost:38612
, which hosts the test request server in app.js
. For some reason, this port refuses any connections in Windows + Chrome 71, but when the test runs in Windows + Chrome 70, the request returns a response and the test succeeds. I verified this with the test server running on my local machine, and Sauce Labs VMs connecting through sc
.
Description | Image
--|--
Chrome 70 request|
Chrome 70 localhost:31862|
Chrome 71 request|
Chrome 71 localhost:31862|
I believe this is the root cause. This explains why the test succeeds on Chrome 71 locally, but not on Sauce Labs. Would Karma or Sauce Labs configure the server endpoint differently for different versions of Chrome?
/to @rsimha + @estherkim for Carlos's question
Very nice sleuthing and analysis, @cvializ!
I poked around the Sauce labs documentation to see if accessing localhost
via a port on Sauce VMs is a known problem. I found this year-old article about older Safari browsers, which may offer a hint about what's going on.
There are some port
related questions in the Sauce Connect FAQs page. This one, titled "Can I Access Applications on localhost?" seemed like it might lead us to a fix.
Maybe we could try changing the port number from 38612
to one of the known proxied ports?
I can give this a try, let me change the port to 8081 (since 8080 is warned not to work on Android, and although we don't test it now, we might in the future) and I'll report my findings
Most helpful comment
OK, I was able to make it fail consistently in Sauce Labs. The test makes a request to
localhost:38612
, which hosts the test request server inapp.js
. For some reason, this port refuses any connections in Windows + Chrome 71, but when the test runs in Windows + Chrome 70, the request returns a response and the test succeeds. I verified this with the test server running on my local machine, and Sauce Labs VMs connecting throughsc
.Description | Image



--|--
Chrome 70 request|
Chrome 70 localhost:31862|
Chrome 71 request|
Chrome 71 localhost:31862|
I believe this is the root cause. This explains why the test succeeds on Chrome 71 locally, but not on Sauce Labs. Would Karma or Sauce Labs configure the server endpoint differently for different versions of Chrome?