Amphtml: amp-base-carousel e2e test is flaky

Created on 24 Aug 2019 · 12Comments · Source: ampproject/amphtml

 1) AMP carousel arrows with custom arrows
       firefox
          Standalone environment 

             should go to the next slide:
      AssertionError: expected { Object (bottom, height, ...) } to have property 'x' of 0, but got 800
      + expected - actual
      -800
      +0

Saw the above failures a couple of times this week. It has always been in firefox.
https://travis-ci.org/ampproject/amphtml/jobs/576131623

High Priority Flaky Tests components

Source

lannka

All 12 comments

Another failure: https://travis-ci.org/ampproject/amphtml/jobs/576888755

 1) AMP carousel
       chrome
          Viewer environment 

             should snap when scrolling:
      AssertionError: expected 4800 to equal 5600
      + expected - actual
      -4800
      +5600

This seems to have started after migrating e2e tests to MacOS (to support Safari).

lannka on 26 Aug 2019

Bumping up its priority, as it's continuously failing now which blocking PR merge and release.

lannka on 26 Aug 2019

Bumping the Priority down since the test is skipped and the component is experimental

nainar on 20 Sep 2019

Bumping this issue to P1, since we have newly flaky AMP carousel tests.

Logs from today's master builds:

https://travis-ci.org/github/ampproject/amphtml/jobs/673119438#L2311
https://travis-ci.org/github/ampproject/amphtml/jobs/673154489#L2311

  1) AMP carousel
       chrome
          Viewer environment 

             should have the correct scroll position when resizing:
      AssertionError: expected { Object (bottom, height, ...) } to have property 'x' of 0, but got -900
      + expected - actual
      --900
      +0


Executed 434 of 484 (Skipped 51) 1 FAILED

/cc @ampproject/wg-ui-and-a11y for triage and assignment.

rsimha on 10 Apr 2020

@ampproject/wg-components This suite of tests has been failing quite regularly of late and is affecting most CI runs. For example: this one.

The error is in beforeEach() (likely because the carousel arrows aren't appearing on time). Assuming the fix isn't straightforward, okay if I send out a PR to temporarily skip this suite?

rsimha on 21 Jan 2021

@rsimha SGTM. I'll take a look ASAP.

micajuine-ho on 21 Jan 2021

@rsimha As a point of clarification, the above screenshot looks relevant to 1.0 (bento) amp-base-carousel and may not be related to the other pre-existing flaky tests this issue was filed for. I'll take a look. I see that it is passing most of the time, is there a better resource that shows the higher rates of failure?

caroqliu on 21 Jan 2021

@caroqliu Here is the list of master builds on CircleCI: https://app.circleci.com/pipelines/github/ampproject/amphtml?branch=master. If you click through failing builds, most if not all the E2E and Experiment A/B/C failures are due to this test suite. (Obviously, feel free to ignore the other failing suites like unit / performance. I'm following up on those separately.)

rsimha on 21 Jan 2021

👍1

Thank you! I'll look briefly and send out a skip PR by EOD if there isn't an obvious fix.

caroqliu on 21 Jan 2021

👍1

Some notes after reviewing runs 99-213:

amp-base-carousel test snap property is a 0.1 failure with StaleElementReferenceError: stale element reference: element is not attached to the page document
AMP carousel arrows when non-looping is a 1.0 failure with TimeoutError: Waiting for element to be located .arrow-prev-34d526e
AMP carousel should snap when scrolling is a 1.0 failure with TimeoutError: Waiting for element to be located .hide-scrollbar-34d526e
AMP carousel should snap on center point is a 1.0 failure with TimeoutError: Waiting for element to be located .hide-scrollbar-34d526e

@micajuine-ho want to investigate the stale element reference in 0.1?

For the others I wonder if increasing the timeouts for those test files will help. I'm not able to reproduce any of them locally unfortunately and wonder if it has to do with the processing speed of the test runner. :/ Are Circle CI tests still non-blocking? Would it be possible to increase the timeout for the next day to see if it helps, and skip otherwise?

Also for more obvious version differentiation I'll send a PR to update the test names to include their versions.

caroqliu on 21 Jan 2021

👍1

Wow, that was comprehensive. Thanks for the careful analysis, @caroqliu!

For the others I wonder if increasing the timeouts for those test files will help. I'm not able to reproduce any of them locally unfortunately and wonder if it has to do with the processing speed of the test runner. :/ Are Circle CI tests still non-blocking? Would it be possible to increase the timeout for the next day to see if it helps, and skip otherwise?

We're currently using the "XLarge" VM provided by CircleCI with the latest stable version of Chrome, so tests should run fairly fast.

https://github.com/ampproject/amphtml/blob/ca87c8e7f762964c5bd21aeab438d1b42d051aa2/.circleci/config.yml#L21-L25

https://github.com/ampproject/amphtml/blob/ca87c8e7f762964c5bd21aeab438d1b42d051aa2/.circleci/config.yml#L35-L36

CircleCI tests are indeed non-blocking for now (as is the ampproject/tests/e2e (nomodule) test status). The plan is to make it blocking in the near future and switch off of Travis. Not opposed to increasing the timeouts, since a few extra seconds can help us avoid multiple minutes of re-running time.

rsimha on 21 Jan 2021

amp-base-carousel test snap property is a 0.1 failure with StaleElementReferenceError: stale element reference: element is not attached to the page document

I have more data points after observing weekend builds. The remaining flake is from the 0.1 tests, and likely has to do with initial state. This might need a fix for the beforeEach(), since I'm not sure skipping individual tests is sufficient. For example, this build:

  1) amp-base-carousel:0.1 - arrows when non-looping
       chrome
          Viewer environment 
           "before each" hook for "should have the arrows in the correct initial state":
     Error: Server terminated early with status 1
      at /home/circleci/project/build-system/tasks/e2e/node_modules/selenium-webdriver/remote/index.js:248:24
      at processTicksAndRejections (internal/process/task_queues.js:93:5)

  2) amp-base-carousel:0.1 - arrows when non-looping
       chrome
          Viewer environment 
           "after each" hook for "should have the arrows in the correct initial state":
     Error: Server terminated early with status 1
      at /home/circleci/project/build-system/tasks/e2e/node_modules/selenium-webdriver/remote/index.js:248:24
      at processTicksAndRejections (internal/process/task_queues.js:93:5)

There's also this failure in the 0.2 tests:

  1) AMP carousel 0.2 with responsive slides
       chrome
          Standalone environment 

             layout properly and show images:

      AssertionError: expected 0 to be above 0
      + expected - actual


      at _callee2$ (extensions/amp-carousel/0.2/test/test-e2e/test-repsonsive-slides.js:44:46)
      at tryCatch (build-system/tasks/e2e/node_modules/babel-regenerator-runtime/runtime.js:61:40)
      at GeneratorFunctionPrototype.invoke [as _invoke] (build-system/tasks/e2e/node_modules/babel-regenerator-runtime/runtime.js:329:22)
      at GeneratorFunctionPrototype.prototype.<computed> [as next] (build-system/tasks/e2e/node_modules/babel-regenerator-runtime/runtime.js:94:21)
      at asyncGeneratorStep (extensions/amp-carousel/0.2/test/test-e2e/test-repsonsive-slides.js:5:103)
      at _next (extensions/amp-carousel/0.2/test/test-e2e/test-repsonsive-slides.js:7:194)
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (internal/process/task_queues.js:93:5)

rsimha on 25 Jan 2021

Was this page helpful?

0 / 5 - 0 ratings