Site-kit-wp: E2E test consistency

[aaemnnosttv]

Bug Description

E2E tests have become less stable in the past few weeks resulting in a few specific tests failing roughly 10-15% of the time and succeed after re-running.

Tests

The most common failing test is:

 FAIL  specs/modules/analytics/write-scope-requests.test.js (22.27s)
  ● Analytics write scope requests › prompts for additional permissions during a new Analytics property creation if the user has not granted the Analytics edit scope
    TimeoutError: Element .mdc-dialog__actions .mdc-button (text: "/proceed/i") not found
      waiting for function failed: timeout 500ms exceeded

Another test which is subject to failure is from the Admin tracking tests

 FAIL  specs/admin-tracking.test.js (61.33s)
  ● initialization on load for Site Kit screens › splash page › loads tracking when opted-in
    TimeoutError: Timeout exceeded while waiting for event
      36 |  await page.waitForSelector( '#googlesitekit-opt-in' );
      37 |  await expect( page ).toClick( '#googlesitekit-opt-in' );
    > 38 |  await page.waitForResponse( ( res ) => res.url().match( 'wp/v2/users/me' ) );
         |             ^
      39 | }
      40 | 
      41 | describe( 'management of tracking opt-in/out via settings page', () => {
      at Timeout.setTimeout (../../node_modules/puppeteer/lib/helper.js:196:24)
        -- ASYNC --
      at Page.<anonymous> (../../node_modules/puppeteer/lib/helper.js:111:15)
      at waitForResponse (specs/admin-tracking.test.js:38:13)
      at tryCatch (../../node_modules/regenerator-runtime/runtime.js:45:40)
      at Generator.invoke [as _invoke] (../../node_modules/regenerator-runtime/runtime.js:274:22)
      at Generator.prototype.(anonymous function) [as next] (../../node_modules/regenerator-runtime/runtime.js:97:21)
      at asyncGeneratorStep (../../node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24)
      at _next (../../node_modules/@babel/runtime/helpers/asyncToGenerator.js:25:9)

https://travis-ci.com/github/google/site-kit-wp/jobs/363574273

Both tests failing

• https://travis-ci.com/github/google/site-kit-wp/jobs/362232344
• https://travis-ci.com/github/google/site-kit-wp/jobs/361847838

---

_Do not alter or remove anything below. The following sections will be managed by moderators only._

Acceptance criteria

• All E2E tests should consistently pass or fail

Implementation Brief

This issue may require a follow-up PR but we need many iterations of the build to see due to the inconsistent nature. This should at least be an improvement.

• Merge #1783 - improves the stability of the two tests above

QA Brief

Changelog entry

[felixarntz]

IB ✅

[felixarntz]

@aaemnnosttv Let's see if this makes it more stable. Moving to QA

[aaemnnosttv]

@felixarntz looks like there's still an issue with this test as I just saw it fail in the same place.
https://travis-ci.com/github/google/site-kit-wp/jobs/365871008

 FAIL  specs/modules/analytics/write-scope-requests.test.js (53.821s)
  ● Analytics write scope requests › prompts for additional permissions during a new Analytics profile creation if the user has not granted the Analytics edit scope
    TimeoutError: waiting for selector ".mdc-dialog--open" failed: timeout 30000ms exceeded
      228 |         // Click on confirm changes button and wait for permissions modal dialog.
      229 |         await expect( page ).toClick( '.mdc-button', { text: /configure analytics/i } );
    > 230 |         await page.waitForSelector( '.mdc-dialog--open' );
          |                    ^
      231 | 
      232 |         // Click on proceed button and wait for oauth request.
      233 |         await Promise.all( [
      at new WaitTask (../../node_modules/puppeteer/lib/DOMWorld.js:549:28)
      at DOMWorld._waitForSelectorOrXPath (../../node_modules/puppeteer/lib/DOMWorld.js:478:22)
      at DOMWorld.waitForSelector (../../node_modules/puppeteer/lib/DOMWorld.js:432:17)
      at Frame.waitForSelector (../../node_modules/puppeteer/lib/FrameManager.js:627:47)
      at Frame.<anonymous> (../../node_modules/puppeteer/lib/helper.js:112:23)
      at Page.waitForSelector (../../node_modules/puppeteer/lib/Page.js:1095:29)
      at waitForSelector (specs/modules/analytics/write-scope-requests.test.js:230:14)
      at tryCatch (../../node_modules/regenerator-runtime/runtime.js:45:40)
      at Generator.invoke [as _invoke] (../../node_modules/regenerator-runtime/runtime.js:274:22)
      at Generator.prototype.(anonymous function) [as next] (../../node_modules/regenerator-runtime/runtime.js:97:21)
        -- ASYNC --
      at Frame.<anonymous> (../../node_modules/puppeteer/lib/helper.js:111:15)
      at Page.waitForSelector (../../node_modules/puppeteer/lib/Page.js:1095:29)
      at waitForSelector (specs/modules/analytics/write-scope-requests.test.js:230:14)
      at tryCatch (../../node_modules/regenerator-runtime/runtime.js:45:40)
      at Generator.invoke [as _invoke] (../../node_modules/regenerator-runtime/runtime.js:274:22)
      at Generator.prototype.(anonymous function) [as next] (../../node_modules/regenerator-runtime/runtime.js:97:21)
      at asyncGeneratorStep (../../node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24)
      at _next (../../node_modules/@babel/runtime/helpers/asyncToGenerator.js:25:9)

Moving back to execution, I'll continue looking into this.

[aaemnnosttv]

@felixarntz Moving this back to IB to define what needs to be done here as the PR in the current IB was already merged but did not fix everything. The current stability is better than before so this isn't as urgent as before either.

[aaemnnosttv]

@eugene-manuilov maybe you could take a look at this when you get a chance?

[eugene-manuilov]

@aaemnnosttv i'll try to look at it later this week or maybe at the beginning of the next one.