Mocha: run browser tests on GHA

[boneskull]

Once #4459 lands, the only thing building on Travis-CI will be the browser tests. I'd like to see those moved as well, but I'm struggling to get it working (example).

This CI config is where I left off:

  test-browser:
    name: 'Test Browsers'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v1
        with:
          node-version: 12
      - name: Get npm cache directory
        id: npm-cache
        run: |
          echo "::set-output name=dir::$(npm config get cache)"
      - uses: actions/checkout@v2
      - name: 'Cache node_modules'
        uses: actions/cache@v2
        with:
          path: '${{ steps.npm-cache.outputs.dir }}'
          key: "test-browser-${{ hashFiles('**/package-lock.json') }}"
          restore-keys: |
            test-browser-
      - name: Update to Latest Chrome Stable
        run: |
          sudo apt-get update
          sudo apt-get --only-upgrade install google-chrome-stable
      - name: Install Dependencies
        run: npm ci
      - name: Open Sauce Connect Proxy
        uses: saucelabs/sauce-connect-action@master
        with:
          username: '${{ secrets.SAUCE_USERNAME }}'
          accessKey: '${{ secrets.SAUCE_ACCESS_KEY }}'
          tunnelIdentifier: 'github-actions-${{ github.run_id }}_${{ github.run_number }}'
          sePort: 4444
      - name: Run All Browser Tests
        run: npm start test.browser

In addition, some changes to karma.conf.js will be necessary, e.g.:

// karma.conf.js:L86

  // configuration for CI mode
  if (env.CI) {
    console.error('CI mode enabled');
    if (env.GITHUB_WORKFLOW) {
      console.error('GitHub Actions detected');
      bundleDirPath = path.join(BASE_BUNDLE_DIR_PATH, env.GITHUB_RUN_ID);
      // correlate build/tunnel with Travis
      sauceConfig = {
        build: `GitHub Action #${env.GITHUB_RUN_NUMBER} (${env.GITHUB_RUN_ID})`,
        tunnelIdentifier: `github-actions-${env.GITHUB_RUN_ID}_${env.GITHUB_RUN_NUMBER}`,
        startConnect: false
      };
      console.error('Configured SauceLabs');
    } else {

• For whatever reason, I could not get Karma to try to use a port other than 4444. This seems to be the default, but the Sauce Connect default seems to be 4445.

[boneskull]

I wonder if @christian-bromann is bored and might want to shoot a pointer my way 😇

[christian-bromann]

Let me take a look at that!

[christian-bromann]

Ok, some things I noted:

        [sauceBrowser]: {
          base: 'SauceLabs',
          browserName,
          version,
          platform
        }

these are old capabilities, I recommend to update them to:

        [sauceBrowser]: {
          base: 'SauceLabs',
          browserName,
          browserVersion: version,
          platformName: platform
        }

Then it seems that it tries to run a browser in CI instead of on Sauce Labs. Let me try to setup and example repository to see if I run into the same.

[christian-bromann]

I created a minimal example project that shows the config needed to run the tests on Sauce Labs. Looking at the error from the failing GH workflow above I assume something is not correct with the fact that the tests are suppose to run on Saucelabs.

Here are some references that might help resolve the issue:

• Minimal example project: https://github.com/christian-bromann/saucelabs-karma-example
• Passing GH workflow with full Karma config: https://github.com/christian-bromann/saucelabs-karma-example/runs/1241406904?check_suite_focus=true

Hope this helps to investigate. Let me know if I can help with anything else.

[boneskull]

@christian-bromann Thanks for taking a look. Karma works fine on Travis-CI as configured, so I was hoping there was some misconfiguration around Sauce Connect in the GH action that you'd recognize.

Karma does _not_ start sauce connect itself in CI, since we want to reuse the proxy for several runs (otherwise it's slow).

Karma _does_ try to run a browser on the CI agent--it runs headless Chrome regardless of whether or not SauceLabs is used. If it's in CI, Karma will run headless Chrome + the saucelabs browsers. If it's not in CI, it just runs headless Chrome.

[boneskull]

Anyway, I'll take a look at that example repo and see if I can't get it working.