Testcafe: is-docker fails to detect the Docker environment on BitBucket Pipelines

[dthisner]

When running the E2E tests in the CI (BitBucket) using chromium:headless, they break with following error:

Error: Unable to establish one or more of the specified browser connections. This can be caused by network issues or remote device failure.
    at BrowserSet._waitConnectionsOpened (/opt/atlassian/pipelines/agent/build/node_modules/testcafe/src/runner/browser-set.js:83:30)
    at Promise.resolve.then (/opt/atlassian/pipelines/agent/build/node_modules/testcafe/src/runner/browser-set.js:106:35)

Here comes the weird part, the CI run the custom E2E pipeline at 9th November 2019 at 11:34pm, then at 10th November 2019 at 11:34pm it starts failing, exact same code and it was running inside the same dockerfile as the 9th Nov one.

What I have done?
I tried updating TestCafe to latest version - 1.6.1, not working
Updating Gherking-Testcafe to latest version - 2.4.2

Tried running with:
‘chromium ----no-sandbox’
‘chromium:headless ----no-sandbox’
With no success

Tried running firefox:headless, a lot of tests starts too fail, might have to dig into why they are failing...

Updated the docker container with newer versions of everything, same error

Asked TestCafe to list the browsers and Chromium is in the list

+ npx testcafe --list-browsers
firefox
chromium

The Docker file, created to being able to control the node version

FROM debian:jessie

# Replace shell with bash so we can source files
RUN rm /bin/sh && ln -s /bin/bash /bin/sh

# Set environment variables
ENV appDir /var/www/app/current

# Run updates and install deps
RUN echo "deb http://packages.linuxmint.com debian import" >> /etc/apt/sources.list
RUN apt-get update

RUN apt-get install -y -q --force-yes \
  python python-pip chromium chromium-l10n firefox xvfb curl wget\
  && pip install --upgrade awscli s3cmd python-magic \
  && rm -rf /var/lib/apt/lists/* \
  && apt-get -y autoclean

ENV NVM_DIR /usr/local/nvm
ENV NODE_VERSION 10.8.0

# Install NPM packages
RUN curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.29.0/install.sh | bash \
  && source $NVM_DIR/nvm.sh \
  && nvm install $NODE_VERSION \
  && npm install -g serve \
  && nvm alias default $NODE_VERSION \
  && nvm use default

# Set up our PATH correctly so we don't have to long-reference npm, node, &c.
ENV NODE_PATH $NVM_DIR/versions/node/v$NODE_VERSION/lib/node_modules
ENV PATH      $NVM_DIR/versions/node/v$NODE_VERSION/bin:$PATH

EXPOSE 1337 1338

The Old DockerFile that started breaking things:

FROM alpine:edge
USER root
RUN apk --no-cache --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing/ upgrade && \
  apk --no-cache --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing/ add \
  nodejs nodejs-npm chromium firefox xwininfo xvfb dbus eudev ttf-freefont fluxbox procps
RUN apk -v --update add \
  python \
  py-pip \
  groff \
  less \
  mailcap \
  && \
  pip install --upgrade awscli s3cmd python-magic && \
  apk -v --purge del py-pip && \
  rm /var/cache/apk/* && npm \
  install -g serve

RUN mkdir /data
WORKDIR /data

RUN adduser -D user
USER user
EXPOSE 1337 1338

I had to re-do our old one, because Gulp was creating issues with latest Node, so, needed a way to control the node version using NVM

StackOverflow issue: https://stackoverflow.com/questions/58848875/testcafe-chromium-error-unable-to-establish-one-or-more-of-the-specified-brow

Your Environment details:

• testcafe version: 1.6.1
• Gherkin TestCafe . 2.4.2
• node.js version: 10.8.0 (tested with latest and that creates weird issue with gulp)
• command-line arguments: npm run e2e -- --state ci --tags @smokeTest,~@mobile,~@skip -b chromium -v true
• browser name and version: Chromium
• platform and version: debian:jessie
• other:

Runner File:

/* eslint-disable no-console */
const createTestCafe = require('gherkin-testcafe');
let testcafe = null;
const program = require('commander');
const { src } = require('./e2e/config.js');

program
  .option(
    '-b, --browser <browser>',
    'What browser would you like to use? ex "chrome", "chrome:headlesss", multi - ["chrome", "safari"]'
  )
  .option(
    '-t, --tags <tag>',
    'Which tags to run? ex "@login" or "@login","@logOut" (NO space between tags). To exlude Tag(s) add "~" infront of the tag name ex "~@login"'
  )
  .option(
    '-s, --state <state>',
    'Run it "local" or "ci"? "local" is a quick way to just run the tests and make sure everything works. It will start up a local web server.'
  )
  .option(
    '-c, --concurrency <concurrency>',
    'How many copies of Test Cafe do you wanna run? The more = faster tests & requiers more performance from the machine running it'
  )
  .option(
    '-v, --video <video>',
    'Record the test and see what is going on? Add "--video true" and there you go! Recording just for you!'
  )
  .option(
    '-q, --quarantine <quarantine>',
    'If a test failes, it tries running it two more times and if those two times fails, then the test fails, default is false'
  );

program.parse(process.argv);

function selectBrowser() {
  if (program.browser == 'mobile') {
    if (program.state === 'ci') {
      return 'chromium:headless:emulation:device=iphone X';
    } else {
      return 'chrome:emulation:device=iphone X';
    }
  } else if (program.browser == 'multi') {
    return ['chrome', 'firefox', 'safari'];
  } else if (program.browser) {
    return program.browser;
  } else {
    return 'chrome';
  }
}

createTestCafe('localhost', 1337, 1338)
  .then(tc => {
    testcafe = tc;
    let runner = testcafe.createRunner();
    let concurrencyLevel = 1;
    let path = 'desktop';
    let quarantine = true;
    const pathPattern =
      '${DATE}_${TIME}/${TEST_INDEX}/${USERAGENT}${FILE_INDEX}';

    runner.browsers(selectBrowser());

    if (program.state === 'ci') {
      runner.startApp('serve -s build -l 3000', 5000);
    } else if (program.state === 'local') {
      runner.startApp('npm start', 20000);
    }
    if (program.quarantine === 'false') {
      quarantine = false;
    }
    if (program.browser == 'mobile') {
      path = 'mobile';
    }
    if (program.tags) {
      runner.tags(program.tags);
    }
    if (program.concurrency) {
      concurrencyLevel = parseInt(program.concurrency, 10);
    }
    if (program.video) {
      runner.video(`test-reports/${path}/videos`, {
        singleFile: false,
        failedOnly: true,
        pathPattern: `${pathPattern}.mp4`
      });
    }

    return runner
      .src([src])
      .reporter([
        'spec',
        {
          name: 'json',
          output: `./test-reports/${path}/report.json`
        },
        {
          name: 'xunit',
          output: `./test-reports/${path}/report.xml`
        }
      ])
      .screenshots({
        path: `./test-reports/${path}/screenshots`,
        takeOnFails: true,
        pathPattern: `${pathPattern}.png`
      })
      .concurrency(concurrencyLevel)
      .run({
        selectorTimeout: 25000,
        assertionTimeout: 8000,
        pageLoadTimeout: 10000,
        skipJsErrors: true,
        quarantineMode: quarantine
      });
  })
  .then(failedCount => {
    console.log('Tests failed: ' + failedCount);
    testcafe.close();
    process.exit(failedCount ? 1 : 0);
  });

[Dmitry-Ostashev]

Thank you for your report. I was able to reproduce this issue on my side with the TestCafe Docker image. We will research it and notify you as soon as we have any result.

[dthisner]

Do we have ideas why this is happening or a work around?

[Farfurix]

@dthisner

Hello,

We will update this thread once we have any results.

[dthisner]

Wanted to drop of a comment that I was able to make it work by changing:
chromium:headless to 'chromium --headless --no-sandbox'

[matthiasprieth]

same issue here. I ended up using the circleci/node:10.14-browsers image instead of the testcafe/testcafe image. So option 2 in the testcafe docs: https://devexpress.github.io/testcafe/documentation/continuous-integration/bitbucket-pipelines.html

[AndreyBelym]

The cause of the problem is within the is-docker package. Looks like BitBucket changed something in their infrastructure and it can't detect that TestCafe runs in a Docker environment.

Chrome/Chromium can't be run without the --no-sandbox flag in an unprivileged container. There is also the --disable-dev-shm-usage flag that can significantly improve Chrome's performance in a container. They were mentioned in our docs: https://github.com/DevExpress/testcafe/commit/3e6230edd978b0ba1376cf1d4c7d482552c92171#diff-22ed3ab213a1450609aa7531560cc345 but were removed when we made TestCafe to automatically apply them when a Docker environment is detected.

So you can manually specify --no-sandbox and --disable-dev-shm-usage to Chromium as a workaround.

I've checked the is-docker sources and the way it uses to detect Docker looks not enough reliable to me. My suggestions for solving this case:

1. Bring back --no-sandbox and --disable-dev-shm-usage flags to the docs, so our users can read about them and apply them manually if our automatic detection fails.
2. Hack our official image to always apply these flags.
3. Try to invest some time and fix https://github.com/sindresorhus/is-docker/issues/7 if is-docker's maintainer doesn't care about it.

@VasilyStrelyaev @miherlosev