Testcafe: Browser disconnecting in Docker

[grant-mccarriagher]

What is your Test Scenario?

Chrome/Firefox headless. Logging in, clicking some checkbox elements, verifying some element text. This was mostly a PoC for using TestCafe as our e2e framework, so the tests aren't too complicated.

What is the Current behavior?

Locally, the tests seem to pass every time, on every browser we use. In Docker however, the tests are extremely flakey. Often they fail to run at all, with the following error:
```e2e_1 | Using locally installed version of TestCafe.

e2e_1 | Running tests in:

e2e_1 | - Firefox 62.0.0 / Linux 0.0.0

e2e_1 |

e2e_1 | Filters

e2e_1 | ERROR The Firefox 62.0.0 / Linux 0.0.0 browser disconnected. This problem may appear when a browser hangs or is closed, or due to network issues.

e2e_1 |

e2e_1 | Type "testcafe -h" for help.

Other times, two or three tests complete, and then the above error occurs. We have tried many approaches to running TestCafe in a container, including:
1. Using the official TestCafe image directly, as the docs direct. This also seems to prevent us from using the config file introduced in 1.0.
2. Using an image based on the TestCafe official image (this is the approach we are using in the code samples for this issue).
3. Creating an image from scratch based on Ubuntu.

All approaches result in the same intermittent error as posted above. Note that when we try with Chromium, we are using the `--no-sandbox` flag.

### What is the Expected behavior?
The tests pass running in a container, just as they do running locally.

### What is your web application and your TestCafe test code?
It's not publicly available, so I'd prefer to message it directly if contact information can be provided. I can also provide read access to a copy of the repo.

<details>
<summary>Your complete test code (or attach your test files):</summary>

<!-- Paste your test code here: --> 
Dockerfile:

FROM testcafe/testcafe

USER root

RUN apk --no-cache add \
ffmpeg

RUN mkdir /home/e2e
WORKDIR /home/e2e
COPY . .
RUN chmod -R 757 /home/e2e

USER user

RUN npm install

ENV NODE_PATH=/home/e2e/node_modules
ENV PATH=${PATH}:/home/e2e/node_modules/.bin

EXPOSE 1337 1338

ENTRYPOINT ["/home/e2e/testcafe-docker.sh"]

entrypoint:

!/bin/sh

XVFB_SCREEN_WIDTH=${1920}
XVFB_SCREEN_HEIGHT=${1080}

dbus-daemon --session --fork
Xvfb :1 -screen 0 "${XVFB_SCREEN_WIDTH}x${XVFB_SCREEN_HEIGHT}x24" >/dev/null 2>&1 &
export DISPLAY=:1.0
fluxbox >/dev/null 2>&1 &
testcafe "$@"

docker-compose:

version: '3.5'
services:
e2e:
entrypoint: /home/e2e/testcafe-docker.sh
build:
context: .

.testcaferc.json:

{
"browsers": "firefox:headless",
"src": ["tests/*"],
"reporter": "spec",
"skipJsErrors": true,
"developmentMode": true,
"port1": 1337,
"port2": 1338
}

</details>

<details>
<summary>Your complete test report:</summary>

<!-- Paste your complete result test report here (even if it is huge): -->

e2e_1 | Using locally installed version of TestCafe.

e2e_1 | Running tests in:

e2e_1 | - Firefox 62.0.0 / Linux 0.0.0

e2e_1 |

e2e_1 | Filters

e2e_1 | ERROR The Firefox 62.0.0 / Linux 0.0.0 browser disconnected. This problem may appear when a browser hangs or is closed, or due to network issues.

e2e_1 |

e2e_1 | Type "testcafe -h" for help.
```

Your Environment details:

Run in the Docker container:

• testcafe version: 1.1.0
• node.js version: v10.14.2
• command-line arguments: testcafe (other options are provided by the .testcaferc.json config file)
• browser name and version: Chromium 72.0.3626.109, Firefox 62.0.0
• platform and version: Alpine Linux from the official TestCafe Docker image

[miherlosev]

Hi @grant-mccarriagher

Try to to specify the Docker's --shm-size option.

docker run --shm-size=1g -v /d/tests:/tests -it testcafe/testcafe firefox /tests/test.js

If the problem is still reproduced, please feel free to reopen the issue and send the all necessary information to [email protected].

[grant-mccarriagher]

Thanks for the response @miherlosev . This issue persists even with an shm-size of 2gb in the docker-compose.yml I posted above.

I will email support with the test site used.

[grant-mccarriagher]

Actually, @miherlosev , if anybody from the TestCafe team would like to look at this issue, please post here and I can give you read access to a copy of the GitHub project. Thanks!

EDIT: Also, I cannot reopen this issue as it wasn't closed by me.

[AndreyBelym]

I apologize for the inconvenience, it seems that an original poster can't re-open an issue if it was closed by a repository collaborator.

[grant-mccarriagher]

Appreciate it! Per the email from support, I have added @miherlosev and @AndreyBelym to a copy of my repo at https://github.com/grant-mccarriagher/testcafe-poc

[AndreyBelym]

Thank you for the invite, I will take a look at your repository.

[benjclark]

We have the same issue and the common theme seems to be that everything works on HeadlessChrome 74.0.3723 / Mac OS X 10.11.5 but always ends up hanging indefinetely on HeadlessChrome 74.0.3723 / Linux 0.0.0. At first we thought it was Docker as well, as we also run in a container on the build machine but not locally. But subsequent testing on different machines has led us to see that there is some issue currently associated with Testcafe and Linux. I will try and produce a repo for you to demonstrate our implementation and tests.

[saulokada]

I'm facing the same problem but mine is with Headless Firefox (Firefox 62.0.0 / Linux 0.0.0) in bitbucket pipeline running with docker images.

[AndreyBelym]

@benjclark, thank you for using TestCafe. It will be nice if you share an example. By the way, have you tried the --disable-dev-shm-usage Chrome flag:

testcafe "chrome:headless --disable-dev-shm-usage --no-sandbox" ...

@saulokada, have you tried to run your tests without Docker?

[saulokada]

@AndreyBelym Yes. When I run on my machine without docker it works normally.

[infctr]

Wow, adding --disable-dev-shm-usage actually helped!

[grant-mccarriagher]

I'm getting consistent green builds at this point with chromium:headless --no-sandbox --disable-dev-shm-usage. Which is strange, because even with an shm size of 2gb, my builds were still failing.

[miherlosev]

@infctr @grant-mccarriagher

I happy to hear that. I've created a PR that should protect other customers from the problem you found.