Storybook: StoryShots/Storyshots-Puppeteer is inconsistent and buggy with docker

[EdenTurgeman]

Describe the bug
We are trying to use Storyshots with puppeteer to take snapshots of our storybook and integrate it to out ci Jenkins pipeline, but are running into many problems when using it with docker, specifically these issues :
Screenshots are inconsistent, primarily each time we run the tests with the exact same project and storybook hosted locally, we keep getting these errors related to mismatched resolution and diffs show all pages are slightly misaligned to the right.

We have 107 stories and somehow this is what happens when we run in docker

107 Fails + 107 Passes ?

To Reproduce
My base chromium docker image, the rest is copying the project and running yarn test

FROM node:11.7.0-alpine
ENV CHROME_BIN="/usr/bin/chromium-browser"
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD="true"

RUN apk update && apk upgrade && \
    echo @edge http://nl.alpinelinux.org/alpine/edge/community >> /etc/apk/repositories && \
    echo @edge http://nl.alpinelinux.org/alpine/edge/main >> /etc/apk/repositories && \
    apk add --no-cache \
      chromium@edge=~73.0.3683.103 \
      nss@edge \
      freetype@edge \
      freetype-dev@edge \
      harfbuzz@edge \
      ttf-freefont@edge

RUN addgroup -S pptruser && adduser -S -g pptruser pptruser \
    && mkdir -p /home/pptruser/Downloads /app \
    && chown -R pptruser:pptruser /home/pptruser \
    && chown -R pptruser:pptruser /app

The docker image based on it.
Copies the storybook and runs the tests.

FROM node:11.7.0-puppeteer

WORKDIR /src
COPY --from= /src       .
COPY .                      storybook
RUN yarn install:ci:dev
CMD cd storybook && yarn test

imageSnapshots.js

import initStoryshots from '@storybook/addon-storyshots';
import { imageSnapshot } from '@storybook/addon-storyshots-puppeteer';

require('babel-plugin-require-context-hook/register')();

const waitBeforeSnapshotTime = 400;

// Hide all elements with classes containing these phrases
const ignoredElementsSelectors = ['MuiCircularProgress', 'MuiLinearProgress'];

const disableAnimationElements = ignoredElementsSelectors => {
  const animationSelectors = ignoredElementsSelectors.map(animationClass => `[class*='${animationClass}']`);
  const animatedPageElements = document.querySelectorAll(animationSelectors);

  animatedPageElements.forEach(pageElement => {
    pageElement.style.visibility = 'hidden';
  });
};

const storybookUrl = 'http://app-storybook:7071/';

const storiesToTestRegex = '^((?!.*?(Sketch|IFrame)).)*$';

const beforeScreenshot = page =>
  new Promise(resolve =>
    page
      .evaluate(disableAnimationElements, ignoredElementsSelectors)
      .then(() => page.setViewport({ width: 1920, height: 1080 }))
      .then(() => setTimeout(resolve, waitBeforeSnapshotTime)),
  );

const customizePage = page => page.setViewport({ width: 1920, height: 1080 });


initStoryshots({
  framework: 'react',
  suite: 'Image Storyshots',
  storyKindRegex: storiesToTestRegex,
  test: imageSnapshot({
    storybookUrl,
    beforeScreenshot,
    customizePage,
    chromeExecutablePath: process.env.CHROME_BIN,
  }),
});

Expected behavior
The screenshots should consistent with the local ones especially since I only install puppeteer with addon-storyshots-puppeteer so there is no version mismatch.

The documentation does not make it clear that there might be differences between os's and that results might be inconsistent.

Currently, for me, it is broken and I am trying super hard to make it work and to understand most the process required for it to work.

Thank you for reading my issue, I'm really hoping for a response and hope this will be helpful for others. :)

[stale[bot]]

Hi everyone! Seems like there hasn't been much going on in this issue lately. If there are still questions, comments, or bugs, please feel free to continue the discussion. Unfortunately, we don't have time to get to every issue. We are always open to contributions so please send us a pull request if you would like to help. Inactive issues will be closed after 30 days. Thanks!

[EdenTurgeman]

This issue is still relevant. Would love to get a reply and looking into my case.

[B3Kay]

I also got this problem. It renders different snapshots between computers.

[stale[bot]]

Hi everyone! Seems like there hasn't been much going on in this issue lately. If there are still questions, comments, or bugs, please feel free to continue the discussion. Unfortunately, we don't have time to get to every issue. We are always open to contributions so please send us a pull request if you would like to help. Inactive issues will be closed after 30 days. Thanks!

[meeech]

anyone find a solution yet? i've started down the path of dockerizing the snapshot creation (with goal of images being consistent between creation and pipeline testing), but this ticket makes me wonder if I'm going in the wrong direction?

[EdenTurgeman]

@meeech Hey we've done the same here for a little while.. doesn't seem like there is a better option for now, would love to know why this is happening.

[stale[bot]]

Hi everyone! Seems like there hasn't been much going on in this issue lately. If there are still questions, comments, or bugs, please feel free to continue the discussion. Unfortunately, we don't have time to get to every issue. We are always open to contributions so please send us a pull request if you would like to help. Inactive issues will be closed after 30 days. Thanks!

[stale[bot]]

Hey there, it's me again! I am going close this issue to help our maintainers focus on the current development roadmap instead. If the issue mentioned is still a concern, please open a new ticket and mention this old one. Cheers and thanks for using Storybook!

[bohdanbirdie]

I wonder if anyone got this working properly

[petermikitsh]

It is possible to get consistent screenshots across environments. Here's how:

I ran into this issue and followed the strategy outlined in storybook-addon-image-snapshots (https://github.com/spring-media/storybook-addon-image-snapshots#testing-with-a-local-running-storybook-server). Two general principles I picked up:

• Generate your baseline snapshots and test snapshots using the same docker image. I've had success with browserless/chrome. You can mount your built storybook like so using -v.
• Always create snapshots against a built storybook (not when running start-storybook).

docker run -d --rm \
  -p 9222:3000 \
  -e "CONNECTION_TIMEOUT=600000" \
  -v $STORYBOOK_DIR/dist:/opt/storybook-static \
  browserless/chrome

Then, setup your tests with a custom puppeteer browser. The puppeteer driver will use Chrome from the docker container (above) exposed on port 9222.

Test Setup: storyshots.test.js

```js
import initStoryshots from '@storybook/addon-storyshots';
import { imageSnapshot } from '@storybook/addon-storyshots-puppeteer';
import puppeteer from 'puppeteer';

const HOST_IP = process.env.HOST_IP;
let browser;

const testFn = imageSnapshot({
storybookUrl: 'file:///opt/storybook-static',
getCustomBrowser: async () => {
browser = await puppeteer.connect({
browserURL: (() => {
// For CI env's: If running in docker inside docker, you'll need the IP address of the host
if (HOST_IP) {
return http://${HOST_IP}:9222;
}
return 'http://localhost:9222';
})(),
});
return browser;
},
beforeScreenshot: async (page, { context: { kind, story }, url }) => {
await page.evaluateHandle('document.fonts.ready');
await page.setViewport({ deviceScaleFactor: 2, width: 800, height: 600 });
},
getScreenshotOptions: ({ context, url }) => {
return {
encoding: 'base64',
fullPage: false,
};
},
});

// Override 'afterAll' so jest doesn't hang
testFn.afterAll = () => {
if (browser) {
browser.close();
}
};

initStoryshots({
test: testFn,
});
```

You'll probably want 2 separate scripts: one for updating snapshots, the other for testing.

snapshots.sh

```sh
DOCKER_ID=$(docker run -d --rm \
-p 9222:3000 \
-e "CONNECTION_TIMEOUT=600000" \
-v $STORYBOOK_DIR/dist:/opt/storybook-static \
browserless/chrome)

npx --no-install jest --useStderr --update-snapshot

docker kill $DOCKER_ID
```

test.sh

```sh
DOCKER_ID=$(docker run -d --rm \
-p 9222:3000 \
-e "CONNECTION_TIMEOUT=600000" \
-v $STORYBOOK_DIR/dist:/opt/storybook-static \
browserless/chrome)

if [[ "$CI" ]]; then
DOCKER_HOST_IP=$(/sbin/ip route|awk '/default/ { print $3 }')
fi

HOST_IP=$DOCKER_HOST_IP npx --no-install jest --useStderr
docker kill $DOCKER_ID
```