Storybook: Storyshots, same visual regression test fails

[florianbepunkt]

Describe the bug
I have a dockerfile based on node:10-stretch image. running my tests locally on mac or with the docker image works fine. if I use the same docker image in jenkins and run the tests, they fail due to different image size:

Expected image to be the same size as the snapshot (2064x1568), but was different (2048x1536).

How can I enforce the same image size across environments?

To Reproduce
This is my visual regression test setup

const devices = require("puppeteer/DeviceDescriptors");
const tablet = devices["iPad landscape"];
const customizePage = page => {
  return page
    .emulate(tablet)
    .then(resp => resp)
    .catch(ex => console.error(ex));
};

const beforeScreenshot = (page, { context: { kind, story }, url }) => {
  return page
    .emulate(tablet)
    .then(resp => resp)
    .catch(ex => console.error(ex));
};

const getMatchOptions = () => ({
  failureThreshold: 0.2,
  failureThresholdType: "percent"
});

initStoryshots({
  config: ({ configure }) => configure(loadStoriesForVRT, module),
  suite: "Visual Regression Testing",
  test: imageSnapshot({
    storybookUrl: `file://${__dirname}/__staticStorybook`,
    customizePage,
    getMatchOptions,
    beforeScreenshot
  })
});

Expected behavior
Tests should pass. Snapshots should render same size.

[florianbepunkt]

Made it pass…

import { imageSnapshot } from "@storybook/addon-storyshots-puppeteer";
import initStoryshots from "@storybook/addon-storyshots";

const req = require.context("../src/components", true, /stories.js$/);

// visual regression testing
const devices = require("puppeteer/DeviceDescriptors");
const tablet = devices["iPad landscape"];
const customizePage = page => page.emulate(tablet);
const beforeScreenshot = page => page.emulate(tablet);
const getMatchOptions = () => ({
  failureThreshold: 0.2,
  failureThresholdType: "percent"
});

const loadStoriesForVRT = () =>
  req
    .keys()
    .forEach(filename => req(filename));

initStoryshots({
  config: ({ configure }) => configure(loadStoriesForVRT, module),
  suite: "Visual Regression Testing",
  test: imageSnapshot({
    storybookUrl: `file://${__dirname}/__staticStorybook`,
    customizePage,
    getMatchOptions,
    beforeScreenshot
  })
});

Pupeteers snapshot method behaves inconsistently across os systems, which might be good to mention in the storyshots docs as it introduces a lot of hedache when tests that run perfectly fine suddenly crash in CI.

[paschalidi]

thanks for this one . I can confirm that this will solve the issue

[pmowrer]

@florianbepunkt What did you do to work around this problem? Can't tell what the significant difference is between the two snippets you pasted 😅

[florianbepunkt]

@pmowrer: I think the code in my initial post is wrong. I had to use both parameters below in order to correctly set the screensize.

const customizePage = page => page.emulate(tablet);
const beforeScreenshot = page => page.emulate(tablet);

[segalr3]

@florianbepunkt i tried your solution and still getting different image sizes...anything else you recommend checking?

[florianbepunkt]

Can you post a repo?