Storybook: Storyshot fails to respect threshold settings

[patthiel]

Bug summary

I've setup storyshots in this PR for use in the Linode Manager project. Running the tests locally passes just fine, but in Travis one of the tests consistently fails the comparison (ranging between a 4 - 6% difference).. I tried compensating for this by bumping the failure threshold to 10 percent, then to 20 percent. Nothing seems to be doing the trick. The travis results can be found here: https://api.travis-ci.org/v3/job/396935367/log.txt

Expected image to match or be a close match to snapshot
but was 4.000048562548562% different from snapshot (82369 differing pixels).

See diff for details:
/home/travis/build/linode/manager/src/components/__image_snapshots__/__diff_output__/storyshots-test-tsx-image-storyshots-selection-row-interactive-example-1-diff.png

Is there something wrong with my configuration? https://github.com/patthiel/manager/blob/1c4a9195d62fc9c489ca7a060d4b275e53cc4f24/src/components/Storyshots.test.tsx

import initStoryshots, { imageSnapshot } from '@storybook/addon-storyshots';

const getMatchOptions = ({context : {kind, story}, url}) => ({
    // Allow 20% difference
    failureThreshold: 0.2,
    failureThresholdType: 'percent',
    customDiffConfig:  { threshold: 0.15 }
});

jest.setTimeout(30000);

initStoryshots({
    suite: 'Image storyshots',
    test: imageSnapshot({storybookUrl: 'http://localhost:6006', getMatchOptions})
});

Steps to reproduce

• See linked PR and Travis build.

Please specify which version of Storybook and optionally any affected addons that you're running:

storybook/[email protected]
storybook/[email protected]

[patthiel]

@igor-dv @shilman any ideas here? We'd really like to see this integrated into the Linode Manager test suite.

[igor-dv]

Maybe @thomasbertet can suggest something

[igor-dv]

BTW, did you try to use beforeScreenshot like here ?

[patthiel]

@igor-dv I just tried adding that, didn't seem to help.. here's the config

And the build failure here - Same issue.

[igor-dv]

Yeah, it's not so clear. Did you see those problematic generated images somehow? Like what is the difference?

[patthiel]

@igor-dv here's the diff image:

Seems like manually settting the viewport for the tests could address the issue, although, i'm not sure where in the config i can do that?

[igor-dv]

I see the place where it starts breaking

In the latest alpha, you have the ability to define viewports for stories, I am not sure this is the problem though.

Does your ci run under the same system preferences as your local env? Most of the problems are related to the font differences between a local env and the CI machine.

[patthiel]

@igor-dv ah, ok. Interesting..

So, I managed to get it working by updating and running everything via docker-compose. This will ensure system preferences are always the same. (That process was not as straightforward as i had hoped, but i found the puppeteer workaround helped resolve the issues i ran into.)

Unfortunately, it doesn't seem like we solved the underlying issue initially reported regarding threshold settings not being respected, but i am at least happy to have integrated storyshot into our project. When I have time, I will attempt to dig more into this issue.

For those that may find this thread, here is my PR using docker-compose to run storybook/storyshots. a few improvements will be made in the next pass.

[stale[bot]]

Hi everyone! Seems like there hasn't been much going on in this issue lately. If there are still questions, comments, or bugs, please feel free to continue the discussion. Unfortunately, we don't have time to get to every issue. We are always open to contributions so please send us a pull request if you would like to help. Inactive issues will be closed after 30 days. Thanks!