Cypress: Headless runner screenshots use the last width of interactive test runner

[dallonf]

Mentioned and diagnosed in https://github.com/cypress-io/cypress/issues/181#issuecomment-365407977.

• Operating System: macOS 10.12.6
• Cypress Version: 1.4.2
• Browser Version: Google Chrome 64.0.3282.140

Is this a Feature or Bug?

Unintuitive feature

Current behavior:

The headless runner will have a test runner panel just as wide as the panel from the last time you used the interactive runner. This can result in a hilariously small view for the tested page itself:

Desired behavior:

I would say it should use a static but configurable width in headless mode (separate from its width in interactive mode). Another solution could be to cache the _percentage_ of the window's width instead of the actual pixels, so that it's always proportional.

How to reproduce:

1. Open an interactive test runner
2. Size your window to take up your entire screen
3. Size the test runner sidebar to be comfortably large
4. Run the tests in headless mode (with viewportWidth set to the default of 1000)
5. Check the outputted video or screenshots

[braden-m]

Is there the possibility of a workaround for this? Clearing the cache which stores this 'tab width' value seems like it would work, but I've had no luck finding where this value would be stored (not in localstorage, to be sure).

[brian-mann]

It depends on your operating system. We store it where the operating system is supposed to store application settings.

Open up cypress and goto File -> View App Data

Then goto production/projects/{your_project}/state.json

If you use Cypress in global mode then it would be: production/state.json

[braden-m]

Thanks, that's worked great! That was the one hurdle we had remaining with Cypress to setting up visual diffing in our workflow.

[brian-mann]

@braden-m do you mind explaining what your visual diffing workflow is?

We're looking at implementing this in Cypress now and it would be helpful to understand your process. What are you diffing? Where are you storing the masters? In source control? Where do you perform the visual diff checks? CI? What's the process for approving new ones?

[braden-m]

We take screenshots through headless Cypress during our usual E2E tests. Masters are stored in source control, though the process for updating/approving them is manual (i.e. copying the updated screenshots from screenshots into screenshots-base. A process of accepting/rejecting new screenshots on failure locally could be our next step).

After Cypress has been run, visual diffing is done using blink-diff as a separate script on CI, with the output saved to screenshots-diff (saved as a Buildkite artifact).

We use blink-diff over, say, pixelmatch, for the ease of cropping images (since the test runner tab and URL will not be constant between our tests, we remove it).

[brian-mann]

Right. But when the screenshots fail - what do you do?

You go inspect the buildkite artifacts and then manually reconcile? You have to go see if its a legitimate failure?

And what if its a natural change? You copy the buildkite artifact and then update your master in source? And then rerun all the tests?

[brian-mann]

Since you're using a single environment as the source of your screenshots (linux / chrome) what do you do about taking screenshots locally?

What I mean is - if you know something is changing, there's no way to take the screenshot locally and update the masters because they won't match what CI outputs... in that case you just wait for them fail in order to generate them correctly and then update the masters in source?

[braden-m]

Though we're still in early stages of integrating with CI, it appears that running Cypress headlessly, both locally and on CI, produces the same screenshots (provided reporterWidth is cleared, since this could be changed locally).

Because of this, we can produce and update screenshots locally.

[brian-mann]

Are you taking screenshots locally in the same environment as CI?

What I mean is - there is no way linux (which has disabled GPU) renders pixels the same way that OSX does (or Windows for that matter).

Your environments would all have to be perfectly homogenous. Is this the case? Or are they different?

[brian-mann]

Or perhaps you have the blink-diff threshold set relatively high to account for those differences

[braden-m]

We use the Cypress Docker image on CI (so, whichever environment is specified there), and locally running OSX.

You're right though, at current it seems we need to set a higher threshold to account for differences in rendering (mostly rendering of text). Going forward, we plan to run the screenshotting locally using a Docker container.

[brian-mann]

Related to #2132 and #1445

[rodoabad]

Any ETA on when we can have a fix for this?