Cypress: Visual diffing - ensuring screenshot quality across runs.

[egucciar]

I am using Cypress match screenshots which internally uses blink diff and Cypress own screenshot ability to compare images. We are getting tons of false negatives because cypress screenshot quality varies greatly across environments and resolutions. Despite the images being the same size, the quality is so degraded between runs it is unusable. So how do we ensure Cypress screen quality can be guaranteed to be the same from one run to the next ?

[jennifer-shehane]

Could you explain more about the environments and resolutions you are running Cypress in? Are these local runs on computers? Runs in CI? I'd like to be able to recreate the issue in order to track down the quality issue.

This is something that will definitely need to be tackled as part of our larger Screenshot Diffing proposal #495

[egucciar]

@jennifer-shehane

Sure thing. So basically use blink-diff internally to do the comparison. If the screenshots are taken at different resolutions - say a laptop monitor verses a big desktop monitor - despite the cypress viewport being the same, changing the resolutions in which you are running the tests changes the size and quality of the screenshot so greatly that you get false negatives. Same scenario on run and switching from one machine to another. So essentially it seems like if you haven't taken subsequent screenshots in the same resolution and machine the diff fails, which is bad, because it's impossible to guarantee that tests will be run in the same resolution and same machine without degrading test development experience

I could setup a simple example repo to repro, if youd like but i am strapped for time and wont have the chance till later in the week.

[brian-mann]

@egucciar for us to even begin investigating this we would need many more specifics:

• a reproducible repo
• examples of the different screenshots
• your blink diff + cypress code (as minimal as possible)
• exact specifications of the systems being run that produce different images
  
  
  
  • operating system
  
  
  • screen size
  
  
  • browser (chrome or electron)
  
  
• how you're running cypress... cypress run or cypress open
• whether this is locally or in CI or both
• what your process is for creating / storing / comparing the screenshot masters

[egucciar]

@brian-mann of course i cannot guarantee a comprehensive response with regards to all the screen sizes, since simply resizing the browser window is enough to create a bad diff. So really just any two screens.

For once i can actually repro this easily so ill get the rest.

[brian-mann]

Sure - I should have meant not all of those things are required - they are just items that may or may not play into what is causing this. As long as we can demonstrate and repro the root cause - that's the minimum we need.

[egucciar]

@brian-mann

this is working (by which i mean not working ;) )with just my laptop monitor and resizing it enough. It is easier to repro for me personally when I have my big monitors at work (but wont be able to put much focus on this task during work). I am also new to circle ci so i am not sure how to get this ((quickly)) compatible with a circle run, but i figured it wouldnt be hard to add if needed. I can gather additional specifications tomorrow. Answers to the questions and some sample false diffs all in the readme. Master branch

PS. this uses cypress match screenshot internally. I reviewed their code (3 small files) and that is just taking Cypress screenshots, resizing so they can be compared by Blink-diff and passing along to blink diff. Mind you my version in the company's app was a simple forked version of that just output the shots into a different directory before i knew about the config for screenshot trashing. Very easy to grasp code.

https://github.com/egucciar/cypress-support/tree/visual-diff

EDIT: actually i pushed the cypress-match-screenshot code directly into this repo to make it easier for you to analyze the actual diffing logic. I did look through this and felt it was solid and the screenshots themselves have barely perceptible differences.

[brian-mann]

I glanced through the code and the match snapshot plugin on my phone.

I haven’t yet run it but it’s likely due to us scaling the application when we take screenshots. If that’s the reason then you are in luck because we just finished a few weeks of work which updates screenshots to be taken without scaling and even hides the whole cypress runner area. @chrisbreiding has been working on this for awhile.

I will say this though - what you’re experiencing is not going to really go away completely though. Whenever you’re taking screenshots it needs to be on a homogenous consistent environment. Linux does not render the same way as OSX. Neither does Windows. And when you remove app scaling that means that different resolutions will display different amounts of pixels based on width and heights the same way that it works on different sized laptops monitors or phones. There’s no way to make it all consistent unless you can run in the same resolution on all devices. Even then there are differences at the operating system level.

Have you gotten a screenshot diffing process to work before? Can you articulate details if you were able to get consistent results across different machine OS and resolutions? Thanks!

Our team will look at this issue in depth tomorrow to ensure it’s due to the app scaling I’m suggesting.

On Apr 11, 2018, at 12:38 AM, Erica Gucciardo [e-mail] wrote:

@brian-mann

this is working with just my laptop monitor and resizing it enough. It is easier to repro for me personally when I have my big monitors at work (but wont be able to put much focus on this task during work). I am also new to circle ci so i am not sure how to get this ((quickly)) compatible with a circle run, but i figured it wouldnt be hard to add if needed. I can gather additional specifications tomorrow. Answers to the questions and some sample false diffs all in the readme. Master branch

https://github.com/egucciar/cypress-support

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[egucciar]

@brian-mann

Thanks very much for looking.

Interesting. Then I'd say that the best solution for us would be the following

1. Modify the screen shot code to output into different directories based on argument. Pass in argument based on environment - CI or not.
2. Gitignore the output in a users directory, so user specific screens cannot be checked in
3. Check in the files in the CI after first run. Manually confirm screens are to liking after building twice in CI

[egucciar]

@brian-mann

One last thing, I guess I'm unsure how it others have been doing visual diffing for years. If what I'm imagining isn't possible then I think my new proposed solution would do the trick. I will ask my team if anyone had prior experience with visual diffing

[brian-mann]

Visual diffing is always done against a baseline environment which is static and does not change. There is also a really specific workflow that people do as opposed to machines. Screenshots change every time the app does and just because they don't match doesn't mean they "failed". It takes a process where a human reviews the diff's and chooses new masters.

This is something we're building first class into Cypress which is why we've been doing work on it now.

[edelgado]

@jennifer-shehane I think the link to the epic is incorrect. It points to this same issue.

Question: Is visual regression testing in the roadmap for Cypress? If so, how far out is it (1-6 months, 6+ months, > 1 year)? We are evaluating Cypress and this is something we are interested in. Thanks!

[jennifer-shehane]

Ugh, I do this all the time. I'm sorry 🤦‍♀️

Closing in favor of epic: https://github.com/cypress-io/cypress/issues/495

It is on our roadmap. We are currently experimenting with the implementation details.