Site-kit-wp: Update VRT threshold and update Storybook VRT images with minor changes

Created on 19 Oct 2020  Â·  5Comments  Â·  Source: google/site-kit-wp

Bug Description

Our VRT tests currently have a difference threshold of 5%, owing to some historical issues with SVGs. This means that some small changes don't actually trigger VRT updates, which is not great.

A 0% threshold can be a bit too aggressive, but in testing we found a 1% threshold was enough to catch a lot of legitimate differences and we should switch to that and update our current VRT references accordingly.

Steps to reproduce

  1. Check out the develop branch without changes
  2. Change https://github.com/google/site-kit-wp/blob/f8b2c6570f0748f7b5a15f40063d1aa1d8252502/tests/backstop/config.js#L54 from 0.05 to 0.01
  3. Run npm run test:visualtest
  4. Observe test failures unrelated to current changes

_Do not alter or remove anything below. The following sections will be managed by moderators only._

Acceptance criteria

  • The VRT margin of error should be reduced to 1%—misMatchThreshold should be 0.01 instead of 0.05.
  • All VRT reference images should reflect the current state of the app and not have slightly out-of-date images that were within the 5% margin of error in our old VRT rules.
  • New changes, even very minor ones in large screens, should trigger VRT failures. (Changing something like the external prop on a Link should trigger a VRT failure, as in the past it would not.)

Implementation Brief

Test Coverage

  • N/A

Visual Regression Changes

  • The VRT tests should update, see above.

QA Brief

  • New changes, even very minor ones in large screens, should trigger VRT failures. (Changing something like the external prop on a Link should trigger a VRT failure, as in the past it would not.)

Changelog entry

P1 Eng Bug

Most helpful comment

@aaemnnosttv @tofumatt @eugene-manuilov I believe we can close this issue now as this has already been taken care of in https://github.com/google/site-kit-wp/pull/2213/files#diff-632eaaf92e395d37506eda70160a8476d57c224d5bd3dcedfd3f6dbf6a52b5fdR56.

All 5 comments

(Larger estimate on this one as it will involve ensuring a lot of VRT test updates are appropriate and might involve creating new issues when something has changed unintentionally.)

IB ✅

As a clarification to the description here, the misMatchThreshold is the percentage to use for the threshold, not a decimal representation of a percentage, so 0.05 is actually half of 1%, not 5%. The default value used by Backstop is .1; see Changing test sensitivity.

@aaemnnosttv @tofumatt @eugene-manuilov I believe we can close this issue now as this has already been taken care of in https://github.com/google/site-kit-wp/pull/2213/files#diff-632eaaf92e395d37506eda70160a8476d57c224d5bd3dcedfd3f6dbf6a52b5fdR56.

@felixarntz yes, we can close it. It has been already addressed.

Was this page helpful?
0 / 5 - 0 ratings