React-styleguidist: Automated Visual Testing

[kof]

We spoke about it once, I though it might be good to at least kick off the discussion here.

So the idea is that we can trigger visual snapshot testing which will essentially create graphics for each demo so that next time you run it it can compare and report mismatches.

I think react-styleguidist is a good place to do this because it already has lots of configured components if one uses it havily, so it would not cost much for the user to use this.

List of screenshot tools:

• https://github.com/garris/BackstopJS
• https://github.com/segmentio/niffy

[okonet]

I was thinking about this lately and would be interested in this. @kof let's pair on the implementation?

[kof]

Lets do the research, which tools are the best, who is maintaining them, maybe they can help integrating.

[sapegin]

Looks like you need an API to return all examples for this?

[n1313]

You're looking for something like backstopjs and you want to address examples by using existing data-attributes. They are there precisely for the purposes of integration RSG with external tools like this.

[sapegin]

https://screener.io/ — visual testing for Storybook.

[kof]

Looks nice, but a service …

[sapegin]

But Storybook I’d say :-)

[kof]

More questions:

1. Where would you want to store the images? Repository?
2. How big can images folder grow?
3. How can we snapshot in different browsers? Karma?

[n1313]

I'm not sure if I understand the topic correctly. Are we talking about implementing a new feature into RSG, or about making RSG work well with existing third-party tools?

Sent from my iPhone

On 8 Jun 2017, at 14:17, Oleg Slobodskoi [e-mail] wrote:

More questions:

Where would you want to store the images? Repository?
How big can this grow?
How can we snapshot in different browsers? Karma?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[kof]

Depends on how we want it to work. A third-party tool might be also used as a dependency and integrated into sg as well as sg might provide whatever is required to work with third-party tools nicely.

[sapegin]

@n1313 I think both would work ;-) Do you know any third-party tools that we can use? Most likely it will be a plugin anyway ;-)

[n1313]

RSG already has everything it needs to integrate with external visual regression tools: #238 #334. I don't know what a plugin would be needed for. Integrating an external tool into RSG sounds like a massive feature creep to me.

[sapegin]

RSG already has everything it needs to integrate with external visual regression tools: #238 #334.

Has everything is not integration. I have no idea what to do for example ;-)

Integrating an external tool into RSG sounds like a massive feature creep to me.

Nobody is saying that and it won’t happen. That’s why the plugin API is so important.

[n1313]

Nobody is saying that

I thought @kof was saying precisely that: "I think react-styleguidist is a good place to do this", "used as a dependency and integrated into sg as well". Are you sure you and @kof share the same definition for the word "integration"?

[sapegin]

I think react-styleguidist is a good place to do this

Totally agree with that ;-) It doesn’t imply it should be in the core.

The second quote is out of context, you’ve removed the important “might be”.

We don’t have any concrete plans yet and there’s no point in arguing about the words — it doesn’t add anything to the discussion. If you have any ideas how to implement it in an easy to use way please share them ;-)

[n1313]

The point I'm trying to make is that RSG already has everything that a visual regression testing tool like backstopjs to work. I know that because I was using them together pretty successfully for several months. It's quite simple: you serve your built docs (or a live server url) to the tool, it takes screenshots of component previews and compares them with previous screenshots. That's it. The only thing needed is a way to address the preview containers (we have this) or open examples in isolated pages (we have this). I don't know what else needs to be done here, so when you ask me for ideas on implementation I don't understand what exactly you want to be implemented.

[sapegin]

Could write a blog post or something like that on how to do that? So far it doesn't look simple and user friendly ;-)

[kof]

@n1313 I think the way you described, you would be creating screenshots of the entire document with styleguidist UI elements, which means if styleguidist changes the ui, your tests will all break. Am I wrong?

[n1313]

Yes, if you don't configure your tool to target individual HTML nodes for your screenshots(or if it can't do that) then yes, you will be screenshotting the entire document.

Sent from my iPhone

On 9 Jun 2017, at 17:15, Oleg Slobodskoi [e-mail] wrote:

@n1313 I think you would be creating screenshots of the entire document with styleguidist, which means if styleguidist changes the ui, your tests will all break. Am I wrong?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[kof]

I haven't tried yet, but from what I can see we need to be able to:

1. create snapshots for each component without the sg ui
2. when getting snapshot mismatches, the report should point to the exact component

I think those things is what sg needs to enable if everything else works out of the box. But yet, lets compile the list of tools for the job, find the best once and see how we can support them in a generic way.

I have started the list in the main description, please update it or post your suggestions so someone else can update.

[n1313]

It's not "each component", it's "each test". In simplest case a test is a code block, but it could also be a result of some scripted interaction like a click or typing into a textfield.

Sent from my iPhone

On 9 Jun 2017, at 17:31, Oleg Slobodskoi [e-mail] wrote:

I haven't tried yet, but from what I can see we need to be able to:

create snapshots for each component without the sg ui
when getting snapshots mismatches, the report should point to the exact component
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[kof]

It's not "each component", it's "each test".

Each code block in styleguidist at the end renders one react component. So visually you always end up with one component. But you can call it whatever you want, I think you know what I mean.

result of some scripted interaction like a click

So those visual testing tools provide API's for interactions to be made before a screenshot will be taken?

Yeah I see that backstopjs has "Simulate user interactions with CasperJS scripts."

[n1313]

Usually they provide a way to call PhantomJS APIs directly or provide a reasonable substitution for that.

Sent from my iPhone

On 9 Jun 2017, at 18:01, Oleg Slobodskoi [e-mail] wrote:

It's not "each component", it's "each test".

Each code block in styleguidist at the end renders one react component. So visually you always end up with one component. But you can call it whatever you want, I think you know what I mean.

result of some scripted interaction like a click

So those visual testing tools provide API's for interactions to be made before a screenshot will be taken?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[tzapu]

i may be missing something, so sorry about that, but isn't this what you are looking for?
https://github.com/styleguidist/snapguidist

[sapegin]

@tzapu Snapguidist doesn’t do visual snapshots.

[tzapu]

thanks, was missing something as i was saying :D

[abhimax]

https://github.com/screener-io/screener-storybook
Great tool. but no any free version

[Andarist]

Without knowing about this thread I've setup today simple visual tests, using jest, jest-image-snapshot and puppeteer. There are some missing pieces I'd like to get from the RSG as new features (can offer working on those if we agree they are needed):

• pages with isolated (even more) components - that was actually mentioned in this thread already
• possibility to name component examples with markdown comments or something, so it is possible to tie exact example with the test

How I have done it so far:

import puppeteer from 'puppeteer'
import ExampleComponent from '.'

test('renders fine', async () => {
    const browser = await puppeteer.launch()
    const page = await browser.newPage()
    await page.goto(`http://localhost:6060/#!/${ ExampleComponent.name }/1`)
    const [, handle] = await page.$$('[data-reactroot]')
    const screenshot = await handle.screenshot()
    expect(screenshot).toMatchImageSnapshot()
    await browser.close()
})

[Andarist]

Just if anyone is interested, I've written how we are approaching the subject right now https://developers.livechatinc.com/blog/snapshots-visual-testing/

Also we have published reusable scrapper script for generated documentation - https://github.com/livechat/styleguidist-scrapper . It ain't perfect ofc, no configuration or anything, it's here only to get the job done for us at the moment - but if you find if useful I will be happy to collaborate on the project, so you could benefit from it too.

That being said - I would love to see the idea of visual tests incorporated into the styleguidist's core of course.

[unindented]

@Andarist nice! We had done something very similar (taking screenshots of all examples with puppeteer, and diffing them with pixelmatch) but we didn't integrate it with jest.

The module is here: https://www.npmjs.com/package/react-styleguidist-visual

We're currently using it in this project: https://microsoft.github.io/YamUI/

[sapegin]

@Andarist @unindented That’s awesome! Feel free to send PRs to add these links to the docs ;-)

I think we need a new testing guide too ;-)

[ttiurani]

@Andarist thanks for your examples, they were really useful!

It would be awesome if react-styleguidist could offer an official metadata API, which could be used to get information about all of styleguided components' examples, their urls and selectors to them. Like the ones defined here https://github.com/livechat/styleguidist-scrapper:

type ExampleMetadata = {
  // component name
  component: string,
  // directory where it is located
  dir: string,
  // URL with isolated example
  url: string,
  // title for the example
  // assumed to be paragraph above the example
  title: string,
  // CSS selector for the example
  // useful for targeting screenshots
  selector: string,
}

Something like localhost:6060/metadata ?

[sapegin]

@ttiurani It sounds like a great idea! Should it be a REST or Node API?

[Andarist]

I'd say node API would be more useful - however the title part of my proposed metadata is based on the assumption on having some text (which gets rendered as paragraph) just above the code snippet. Would be great to have some official way to name specific examples.

It would also be easier if an example could get rendered in complete isolation under specific route, I had to target rendered component and create unique selector for it in order to create focused screenshots later (otherwise I would be making screenshots of the component example in styleguidist UI environment)

[ttiurani]

How would the node API work? I'm spinning up the styleguidist with npm-run-all in another thread and running jest in another. From the jest test a REST call is trivial, but how could I call a node API?

And yeah, having a totally isolated URL for the example would be easiest: I made the assumption that the example has a h3 element in front of it, which works for us.

[sapegin]

Node API maybe useful for other tools but I’m just asking ;-) In any case it would be easy to expose if we need it.

[ttiurani]

Yeah, having both a REST endpoint and a node API is of course the best choice. And if there was some official way to add a title/id for each example in the markdown file, that would be even better.

Anyway, this suggested scraping solution is quite brittle, so having an official way to do this would be great.

And thanks for this project btw, it's super useful!

p.s. A slight bug report: from the front page if you click the button that highlights a single component, and then click on the button that higlights a single example, when you un-highlight that example and go back to the component page, the other examples remain hidden in that component and also the front page. Only a refresh brings the other examples back.

[sapegin]

Any help with this API would be super awesome! 🦄

way to add a title/id for each example in the markdown file, that would be even better

Better to open a separate issue with some use cases.

p.s. A slight bug report

I think it will be fixed in 6.1. But please open a new issue if you find another bug.

[jneuendorf]

Hey, I just wanted to throw a LOKI for storybook in ;) Maybe inspecting it may be useful

[stale[bot]]

😴 This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week without any further activity. Thank you for your contributions.