Shoelace: Add unit and e2e tests

[lamplightdev]

It would be good to have some tests to check for breaking changes during development - the Stencil built-in test runners seem the obvious tool. I'm no testing expert but I'd be interested in writing some tests to get this going, any pointers on what kind of tests would be the most beneficial to start with?

[claviska]

Good call. I was thinking before 1.0 stable comes out, some basic tests can be put in place. I haven't worked with Stencil's e2e tests yet, but I agree it's a good place to start.

I've also been thinking of visual regression testing, possibly with BackstopJS. I've set this up in another project and it's pretty easy to work with. Might have to tweak some things to make it work well with GitHub, but it will let us know when things break visually across all components (this is especially useful for composed views).

If you want to start experimenting with Stencil's testing that would be awesome. I'm not sure what it's capable of so use your best judgement. Perhaps ensuring basic behaviors is a good place to start (e.g. things opening and closing when the correct triggers or methods are called, custom events firing with the correct data, etc.).

[ingomc]

We're using Stencil Components on work, we build a similary library and added a lot of e2e tests... now 1 year later i think it is more important to habe visual regression tests, because the more components you have... the more components will be nested in each other... and at the end, you total lose the overview if you change somethin in a component, which is used in many other components... you will never get the edge cases. With Visual Regression Tests this is no problem at all, there will be imidiately an error if something looks like its not supposed to be.
Events not firing, error in methods will be obvious, causing a Problem in pretty much every nested component.

So in my experience a would definitly start with visual regression tests, which is far more difficult to implement because its not really supported right now... adding e2e and spectest at the end will be super fast and easy once VRT is working

[lamplightdev]

Definitely agree we should have visual regression tests, but I think e2e tests are worth having too, to catch some possible edge case bugs that don't make themselves obvious. I don't think they need to wait until the VRT are finished, they shouldn't take long, and no reason we can't work on both at the same time.

Do you have any pointers to how we can handle the visual tests? I had a brief look at the Stencil support and it seems very early stages. Maybe BackstopJS as @claviska mentions above?

[ingomc]

we dont have a technical solution now, backstop ist definatly on our list. We will test sending out screenshots to applitools, so we dont have to deal with images other then sending it to the service and handle it in the web environment.

You dont have to wait until VRT is finished, but some things are at some point irrelevant as soon as VRT is working. So the start should be really barebone, perharps custom events firing, triggering and spectest on functions. So ya i agree

[leewhitbeck]

Has anyone looked at https://storybook.js.org/ ? I haven't used it, but from what I see in the docs it can help with visual regression among other things. Also, i've been looking at Ionic's approach too, which seems pretty nice.

[claviska]

I previously used Storybook for a similar web component library. It's slow, bloated, difficult to customize, and I have absolutely zero interest in using it to develop, test, or document custom elements ever again.

It also doesn't help with visual regression unless you're using it along with their own product. Similar results can be found with BackstopJS which doesn't have the Storybook dependency.

[claviska]

I'm tracking this in #296 so please follow that issue for testing progress.