Storybook: Run storyshot in parallel

[cgcote]

According to #4996 and #3802 it seems that running storyshots in parallel has been a topic some months ago. I have run image snapshots in CircleCI with various instance sizes and I get roughly the same execution time so I guess it's not run in parallel at the moment

Do you have a plan to support parallelism and is there any way we could assist?

Thanks!

[SQReder]

Bump for parallelization.
Currently, Loki has some kind of it, as reference for implementation

[ivan3177]

Any update on this? We currently have lot of stories in our project and it takes some time to run them on CI now

[raDiesle]

I guess it is a prerequisite to enable parallel execution of Storyshot Puppeteer, which might end up in another ticket, correct?

[shilman]

PR's very welcome! I also want to recommend Chromatic, which has a free plan since Chromatic 2.0 launched earlier this spring.

Chromatic includes hyper-parallel, cloud-based visual regression testing that I work on with some of the other core maintainers. We've spent a lot of effort making it considerably faster than anything else out there.

[lyleunderwood]

I hesitate to be this guy @shilman, but after seeing this comment I did try chromatic and it is both surprisingly awesome and exactly what my company needed. A little pricey, but hopefully it will replace our storyshots (which is painful), gemini-based screenshot testing of our stories, and now.sh / vercel automated hosting of our storybooks, and as a bonus has this nice UI review process. Very cool. :+1:

[raDiesle]

good @shilman you pointed out Chromatic, which seems to envolved much further the last time I checked.
Nevertheless do you have any hint for current Storyshots setup how to parallelize?
About Chromatic, so to run 200 pages for tests would required ramp up time + time for longest page ~ 20 seconds in total BUT with Puppeteer pixel based screenshots, is it correct?

[shilman]

@raDiesle re: chromatic. yeah, there's a few seconds latency and then we crunch through snapshots/diffs in small batches. there can be some variation based on the framework you're using and the content of your component library, but our internal storybook contains about 1500 stories and the whole process including image snapshots & diffing takes around 50 seconds end to end.

as for parallelizing storyshots, jest itself supports parallelization really well. so i think the thing to do would be to rewrite storyshots to take advantage of jest's structure. if somebody did that it would be a wonderful addition, probably first as a separate package and then maybe as a breaking replacement for storyshots

[felixfbecker]

We use Chromatic Pro, but we also need to run Storyshots just to record test coverage achieved by Chromatic for Codecov. This step has become the longest now in our entire CI pipeline (7min).

I don't see how we could rely on Jests parallization as Jest runs tests in parallel on a file-basis, and storyshots is initialized in a single file (it wouldn't make sense to have multiple files for it). Could you explain a bit more how you imagine this?

[SQReder]

Maybe this would help?
https://jestjs.io/docs/en/api#testconcurrentname-fn-timeout

ср, 23 сент. 2020 г., 15:41 Felix Becker [e-mail]:

We use Chromatic, but we also need to run Storyshots just to record test
coverage achieved by Chromatic for Codecov. This step has become the
longest now in our entire CI pipeline (7min).

I don't see how we could rely on Jests parallization as Jest runs tests in
parallel on a file-basis, and storyshots is initialized in a single file
(it wouldn't make sense to have multiple files for it). Could you explain a
bit more how you imagine this?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/storybookjs/storybook/issues/7863#issuecomment-697338118,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AALTBCL7LQL2KRDS6CBADWTSHHUITANCNFSM4IPNX5NA
.

[felixfbecker]

That could indeed work because it looks like it would make an individual test run in parallel. Just need to make sure that Storyshots opens individual tabs for each test (not change the URL of the same tab)

[shilman]

Working proof of concept: https://twitter.com/tmeasday/status/1316992773493436416?s=20

[vidal7]

WoW! This is amazing! It works only with the CSF format and React but hey, it's just a first stab and this POC proves that storyshots can do better. Thank you.

[raDiesle]

It is also worth to mention a very promising solution: https://github.com/reg-viz/storycap
I vote for replacing puppeteer-shoryshots with this one. I seems to be modular and high effective