Openfoodnetwork: [Testing] Explore and choose a tool to help cross-browser testing

[lin-d-hop]

Description

Currently the OFN is lacking when it comes to understanding browser support.
There are a number of tools out there that might be able to help us with our cross browser testing:
https://crossbrowsertesting.com
https://www.browserstack.com/

The above are both free for open source projects, which is important.

The best choice for OFN will:

• Be free for open source.
• Have a high level of accuracy/reliability
• test across devices - phone, tablet and desktop
• test all the major browsers
• test a range of browser versions
• simple UI

To successfully complete this issue it would be great to create a google doc with a comparison of free tools - at least the two above. If you have any other recommendations of good tools please do include them too. Link the google doc to the issue in the comments.

Off the back of your comparison research make a recommendation for the best tool for OFN.

[RachL]

I've played a bit with both tools.

They are very much alike, basically providing VM without the pain of having to wait for the VM to launch etc. So it's a bit difficult to separate them. In terms of manual testing my choice goes to Browserstack which UI is simplier, seemed faster (Crest wifi standard) and the VM can work on Firefox, which is not the case of crossbrowsertesting where Chrome is preferred. I guess if the testing team grows to non-tech tester this can be important.

We can have unlimitied access on both as we are an open source project, so I guess we just all need to agree on the functionalities we need. So maybe a dev can have a look as well to see if there is anything interesting in the automated tests area that can differentiate the two?
I would be happy to use crossbrowsertesting if it was a better choice on the automated side.

I've added my 7 days trial account of Crossbrowsertesting in Bitwarden. On browserstack the testing is limited to 30 minutes, so I suggest whoever looks into it just creates a second account for the moment.

We could also choose to use the two for a while and decide after. But we will need to post a link to both tools on our GIthub page.

Google doc with criteria: https://docs.google.com/spreadsheets/d/1L-uY-VFHmMX_T_IJFcCcmgfFs5I_doRNeyZCZATuE5I/edit#gid=0

ping @lin-d-hop @sauloperez @Matt-Yorkley @mkllnk @luisramos0

[lin-d-hop]

It looks like Browserstack and Crossbrowsertesting give us the capability of running all of our Rspec and Capybara tests automatically across all the borwsers and devices we choose...
This would include a small amount of dev... but looks pretty straightforward. Both enable integrated cross-browser/device tests to be integrated into CI using their API.
https://www.browserstack.com/automate/capybara

Crossbrowsertesting also gives the functionality to record and replay tests in the browser without using any code, then replay across all the browsers and devices.
https://crossbrowsertesting.com?wvideo=89izqrca0s
That sounds pretty super nuts awesome!

So my question to the devs is:
How good are our Rspec and Capybara tests? Do we ensure sufficient coverage that we don't need the 'record and replay' feature?

ping again @sauloperez @Matt-Yorkley @mkllnk @luisramos0 @RachL

[luisramos0]

imo, we dont need record and replay. those types of tests are not reliable and end up being even more expensive than auto tests.

smarbear /crossbrowsertesting must be an awesome company, they built swagger 🚀
having said that, I'd go with browserstack.
fwiw you can see that browserstack is much more expensive than crossbowsertesting for the same level, with mobile testing 60usd/month vs 199usd/month:
https://www.browserstack.com/pricing
https://crossbrowsertesting.com/pricing

[luisramos0]

"How good are our Rspec and Capybara tests?"
imo, pretty good. It's best thing about ofn code base.

"Do we ensure sufficient coverage that we don't need the 'record and replay' feature?"
imo, if we are not happy with current coverage we should create more auto tests, not record and replay tests.

[lin-d-hop]

Thanks @luisramos0. Very useful.
Note that both offer all features free to open source projects so the issue is not cost.

[luisramos0]

yes, they are both gifts: the price tag on each gift is different.

[RachL]

As no one screamed at : https://openfoodnetwork.slack.com/archives/CAVTM01QB/p1560369807001500 I'm going forward with browserstack. Closing this