We test basic installing and options in our vagrant tests on some systems but so far not on windows. We need that though because every now and then we break elasticserch for windows and then only find out via bug reports (see for example https://github.com/elastic/elasticsearch/pull/18473, https://github.com/elastic/elasticsearch/issues/16086). We need something similar to the bats tests for windows. It might not be possible to provide a public windows image but at least there needs to be a way to test basic functionality for people that have access to one.
cc @elastic/microsoft it'd be good if we take ownership of this one. We need to write similar vagrant tests for our msi in the coming weeks too.
I did some Cucumber tests running against Vagrant boxes with the vagrant-cucumber plugin https://github.com/scalefactory/vagrant-cucumber
With linked clone support in Vagrant 1.8 it should be really fast to reset the VM between each test to have a clean base environment to install MSI package etc. again and again.
Here is a sample for a Linux box: https://github.com/StefanScherer/cups-cucumber-test/blob/master/features/lpr.feature
And this works for Windows VM's as well. I've some (sorry closed source) Windows environments running with VirtualBox Vagrant Boxes.
Just my two cent.
Superseded by #18475
Most helpful comment
cc @elastic/microsoft it'd be good if we take ownership of this one. We need to write similar vagrant tests for our msi in the coming weeks too.