Behat: Allow multiple profiles to combine settings

[diamondsea]

TL;DR: Support merging multiple profiles at runtime (in the order specified) to allow more flexibility in a single configuration file

I have a complex Drupal Multisite (multiple sites and domain names from one repo) environment and need to run test with various combinations of options (url, browser, selenium source, etc) in my local development environment as well as on our servers. Using a single static behat.yml file would require much redundancy, and creating separate bahat-VARIANT.yml files would also result in much redundancy.

What would give me the flexibility I need for running the various combinations of settings that are not currently possible with the single-profile-only behat.yml setup we have now.

This would allow running tests in this type of format:

• Run tests on the MySite production site, using saucelabs for selenium:

bin/behat -p mysite -p saucelabs

• Run tests on the OtherSite production site, using saucelabs for selenium::

bin/behat -p mysite_local -p saucelabs

• Run tests on OtherSite, using saucelabs for selenium:

bin/behat -p othersite -p saucelabs

• Run tests on the local copy of OtherSite, using local selenium

bin/behat -p othersite -p selenium

• Run only IE8 tests on local copy of OtherSite (will automatically send to saucelabs)

bin/behat -p othersite_local -p ie8 --tags="@ie8"

• or with compressed syntax:

bin/behat -p mysite,saucelabs,quicktests

bin/behat -p "othersite, saucelabs, quicktests

For these examples, the above commands could be used with a behat.yml like this:

# default Behat configuration profile (inherited by other profiles)
default:
  suites:
    default:
      contexts:
        - FeatureContext
        - Drupal\DrupalExtension\Context\DrupalContext
        - Drupal\DrupalExtension\Context\MinkContext
        - Drupal\DrupalExtension\Context\MessageContext
        - Drupal\DrupalExtension\Context\DrushContext
  extensions:
    Behat\MinkExtension:
      base_url: http://www.mysite.com/
      goutte: ~
      selenium2: ~
    Drupal\DrupalExtension:
      blackbox: ~

# add "-p saucelabs" to run selenium tests on saucelabs
saucelabs:
  extensions:
    Behat\MinkExtension:
      selenium2:
        wd_host: USERNAME:[email protected]/wd/hub

# add "-p selenium" to run selenium tests locally
selenium:
  extensions:
    Behat\MinkExtension:
      selenium2:
        wd_host: localhost:4444/wd/hub

# add "-p mysite" to run tests against the production web server
mysite:
  extensions:
    Behat\MinkExtension:
      base_url: http://www.mysite.com/

# add "-p mysite_local" to run tests against the local web server
mysite_local:
  extensions:
    Behat\MinkExtension:
      base_url: http://mysite.com.local/

# add "-p othersite" to run tests against the production web server
othersite:
  extensions:
    Behat\MinkExtension:
      base_url: http://www.othersite.com/

# add "-p othersite_local" to run tests against the local web server
othersite_local:
  extensions:
    Behat\MinkExtension:
      base_url: http://othersite.com.local/

# add "-p ie8" to run test on IE8
# This will automatically switch the test to Saucelabs for the IE testing
ie8:
  extensions:
    Behat\MinkExtension:
      selenium2:
        wd_host: USERNAME:[email protected]/wd/hub
        browser: iexplore
        capabilities:
          name: ie8 test suite
          platform: Win7
          browser: ie
          version: 8

# add "-p ie9" to run test on IE9
# This will automatically switch the test to Saucelabs for the IE testing
ie9:
  extensions:
    Behat\MinkExtension:
      selenium2:
        wd_host: USERNAME:[email protected]/wd/hub
        browser: iexplore
        capabilities:
          name: ie9 test suite
          platform: Win7
          browser: ie
          version: 9

The only way that I've been able to create similar functionality is by removing all the needed settings from the behat.yml (due to the issue described in https://github.com/Behat/Behat/pull/725) and then setting the desired options in the BEHAT_PARAMS variable, which has to be in JSON and is difficult to manage easily (esp from a shell script).

[Sorbing]

I also hope for the implementation of support multiple profiles.. We have already discussed such a possibility - https://github.com/Behat/Behat/issues/629

[TheilPetersen]

I have the exact same issue and would like to see the implementation of a profile array possibility. The format bin/behat -p mysite,saucelabs,quicktests seems quite natural to me.

[JordiGiros]

I agree with all of you guys. Same problem here...

[everzet]

Use suites and profiles inheritance instead. Every profile inherits entire configuration from default, including suite. That means if you provide your entire multisite configuration in form of different suites under default, you then will be able to add custom profiles that will enable or disable particular suites on premise. Example: https://github.com/Behat/Borg/blob/master/behat.yml

[Trogie]

I don't get it...

Take:

• 2 different base_urls A and B
• 3 browsers: X, Y and Z

How do I build up the config so I can run my tests on base_url A and on browser X?

I have 6 different profiles: 2 x 3 and this number will increase fast with more browsers/devices/...

[Trogie]

Damned again fighting this problem!