Behat: 3.4.0 breaks the ability to run individual tests

[pfrenssen]

Update + quick fix

This is a long standing issue which is affecting many users. It has been solved but the PR is not yet accepted. If you are affected by this you can solve it by adding the following lines to your composer.json:

"require-dev": {
    "behat/gherkin": "^4.6.0",
    "cweagans/composer-patches": "^1.6.7",
},
"patches": {
    "behat/gherkin": {
        "Correctly filter features, now that the base path is correctly set": "https://github.com/Behat/Gherkin/compare/v4.6.0...a7d84eb6ba245ea268a304312d14660788bf6c29.patch"
    }
}

Note that this problem is not caused directly by Behat but it is due to a confusion between absolute and relative paths in the Behat/Gherkin library. We are keeping this issue open in the Behat queue so that it is easier to find for people who encounter this problem when using Behat.

If possible please vote for + review + approve the fix at https://github.com/Behat/Gherkin/pull/129 - hopefully once it is approved it can be merged.

Original summary

This was originally reported at https://github.com/Behat/Gherkin/issues/128 but it appears to be a bug in Behat and not in Gherkin.

In https://github.com/Behat/Behat/pull/1056 a change was introduced to make it possible to run individual tests from inside a different folder than where the tests are located. Unfortunately this breaks the more common use case of running the tests from inside the actual folder where they are located.

To replicate this, create a folder structure in which the tests are located in a subfolder of the root of the project, e.g in the tests/ folder:

$ composer require behat/behat-3.4.0
$ mkdir tests && cd tests/
$ ../vendor/bin/behat --init
# Create a test
$ vim features/homepage.feature
# Make sure the gherkin cache is cleared, this might "hide" the problem
$ rm -rf /tmp/behat_*
# Try to run a single test
$ ../vendor/bin/behat features/homepage.feature -vvv

Result:

$ ../vendor/bin/behat features/homepage.feature -vvv


No scenarios
No steps
0m0.73s (9.22Mb)


  [Behat\Testwork\Tester\Exception\WrongPathsException]                                                                                     
  No specifications found at path(s) `features/homepage.feature`. This might be because of incorrect paths configuration in your `suites`.  


Exception trace:
 () at /home/pieter/v/joinup-dev/vendor/behat/behat/src/Behat/Testwork/Tester/Cli/ExerciseController.php:111
 Behat\Testwork\Tester\Cli\ExerciseController->execute() at /home/pieter/v/joinup-dev/vendor/behat/behat/src/Behat/Testwork/Cli/Command.php:63
 Behat\Testwork\Cli\Command->execute() at /home/pieter/v/joinup-dev/vendor/symfony/console/Command/Command.php:264
 Symfony\Component\Console\Command\Command->run() at /home/pieter/v/joinup-dev/vendor/symfony/console/Application.php:841
 Symfony\Component\Console\Application->doRunCommand() at /home/pieter/v/joinup-dev/vendor/symfony/console/Application.php:206
 Symfony\Component\Console\Application->doRun() at /home/pieter/v/joinup-dev/vendor/behat/behat/src/Behat/Testwork/Cli/Application.php:124
 Behat\Testwork\Cli\Application->doRun() at /home/pieter/v/joinup-dev/vendor/symfony/console/Application.php:125
 Symfony\Component\Console\Application->run() at /home/pieter/v/joinup-dev/vendor/behat/behat/bin/behat:34

Using git bisect I have pinpointed the introduction of this bug to https://github.com/Behat/Behat/commit/08802b7f2d736b88df339e5411a1c722e92adbe3

[QassaDevelopers]

We too have the exact same problem. The problem lies indeed with the referenced commit above.
Because of this commit, this code fails to work properly:

// Behat\Gherkin\Filter\PathsFilter.php:50

public function isFeatureMatch(FeatureNode $feature)
{
    foreach ($this->filterPaths as $path) {
        if (0 === strpos(realpath($feature->getFile()), $path)) {
            return true;
        }
    }

    return false;
}

The $feature->getFile() returns a relative path, because the base path is now set. Therefor, the realpath() returns false.

Because the feature does not match the PathsFilter, it is not added to the features array in the Gherkin::load() method.

[pfrenssen]

I've been debugging this and the problem seems to reside in behat/gherkin after all:

When the basepath is passed in the new version the PathsFilter::isFeatureMatch() no longer works as expected. There is confusion whether the "path" of a feature is a relative path or an absolute path.

PathsFilter::isFeatureMatch() thinks that $feature->getFile() returns an absolute path, and will always return FALSE if it actually contains a relative path such as feature/homepage.feature.

From the documentation of FeatureNode::getFile() it is not explicitly defined whether this is a relative or an absolute path, but it "feels" that it should be a relative path, otherwise this method would be named FeatureNode::getPath().

[pfrenssen]

I have a potential fix in https://github.com/Behat/Gherkin/pull/129 but I am not sure if this is the right solution, this is the first time I look at the Gherkin code base.

[everzet]

Revert to 3.3 for now, I'll try to look into the bug ASAP.

[tanc]

I'm seeing this issue even though I've required behat/behat at 3.3.1:

  - Installing behat/mink (v1.7.1): Loading from cache
  - Installing behat/mink-browserkit-driver (v1.3.2): Loading from cache
  - Installing behat/mink-goutte-driver (v1.2.1): Loading from cache
  - Installing behat/transliterator (v1.2.0): Loading from cache
  - Installing behat/gherkin (v4.5.1): Loading from cache
  - Installing behat/behat (v3.3.1): Loading from cache
    Skipped installation of bin bin/behat for package behat/behat: name conflicts with an existing file
  - Installing behat/mink-extension (v2.2): Loading from cache
  - Installing behat/mink-selenium2-driver (v1.3.1): Loading from cache
...

bin/behat features/ac_base.feature -vvv

  [Behat\Testwork\Tester\Exception\WrongPathsException]
  No specifications found at path(s) `features/ac_base.feature`. This might be because of incorrect paths configuration in your `suites`.

Exception trace:
 () at /Users/tanc/Sites/l.install_profile_test/gitroot/vendor/behat/behat/src/Behat/Testwork/Tester/Cli/ExerciseController.php:111
 Behat\Testwork\Tester\Cli\ExerciseController->execute() at /Users/tanc/Sites/l.install_profile_test/gitroot/vendor/behat/behat/src/Behat/Testwork/Cli/Command.php:63
 Behat\Testwork\Cli\Command->execute() at /Users/tanc/Sites/l.install_profile_test/gitroot/vendor/symfony/console/Command/Command.php:264
 Symfony\Component\Console\Command\Command->run() at /Users/tanc/Sites/l.install_profile_test/gitroot/vendor/symfony/console/Application.php:841
 Symfony\Component\Console\Application->doRunCommand() at /Users/tanc/Sites/l.install_profile_test/gitroot/vendor/symfony/console/Application.php:206
 Symfony\Component\Console\Application->doRun() at /Users/tanc/Sites/l.install_profile_test/gitroot/vendor/behat/behat/src/Behat/Testwork/Cli/Application.php:124
 Behat\Testwork\Cli\Application->doRun() at /Users/tanc/Sites/l.install_profile_test/gitroot/vendor/symfony/console/Application.php:125
 Symfony\Component\Console\Application->run() at /Users/tanc/Sites/l.install_profile_test/gitroot/vendor/behat/behat/bin/behat:34

Any idea what I'm doing wrong?

[mvoelker]

Hey @tanc
Check this solution: https://github.com/Behat/Behat/issues/1085#issuecomment-335165304
"... pin behat/gherkin to version 4.4.5 ..."
This worked for me and solved the Problem (in PhpStorm).
Hope it helps.

[tanc]

@mvoelker thank you for posting but unfortunately that didn't work for me. I tried blowing away vendor and composer.lock file and re-installing, just in case. I've got gherkin pinned at 4.4.5 and behat at 3.3.1 but I'm still getting the same error.

- Installing behat/gherkin (v4.4.5): Loading from cache
- Installing behat/behat (v3.3.1): Loading from cache

[tanc]

I was wondering whether there was some user cache somewhere for gherkin so I tried running it as another user (root) and it works fine. Very odd. Does anyone know where the gherkin cache is on MacOS?

[QassaDevelopers]

For linux, it is located in the /tmp dir I believe.
Also, you can disable the cache in your behat.yml file, using this:

  gherkin:
    cache: ~

Full answer here.

[tanc]

All good now. I destroyed everything and rebuilt with pinned gherkin and behat and tests are running fine.

[everzet]

@pfrenssen I can't replicate it in 3.4.1:

Can you confirm that bug is still there on your side?

[fbourigault]

To reproduce the issue, you have to run vendor/bin/behat features/homepage.feature:XX where XX is not a Scenario: line (i.e. any step in the scenario).

[ciaranmcnulty]

@fbourigault I don't think that has ever worked

[fbourigault]

Humm right. I got fooled by PHPStorm which resolve the scenario line from where in the file you triggered a behat run.

[warmwaterkruik]

We also can't run a single feature anymore since the update to Behat 3.4. We start a single feature with bin/behat feature/natarch-continuous/ArchiveLandingPage.feature - I seem to have found the culprit within the Behat\Behat\Gherkin\Specification\Locator\FilesystemFeatureLocator::locateSpecifications() - That method instantiates the LazyFeatureIterator with the $locator = 'features/natarch-continuous/ArchiveLandingPage.feature' and $filters = array(new PathsFilter($suiteLocators));
If I set the $filters = []; Then the single feature is run. I can also run bin/behat features/natarch-continuous/ and it finds and runs the features inside the directory.

This is our behat.yml configuration.

default:
  suites:
    natarch_features:
      paths: [ features ]
      contexts:
        - Drupal\DrupalExtension\Context\DrupalContext
        - Drupal\DrupalExtension\Context\MessageContext
        - Drupal\DrupalExtension\Context\DrushContext
        - Drupal\DrupalExtension\Context\MarkupContext
        - rdx\behatvars\BehatVariablesContext
        - FeatureContext
        - OtapDrushContext
        - SeleniumContext
        - CustomSoapContext
        - DebugContext:
           screen_shot_path: project/behat/screenshots
  formatters:
    pretty:
    junit:
      output_path: %paths.base%/build
  extensions:
    Behat\MinkExtension:
      goutte:
        guzzle_parameters:
          verify: false
      javascript_session: selenium2
      selenium2:
        wd_host: http://natarch.dev:8643/wd/hub
#        wd_host: http://<your-local-ip-address>:4444/wd/hub
      base_url: http://natarch.dev
#      default_session: selenium2
#      browser_name: 'firefox'
      files_path: %paths.base%/features/dummy
    Drupal\DrupalExtension:
      subcontexts:
        paths:
          - %paths.base%/features/bootstrap
      blackbox: ~
      api_driver: 'drupal'
      drush_driver: 'drush'
      drush:
        root: '/var/www/natarch'
      drupal:
        drupal_root: '/var/www/natarch'
      text:
        username_field: "Gebruikersnaam"
        password_field: "Wachtwoord"
        log_in: "Inloggen"
        log_out: "Uitloggen"
      region_map:
        breadcrumb: ".region--breadcrumb"
        content: ".region-content, .layout-content"
        footer: ".region--footer"
        header: ".region--header"
        highlighted: ".region--highlighted"
        main_links: ".region--main-links"
        primary_menu: ".region--primary-menu"
        secondary_menu: "#block-natarch-account-menu"
        sidebar_first: ".layout-sidebar-first"
        site_menu_first: ".region--site-menu-first"
        site_menu_second: ".region--site-menu-second"
        title_image: ".region--title-image"
      selectors:
        message_selector: ".messages"
        error_message_selector: ".messages.messages--error"
        success_message_selector: ".messages.messages--status"
    Behat\SoapExtension:
          # An associative array as second argument for \SoapClient::__soapCall().
          options: {}
          # An associative array as parameters for \DOMXpath::registerNamespace().
          namespaces: {}
    rdx\behatvars\BehatVariablesExtension: ~
testserver:
  extensions:
    Behat\MinkExtension:
      selenium2:
        wd_host: http://miauwmiauwmiauw.test.rad.lan:8643/wd/hub
      base_url: https://test.miauwmiauwmiauw.nl
    Drupal\DrupalExtension:
      drush:
        root: '/var/www/miauwmiauwmiauw-web'
      drupal:
        drupal_root: '/var/www/miauwmiauwmiauw-web'
testzap:
  extensions:
    Behat\MinkExtension:
      selenium2:
        #wd_host: http://miauwmiauwmiauw.test.rad.lan:8643/wd/hub
        wd_host: http://miauwmiauwmiauw:8643/wd/hub
        capabilities:
          proxy:
            proxyType: 'manual'
            httpProxy: 'localhost:6667'

Can we remove $filters = array(new PathsFilter($suiteLocators)); from the FilesystemFeatureLocator?

[SunMar]

Hi,

Is there any progress made on this? We're stuk on Behat 3.3 as well because of this.

Using @warmwaterkruik his information I tried to dive deeper into it and it's a real maze.

What I encountered was in PathsFilter the method isFeatureMatch checks if the feature file is in the correct path.

https://github.com/Behat/Gherkin/blob/e66888c99c97b85113822b9f37276c2f9c414c97/src/Behat/Gherkin/Filter/PathsFilter.php#L53

However $feature->getFile() resolves a relative path and not an absolute path, then if your working directory is not the base directory realpath() won't resolve the file, the function returns false and the feature isn't executed. Basically what's discussed in the already referenced #1085 and possibly related to #1056 .

[SunMar]

On a side note, I was able to run tests again from PhpStorm after in the Run/Debug configuration I set the Custom working directory option (in the default configuration for Behat) to the directory in which behat.yml lives (it needs to align with whatever in Behat is %paths.base%).

[jpcaparas]

Thanks @SunMar that was really helpful. Got our tests running now.

[pfrenssen]

I have updated the PR with the proposed fix: https://github.com/Behat/Gherkin/pull/129

[deeplfo]

So, we are having the same problem with Behat 3.4.3, which is that each feature file ($feature->getFile()) has its path represented as relative and not absolute, and isFeatureMatch() always returns false since the base paths set in configuration do not match the relative path attributed to the feature file. Reverting to 3.3.1 fixed this issue.

Is there a solution to this for 3.4.x?

[chrisroane]

This is still a problem for me with behat + gherkin. I was able to work around it by adding a unique tag to tests I want to run individually and just reference that with behat using --tags @MEDSTAT-PUBLICATION-BUCKETS. That seems to be an okay temporary solution until this is fixed.

[fbourigault]

On my side I improved the situation in PHPStorm by setting a working directory in the default run/debug Behat configuration. This works but tests are executed three times (I have three suites defined).

[pfrenssen]

Can you try out https://github.com/Behat/Gherkin/pull/129 ? I have that patch applied and am able to run individual tests, at least on the command line.

Note that this is in a different project (Gherkin) so it's a bit pointless to comment there about Behat tests not being able to run, since that is out of scope for the Gherkin project. It fixes the problem on this side though.

[DonCallisto]

Although issue isn't related to this vendor directly, I came up with a solution that seems to work in my side without any vendor modification: I've just created a behat.yml configuration file not at the same folder level of my scenarios, features and so on.

Something like

/var/www/my_project/tests/behat/{Feature|Suite|Context}
/var/www/my_project/tests/behat/Resources/config/behat.yml

As you can see, Feature don't share a sub path with config file and this result in an absolute path due to

https://github.com/Behat/Gherkin/blob/master/src/Behat/Gherkin/Loader/AbstractFileLoader.php#L32-L46

Running behat as follow

bin/behat --config /var/www/my_project/tests/behat/Resources/config/behat.yml

Seems to make it work from any location (absolute, relative to features, and so on).

Any thoughts?

[blackandred]

Workaround:

In the FeatureContext before class FeatureContext there could be added for example:
chdir(__DIR__ . '/../..');

[jcornide]

Are there any news on this issue?

[DonCallisto]

@jcornide ATM I think there are no updates. You can with my workaround that seems to fit https://github.com/Behat/Behat/issues/1076#issuecomment-374521592

[jcornide]

@DonCallisto it doesn't work for me, I can run the tests from the command line but not from php storm. And the problem is that with symfony 4 only behat 3.4 is compatible

[DonCallisto]

@jcornide Where you run them from is totally irrelevant: maybe you need to adapt configuration I suggest above to config you have into PHPStorm.

[JeffVeit]

This pull request fixes the issue:
https://github.com/Behat/Gherkin/pull/129

[jcornide]

Any news with this issue? this is still being a issue to this day to me, specially with symfony 4

[bgbohling]

I'm having the same problem. I'm trying to run tests from the command line in a Drupal site vendor/bin/behat -c RELATIVE_PATH_TO/behat.local.yml RELATIVE_PATH_TO/features
Here's my suites in behat.local.yml:

  suites:
    default:
      contexts:
        - MY_MODULE_Context
        - Drupal\DrupalExtension\Context\DrupalContext
        - Drupal\DrupalExtension\Context\MinkContext
        - Drupal\DrupalExtension\Context\MessageContext

I've tried reverting to 3.3.1 per some other discussions I've seen but that didn't help.

[jcornide]

@DonCallisto I don't understand exactly what you mean by the different config files. Anyway, is this still an issue? Is someone capable of running behat tests from the IDE with behat 3.5?

[DonCallisto]

@jcornide we have updated behat/gherkin and now we can run individual tests.

This is what I have in composer.json

 "behat/behat": "3.5.0",
 "behat/mink-extension": "2.3.1"

[djdevin]

Still seeing this with 3.5.0.

I had to pull in Behat/Gherkin from https://github.com/Behat/Gherkin/pull/129 to get it to work.

[jcornide]

Same here, I still get the same error

[AlexSkrypnyk]

 "behat/behat": "3.5.0",
 "behat/mink-extension": "2.3.1"

and make sure that you clear Gherkin cache

[RoSk0]

Confirmed versions, clear cache and still doesn't work for me... :cry:

[skilip]

Maybe this documentation helps? http://behat.org/en/latest/user_guide/configuration/suites.html#suite-paths

[mnapoli]

Still happening, here is how I work around it (I want to run Behat in PhpStorm):

default:
    ...

    extensions:
        Behat\Symfony2Extension:
            kernel:
                bootstrap: "tests/tests_bootstrap.php" # <-- add this
                class: App\Kernel

And I create the tests/tests_bootstrap.php file with:

<?php
// Workaround for https://github.com/Behat/Behat/issues/1076
chdir(__DIR__ . '/..');

[milkovsky]

I have a fix. Have a look at the PR: https://github.com/Behat/Behat/pull/1239
I can call single feature file now in terminal as well as in PHPStorm.
e.g. ./bin/behat features/migrate/articles.feature.

Currently my PR fails github build https://ci.appveyor.com/project/everzet/behat/builds/27660949. Maybe somebody can improve that.

At the momemt I am using a patch with the changes from the PR in my project. I have attached the patch in https://github.com/Behat/Behat/pull/1239#issuecomment-535049973

Add this to you composer.json and run composer install:

"extra": {
        "patches": {
            "behat/behat": {
                "#1239: Fix specifications locator to find feature files" : "https://github.com/Behat/Behat/files/3652984/behat-run-single-feature.patch.txt"
            }
        }
  }

[pfrenssen]

@milkovsky doesn't https://github.com/Behat/Gherkin/pull/129 solve it for you?

[SunMar]

@pfrenssen just tried your branch and it does solve that I don't need to specify a custom working directory, Behat properly finds the feature tests

however it does not solve the problem also mentioned by @fbourigault where if you have multiple suites it runs the test against all suites, instead of detecting based on the suite paths configuration which suite(s) to run the tests against

note that this is an issue with and without your branch, what I don't know though is whether this issue was present before commit 08802b7 or not because we only recently split up our tests into multiple suites.

I'll try to do some debugging, see if I can figure out if this is a bug or a "feature"

[SunMar]

@pfrenssen ignore my previous comment about tests being run per suite. apparently if you do not specify a specific suite, then Behat will loop through all the suites. it will perform any BeforeSuite/AfterSuite hooks even if it has no scenario's to run in that suite. so that's what I was seeing being run per suite, not the scenario itself, just the BeforeSuite/AfterSuite cycling. the scenario's themselves were only executed once.

so your branch works fine for me :) it's been two years since this bug was introduced, it'd be nice to have it fixed and no longer need to set a custom working directory

[pfrenssen]

@SunMar thanks for testing!

Yes it is unfortunate this is taking such a long time, but the maintainers have a lot of packages to look after, and there are probably many other things that are more urgent.

[pfrenssen]

I have taken care of https://github.com/Behat/Gherkin/pull/129 just now. It had diverged from the master branch and was affected by a merge conflict and a failing test. I solved them and the build is now green again and ready to merge.

[pfrenssen]

I have updated the issue summary with the lines to add to composer.json to fix this. I hope this will help people more easily find the solution.

[ciaranmcnulty]

This should be resolved in 4.6.1

[milkovsky]

Update to behat 3.6.1 fixed the issue for me.

[joshlopes]

@milkovsky the fix was actually in behat/gherkin 4.6.1 which is what @ciaranmcnulty is trying to say here -- i got confused there for a second, i thought there was a "behat/behat" 4.6.1 version :). Since it was a patch when you updated behat it probably updated gherkin too.

Thanks @ciaranmcnulty !