Behat: Why BeforeFeature and AfterFeature are static methods? I can not use my dependencies for clear db before each feature

[cve]

Hi,

for DRY I placed my assumptions in the Background section which should run once before the first scenario per feature and It works, but how can I prepare database for that if I'm placed my clear db functions in BeforeScenario hook, so the permissions will be clear as soon as first scenario was triggered... hmm so... I'm trying to put this in BeforeFeature and AfterFeature hooks but... it's static methods and how can I use my dependencies?

menu.feature

Feature: Menu structure
  In order to display menu
  As an API consumer
  I need to be able to fetch menu with items which I've access to

  Background:
    Given the following permissions exists:
      | name            | code            |
      | USER_LIST       | USER_LIST       |
      | PERMISSION_LIST | PERMISSION_LIST |

  Scenario: Get menu for super admin
    Given I am logged in as super admin
    When I send a GET request to "/api/menu"
    Then the JSON should be equal to:
        """
        [
          {
            "name": "panel",
            "label": "Panel",
          },
          {
            "name": "permission",
            "label": "Permissions",
          }
        ]
        """

SymfonyDoctrineContext

<?php

use Behat\Behat\Context\Context;
use Doctrine\Common\Persistence\ObjectManager;
use Doctrine\ORM\EntityManagerInterface;
use Doctrine\ORM\Tools\SchemaTool;

/**
 * Provides hooks for building and cleaning up a database schema with Doctrine.
 *
 * While building the schema it takes all the entity metadata known to Doctrine.
 */
class SymfonyDoctrineContext implements Context
{
    /**
     * @var ObjectManager
     */
    private $em;

    /**
     * SymfonyDoctrineContext constructor.
     *
     * @param EntityManagerInterface $em
     */
    public function __construct(EntityManagerInterface $em)
    {
        $this->em = $em;
    }

    /**
     * @BeforeFeature
     *
     * @return null
     */
    static public function beforeFeature()
    {
        // I can not use this here
        $this->buildSchema();
    }

    /**
     * @AfterFeature
     */
    static public function afterFeature()
    {
        // I can not use this here
        $this->em->clear();
    }

    /**
     * @return void
     */
    protected function buildSchema()
    {
        $metadata = $this->getMetadata();

        if (!empty($metadata)) {
            $tool = new SchemaTool($this->em);
            $tool->dropSchema($metadata);
            $tool->createSchema($metadata);
        }
    }

    /**
     * @return array
     */
    protected function getMetadata()
    {
        return $this->em->getMetadataFactory()->getAllMetadata();
    }
}

behat.yml

default:
    extensions:
        Behat\Symfony2Extension: ~
        Rezzza\RestApiBehatExtension\Extension:
            rest:
                base_url: http://example.local/
                store_response: true
                adaptor_name: curl
    suites:
        default:
            contexts:
                - FeatureContext:
                    groupManager:   '@fos_user.group_manager'
                    userManager:   '@fos_user.user_manager'
                    permissionManager: '@manager'
                - SymfonyDoctrineContext:
                    em:   '@doctrine.orm.entity_manager'
                - Rezzza\RestApiBehatExtension\RestApiContext
                - Rezzza\RestApiBehatExtension\Json\JsonContext

[stof]

because we don't have a context instance at this point, as each scenario gets a separate context instance.

[cve]

thanks for your reply, but any suggestions how can I clear db in before and after feature for Symfony?

[spolischook]

@cve you can create subscriber make it as service and inject everything you want.
See for example:

• Subscriber
• service yml configuration

Keep attention to tag event_dispatcher.subscriber

[cve]

@spolischook thanks for your reply, but links are broken

[spolischook]

Sorry, there are right links:

• Subscirber
• service yml configuration

[cve]

I'm wonder why there is no out of the box solution for that... IMHO Functional tests need's db interaction with 99% cases... In Laravel there is DatabaseMigrations and DatabaseTransactions traits https://laravel.com/docs/5.1/testing#working-with-databases.
Is there any easier solution instead of making custom listener?

[spolischook]

/**
     * @var ObjectManager
     */
    static protected $em;

    /**
     * SymfonyDoctrineContext constructor.
     *
     * @param EntityManagerInterface $em
     */
    public function __construct(EntityManagerInterface $em)
    {
        self::em = $em;
    }

but listener will be more clear and extendable.

[pamil]

Just do it before each scenario :)

[cve]

@pamil before each scenario I have to repeat the background section...

[cve]

ok guys, thanks for advices

[cve]

I'm added following and methods beforeFeature and afterFeature are not called.

BehatEventSubscriber

<?php

namespace Behat;


use Behat\Behat\EventDispatcher\Event\AfterFeatureTested;
use Behat\Behat\EventDispatcher\Event\BeforeFeatureTested;
use Doctrine\ORM\EntityManagerInterface;
use Doctrine\ORM\Tools\SchemaTool;
use Symfony\Component\EventDispatcher\EventSubscriberInterface;

class BehatEventSubscriber implements EventSubscriberInterface
{
    /**
     * @var EntityManagerInterface
     */
    private $em;

    /**
     * BeforeAfterFeature constructor.
     * @param EntityManagerInterface $em
     */
    public function __construct(EntityManagerInterface $em)
    {
        $this->em = $em;
    }

    public function beforeFeature()
    {
        $metadata = $this->em->getMetadataFactory()->getAllMetadata();

        if (!empty($metadata)) {
            $tool = new SchemaTool($this->em);
            $tool->dropSchema($metadata);
            $tool->createSchema($metadata);
        }
    }

    public function afterFeature()
    {
        $this->em->clear();
    }

    /**
     * Returns an array of event names this subscriber wants to listen to.
     *
     * @return array The event names to listen to
     */
    public static function getSubscribedEvents()
    {
        return [
            BeforeFeatureTested::BEFORE => ['beforeFeature', 100],
            AfterFeatureTested::AFTER => ['afterFeature', -100],
        ];
    }
}

services.yml

services:
    behat.behat_event_subscriber:
        class: Behat\BehatEventSubscriber
        arguments: ['@doctrine.orm.entity_manager']
        tags:
            - { name: event_dispatcher.subscriber }

[avant1]

There are Before_Feature_ and Before_Scenario_ hooks. The latter are described here: http://docs.behat.org/en/latest/user_guide/context/hooks.html#scenario-hooks

And those methods are not static. So you can inject any service you need.

So the execution order is the following:

• BeforeFeature hook
• BeforeScenario hook
• scenario background
• scenario1
• BeforeScenario hook
• scenario background
• scenario2
  and etc.

before each scenario I have to repeat the background section...

Hooks are not connected with background. Background is a way of making examples shorter and removing duplication of initial steps of multiply scenarios in one feature.

And, if I understand your needs, you probably want to use BeforeScenario hook to clear database.
It is needed to make sure that each scenario uses empty (or some initial state) database, and scenarios of one feature do not depend on each other.

When you have multiply scenarios in one feature and you do not clear application state between scenarios you get fragile tests (if you change order of scenarios execution or run part of scenarios tests may not pass). Also your tests may even pass when your application is broken.

[cve]

What I'm trying to achieve is that:

Feature: Menu structure
  In order to display menu
  As an API consumer
  I need to be able to fetch menu with items which I've access to

  Background:
    Given the following permissions exists:
      | name            | code            |
      | USER_LIST       | USER_LIST       |
      | PERMISSION_LIST | PERMISSION_LIST |

  Scenario: Get menu for super admin
    Given I am logged in as super admin
    When I send a GET request to "/api/menu"
    Then the JSON should be equal to:
    """
    [
      {
        "name": "panel",
        "label": "Panel",
        "path": "/",
        "icon": "fa fa-th"
      },
      {
        "name": "permission",
        "label": "Uprawnienia",
        "path": "/permission",
        "icon": "fa fa-list"
      }
    ]
    """

  Scenario: Get menu for admin group
    Given I am logged in as "admin" with "admin" group
    When I send a GET request to "/api/menu"
    Then the JSON should be equal to:
    """
    [
      {
        "name": "panel",
        "label": "Panel",
        "path": "/",
        "icon": "fa fa-th"
      }
    ]
    """

Please note that preparation db for all scenarios is in one place (in background) for reason which you say:

Background is a way of making examples shorter and removing duplication of initial steps of multiply scenarios in one feature.

But without that, my scenarios have duplications of "Given the following permissions exitsts...":

Feature: Menu structure
  In order to display menu
  As an API consumer
  I need to be able to fetch menu with items which I've access to

  Scenario: Get menu for super admin
    Given the following permissions exists:
      | name            | code            |
      | USER_LIST       | USER_LIST       |
      | PERMISSION_LIST | PERMISSION_LIST |
    Given I am logged in as super admin
    When I send a GET request to "/api/menu"
    Then the JSON should be equal to:
    """
    [
      {
        "name": "panel",
        "label": "Panel",
        "path": "/",
        "icon": "fa fa-th"
      },
      {
        "name": "permission",
        "label": "Uprawnienia",
        "path": "/permission",
        "icon": "fa fa-list"
      }
    ]
    """

  Scenario: Get menu for admin group
    Given the following permissions exists:
      | name            | code            |
      | USER_LIST       | USER_LIST       |
      | PERMISSION_LIST | PERMISSION_LIST |
    Given I am logged in as "admin" with "admin" group
    When I send a GET request to "/api/menu"
    Then the JSON should be equal to:
    """
    [
      {
        "name": "panel",
        "label": "Panel",
        "path": "/",
        "icon": "fa fa-th"
      }
    ]
    """

p.s. @avant1, saying "before each scenario I have to repeat the background section..." I want to reply to @pamil, sorry for not mentioning him before

[spolischook]

@cve You need create (if you not have one) extension to load this configuration and also you need to load extension in behat.yml like:

default:
  extensions:
    Acme\DemoBundle\Tests\Behat\MyExtension: ~

[cve]

@spolischook ok, I will try, thanks

[avant1]

Just do it before each scenario :)

I thought @pamil was talking about BeforeScenario hooks, not background section.
So I wanted to make things more clear about the difference between BeforeScenario, BeforeSuite hooks and Background section.
But my understanding of quoted words could be wrong :smile:

But without that, my scenarios have duplications of "Given the following permissions exitsts...":

Using Background in your first example seems valid for me.
I was just trying to say that it makes sense to clear database before each _scenario_, not before _feature_.
Since you have two scenarios in this feature, second scenario will get "dirty" application state (which was left after first scenario).

[cve]

Since you have two scenarios in this feature, second scenario will get "dirty" application state (which was left after first scenario).

@avant1 you are right, but I can't imagine how to maitain bigger systems with that approach, but thanks for clarification about "dirty" state app - it makes sense.

@pamil sorry for misunderstand you, now I see what you meant, thanks

[spolischook]

it makes sense to clear database before each scenario, not before feature.

second scenario will get "dirty" application state

Clear database take a time that can be decreased if clear it before feature, not the scenario (yes, I know that it violates the BDD spec)
No problem with dirty state if your scenarios locate in one feature - you always can see what was happen before, in previous scenarios - this approach has some disadvantages by coupling scenarios, but also this can be more faster, especially on large db

[pamil]

@spolischook for Sylius we clear the database before each of nearly 500 scenarios and it results in 32s / 8% of the time (Blackfire profile). If you use Travis it should be much more faster, as Travis places databases in the memory.

Enabling FriendsOfBehat/PerformanceExtension have sped up our test suite by 40 seconds / 8% (Sylius/Sylius#5583), it just requires PHP 5.6+.

I just want to remind that there are plenty of the ways to optimize your Behat suite and some even without that big tradeoffs. A few more optimization tips may be found here.

[avant1]

I can't imagine how to maitain bigger systems with that approach

We use this approach on large enough projects (with large number of scenarios, and complex behat suites setups).
The only thing that you need to do is to prepare the application state before each scenario. And since you already use those backgrounds to achieve this - you do not need to do anything else.

Clear database take a time that can be decreased if clear it before feature, not the scenario (yes, I know that it violates the BDD spec)

As @pamil have shown, when we talking about some kind of UI tests (where real calls to application are made) database clear step will take very small amount of time comparing to total time of tests execution.
Basically, all you need is to execute set of DELETE statements in correct order. Recreating database schema may take longer time and is not always needed.

Background section execution may take rather long time. But since you execute it anyway for each example (as I see from your scenario) - you will not make tests run much faster. You could move Background to the the beginning of first scenario, and it will not be executed several times, but please do not do that :smile:

Without clearing application state between scenarios:

• you get fragile and not-easy-to-debug tests
• you cannot just run one example in feature, as it may not pass
• you cannot make some actions in Background (like creating user with concrete email as it may lead to database unique constraint error) - or you should check if this action has already beed done, but this will make your code much more complex
• it can be hard to use your scenarios as documentation (since each scenario may depend on previous scenarios of that feature, and you cannot discuss single scenario without having all previous scenarios in mind)
• it is much more hard to divide your scenarios into groups using tags (often in can be useful)

So you loose part of power that BDD together with behat give to you.

Tests speed is very important, I agree. Nobody likes waiting.
But when you use dirty application state for most of your scenarios, you tests may not detect problems with the application. And one day tests will pass and broken application will be deployed to production servers.

There are several ways to make tests faster.
And here are two of them (in addition to @pamil's comment):

• for some of scenarios use step implementations that will use only your domain objects, without requests to web server or database - it is not always possible, and it is quite hard to implement, but this approach has lots of advantages
• just run your tests in parallel and add more test runners - it always works

[pamil]

@avant1 can't agree more! 👍

[phil-davis]

This is a nice little back and forth discussion of test isolation vs performance. So an example from me. Running in my VM on my i7 laptop, my database fixtures reload takes 9 seconds. On Travis it is taking 5 to 6 seconds - so still a fair bit of time. In a suite of features with 100 scenarios that is close to 10 minutes of the elapsed time.
I am thinking to make a separate suite for all the "readonly" tests that do login, login with wrong password, use the app menus to navigate around, scroll up and down lists of stuff, click on things to open them and confirm that the data displays,.. Such a suite can run happily with the fixtures loaded just once at the start. At least that can save some time. And then spend 5.5 seconds per scenario for test isolation of scenarios that do actually change data.

[stof]

@phil-davis this is a valid way to do it IMO (well, unless one of your read-only scenario is not actually read-only)

[phil-davis]

I also discovered the idea to have a BeforeScenario method that is tagged like:

@BeforeScenario @fixtures

and does the fixtures reload.

Then tag scenarios that need fixtures with @fixtures or you can tag a whole feature at the top with @fixtures and it runs he BeforeScenario "fixtures" code for every scenario in the feature - that saves having to paste @fixtures before every scenario.

[Harmyder]

@avant1

But when you use dirty application state for most of your scenarios, you tests may not detect problems with the application.

The same true in reverse. When you use only a clean state you make life easier for your tests.