Behat: Support testing for expected exceptions

[acoulton]

For testing APIs in particular, it would be helpful to be able to test that a given action throws a particular exception.

For example, a scenario similar to this:

Scenario: Trying to access protected fields without authentication generates an exception
  Given I have a new API object
  Then I should catch an AccessDeniedException when I access the email property
  And the exception message should contain "foo"

The most obvious and reusable way to achieve this would be to chain a step from the "Then I should catch..." definition.

However, after playing around for a while, I can't see a way to implement this in the current architecture. Obviously the existing step chaining won't work because in that case the chained step would fail and so would the scenario. I thought about implementing a hook, but they seem to be able only to read the scenario/step results rather than being able to change them.

I think the best approach would be to support executing a substep directly within a FeatureContext - something like this:

   /**
     * @Then /^I should catch an ([a-zA-Z_]+) when (.+)$/
     */
    public function iShouldCatchAnException($exception_type, $when_step)
    {
        // Execute the action
                $step = new When($when_step);

                try
                {
                    // Execute Step
                }
                catch (\Exception $e)
                {
                   // Test for the expected type and store or rethrow
                }
    }

However, I don't think that's possible at present because the FeatureContext doesn't have a reference to either the StepTester or the DI container, so it's not even able to duplicate the StepTester logic as it can't access the definitions.

I'm not sure the best way to resolve this, but I don't think it's possible without changing the core to pass in one or other of those dependencies to the FeatureContext method somehow? I'm happy to have a go at doing so if you agree that this would be a useful scenario to be able to support?

Otherwise, I think the only option would be to specifically define individual steps, which is fairly brittle and would probably cause duplication especially since often you'll want to test that an exception is thrown from a method that you are anyway testing in non-exception use cases - it would be nice to be able to share that definition.

[everzet]

Behat IS NOT unit testing or specification testing tool. Behat is Scenario-oriented BDD framework with functional testing capabilities as part of communication process between stack-holder and developers. It means, that Behat was created and evolving as agile planning and functional testing methodology. However, if you want to use it as unit testing or spec testing tool, you can, but keep in mind, that we'll not add into Behat functions, that it wasn't intended to do ideologically ;-)

Personally, i would recommend you to use PHPUnit or PHPSpec for internal api testing. Behat was created for black-box like, functional, user-observable behavior testing :-)

[acoulton]

@everzet, thanks for the quick reply (and the tool!).

I appreciate the distinction between Behat and PHPUnit/PHPSpec. I've used PHPUnit before, though I'm new to Behat. However, I thought (and may be wrong) that it was appropriate to use BDD/functional testing for the public API and functionality of a module/library. I'm certainly aware of several Ruby libraries that have cucumber tests for their public API.

In that case, calling a method and receiving an exception seems to me to be equivalent black-box like, functional, user-observable testing in the same way that loading a URL and getting a 404 in Mink is - it all depends on who the user is that is doing the observing? A third-party application developer integrating your library is still a user, aren't they?

Obviously the internal api of your classes and application should be covered by unit testing, as you say.

It's entirely possible I've misunderstood though - as I say I'm new to the BDD concept although already finding it a powerful one. Behat is a great tool!

[everzet]

The whole idea of Behat features describing is built around the idea of defining one single business dictionary for your domain area. It means, that by defining this dictionary one time you can use it in all your project scenarios. But when we're talking bout source code testing, we're talking bout source code, not about custom terms. By building dictionary for a public API testing of your library, you'll finish with duplicating entire php language in step definitions :-) And, what's even worse, at some point, you'll notice, that you're writing dictionary definitions (step definitions) more than testing or describing target behavior. That's the only reason xwhy Behat is oriented on the black box testing.

One of the brillian ideas about BDD - you should describe behavior in terms, that end user should understand. So, when you're talking about website users - it's browser behavior, that you're describing and Behat+Mink fits there awesomely. When you're talking bout console tool user - it's custom CLI user dictionary (run, go to directory, etc.), that you should describe and Behat fits there too: https://github.com/Behat/Behat/blob/master/features/bootstrap/FeatureContext.php#L95. But when you're talking about php library users - you're talking bout source code. And best terms to describe php library behavior to it's users is the source code itself.

It's the reason, why Behat itself tested with Behat features, but Gherkin parser and Mink tool are tested with PHPUnit ;-)

[acoulton]

Many thanks for the explanation, that makes a lot of sense. I'll stick to
phpunit for APIs and move to behat for testing full-stack apps and
components. Was just so impressed with the concept I wanted to use it
everywhere :)

On 8 Oct 2011 15:47, "Konstantin Kudryashov" <
[email protected]>
wrote:

The whole idea of Behat features describing is built around the idea of
defining one single business dictionary for your domain area. It means, that
defining this dictionary one time you can use it in all your project
scenarios. But when we're talking bout source code testing, we're talking
bout source code, not about dictionary. Building dictionary for an public
API testing of your library, you'll finish with duplicating of entire php
language in step definitions :-) And, what's even worse, at some point,
you'll notice, that you're writing dictionary definitions (step definitions)
more than testing or describing target behavior. That's the only reason why
Behat is oriented on the black box testing.

One of the brillian ideas about BDD - you should describe behavior in
terms, that end user should understand. So, when you're talking about
website users - it's browser behavior, that you're describing and Behat+Mink
fits there awesomely. When you're talking bout console tool user - it's
custom CLI user dictionary (run, go to directory, etc.), that you should
describe and Behat fits there too:
https://github.com/Behat/Behat/blob/master/features/bootstrap/FeatureContext.php#L95.
But when you're talking about php library users - you're talking bout source
code. And best terms to describe php library behavior to it's users is the
source code itself.

Reply to this email directly or view it on GitHub:
https://github.com/Behat/Behat/issues/58#issuecomment-2332339

[b01]

@everzet So wait, you're saying that I should not use Behat to perform functional test on my PHP libraries for public APIs.