Cucumber-jvm: Support step hooks

[ShijunK]

Feature: https://github.com/cucumber/cucumber/wiki/Hooks

I am surprised that cucumber-jvm does not support step hooks. For example, when I use cucumber-jvm to build web application acceptance tests, I'd like to take screenshot after each step. Without step hooks, I have to invoke the method in every single step. painful.

Any reason don't include this feature?

[aslakhellesoy]

Where is your pull request?

[ShijunK]

@aslakhellesoy , I did not make any change yet. Just want to see if there is any interesting for this feature in the cucumber-jvm community or any objection. That's it.

[aslakhellesoy]

It hasn't been a frequently requested feature in Cucumber-JVM, which is why nobody has implemented it yet.

I'm curious to understand why you need to take a screenshot for every step instead of just doing it in the After hook?

[ShijunK]

thanks @aslakhellesoy , I will see if I can get it done.

For your question, I'd say because human beings are visual creatures and same reason why National Geography is awesome? Kidding, the real reasons are 1) it is much easier to verify the steps themselves are doing the right thing with screenshot per step; 2) reports with step by step screenshot are much easier for QA and business analyst.

[aslakhellesoy]

1) it is much easier to verify the steps themselves are doing the right thing with screenshot per step

It sounds like you don't trust your steps to fail when something is wrong. It sounds like you rely on a lot of manual work to make sure scenarios are passing.

2) reports with step by step screenshot are much easier for QA and business analyst.

I would actually argue the opposite. It sounds very time consuming for QA and business analysts to look at screenshots for every single step that _passed_.

Don't you think their job would be easier if screenshots were only attached for failed steps? This can be implemented like this.

[ShijunK]

@aslakhellesoy , in an ideal world, ruled by engineer, I agree. But, does engineer make final call in the real world? Make adjustment for other players in the team, that's my strategy to promote cucumber and BDD. It is simple to say:" hey, this is the right way, this is good for you". However, sometimes, it is not going to help you to win the game. ;)

[aslakhellesoy]

@ShijunK so am I right that your cucumber scenarios might report "passed" when actually they should have failed?

[ShijunK]

@aslakhellesoy , that's true, cucumber scenarios themselves are created by developers and QAs, so just like application under testing, they could have bugs as well. That's one of the reasons, why need a afterStep hook to take screenshots step by step, so we can verify the scenario&steps. Of cause, once it is verified, the hook should be turn off, maybe by an environment variable. So that don't waste time and resource on continuous integration server.

[aslakhellesoy]

Sure tests, can have bugs. They come in two categories:

• false positives (tests that pass when they should fail)
• false negatives (tests that fail when they should pass)

False negatives aren't dangerous because the test runner tells you about them when you run the tests.
Then you fix them.

False positives are dangerous because the test runner won't tell you about them.
The vast majority of false positives happen when the programmers forget to write assertions,
or when they write stupid assertions.

A great way to avoid this is to try and force a test to fail when you first write it,
so you know it's able to discover errors.

If you follow TDD or BDD you get this automatically, because you start by writing a failing test.
You can achieve this without TDD or BDD for existing code too. Say you have a screen with a header
"Welcome". When you write a test for this, assert that the header is "Hello". Watch it
fail and then change it assert "Welcome".

Cucumber is an opinionated tool. It assumes you trust your own test code (steps and step definitions).

If someone can come up with a use case for After Step Hooks that is different from
"I don't trust Cucumber or I use it badly" then I'm all for adding them. I haven't
seen such a use case yet, so I kind of regret adding it to Cucumber-Ruby.

[marty916]

@ShijunK it sounds to me like as an engineer you are "pushing" this to QA. By that I mean, you set it up and think it's helpful (as do I) but now you are bringing them in the loop. So in essence treating BDD like TDD, in that it is owned by the programmers. The one of the many benefits of thinking BDD is getting the getting the whole team on board with the living requirement and working through the acceptance scenarios together. QA should be working side by side with the engineer as these scenarios are being developed. I usually pair a programmer and tester in the beginning stages of BDD. They'll see the screens popping and get a comfort level that the feature is implemented and the tests are working.

What happens when you have too many "UI Popping" tests and you want to refactor your testing to do some of the browser pieces headless. You won't have screen shots to capture. What happens then?

It feels to me that capturing all of this output is going to take you away from lean but creating new WIP being that the tests haven't really passed until someone has reviewed them.

[ShijunK]

@aslakhellesoy , you don't need to convince me. I agree with you all but the last part. I am trying to promote Cucumber and BDD. When you introduce something new into an organization, there is aways some doubt and skeptic. Instead of having an attitude that "if you don't trust me, go away", why not doing something to make it easier for use. If that's not enough to convince you to add afterStep feature. I am fine, I can work around it.

[ShijunK]

@marty916 , taking screenshots are optional. As you said, because BDD is getting the whole team (business analyst, QA and developer) together, to make everyone else comfortable, a test report with screenshots is better than plan text and summary numbers.

regarding headless testing, there is alternative, for instance, phantomjs can be hooked with selenium webdriver so that you essentially can run your test on a server without UI, but still have the capability to render the screen and take screenshot.

again, I don't know why some much time and effort are spent on the discussion whether or not taking screenshot every steps for a web application test. And, I never say that every test results have to be reviewed by human beings. No, it shouldn't. Only failed tests need to be taken care. And then, that's when those screenshots become handy.

[aslakhellesoy]

The thing is - lots of people are already using Cucumber badly and getting burnt by it. I have an overdue blog post to write about this.

Adding features that makes it even easier to use it badly doesn't help anyone.

I completely understand that people are skeptical to new tools and workflows. They should be! At the same time I'm also against adding "bad" features to Cucumber just to win people over.

In this case it sounds like you have a trust problem. I'm guessing that QA people don't trust Cucumber to discover bugs. If this is the case I think it's your role to convince them by demonstrating that it can!

[aslakhellesoy]

again, I don't know why some much time and effort are spent on the discussion whether or not taking screenshot every steps for a web application test.

If hadn't spent time discussing whether or not to add requested features to Cucumber it would have looked like this by now:

Also known as Feature creep

[ShijunK]

@aslakhellesoy I see your point after reading the google group discussion that you are not happy people pay too much attention on Cucumber's automated testing facet but too less on it's benefit for collaboration and outside-in development methodology. I agree. However, if Cucumber does not have the automated testing facet, it will be really hard to spread it and convince BA and QA to use Cucumber as a collaboration and communication tool. After all, people will ask what cucumber is different from a business requirement document written in MS word? Gherkin does not even allow fancy chart and pictures. What is the benefit of Cucumber in agile, if the change of business can not be verified and tracked automatically via tests?

[aslakhellesoy]

@ShijunK where did you get the impression that we'll remove test automation from Cucumber?

[ShijunK]

@aslakhellesoy from you? https://groups.google.com/d/msg/cukes/zf2KNRUe_cg/N9Ft99zvJu8J

The gist of it is that contrary to popular belief, Cucumber was never meant to be a testing tool,
but a communication/collaboration tool.

from my point of view, the discussion here is way off track. I saw Cucumber has step hooks, and Cucumber-JVM does not. So that I created this issue to ask opinion of implementation. At the same time, I provided a use case of step hook: taking screenshot after each step when using Cucumber for a web application acceptance test.

What cucumber meant to be, How people should use Cucumber correctly, and so on. All are too big topics for a github issue, IMO.

[aslakhellesoy]

If you read the entire thread you'll see a more nuanced explanation from me.

I agree with you - this discussion belongs on the mailing list.

Regarding after step hooks - if we decide to implement them for Cucumber-JVM it will probably be after Gherkin 3 is released. Introducing them now would also mean changing the formatters APIs, and that's something I'd rather not to before we move to Gherkin 3.

We'll leave this issue open for now. We _might_ implement this in the future.

[K76154]

I think Cucumber needs a after step hook for different reasons. I am doing mobile development, so the UI is much more important than browser apps. However, it's very difficult for mobile automation codes to detect minor UI problems like something is off few pixels or is being cut. I can do image comparison, but that's extremely fragile to any UI changes.

Another reason is that I also need to do globalization test, and that means I need to make sure each string isn't truncated in the UI after being translated into different languages. I don't think there's a reliable way to do that in automation codes today, especially on mobile devices. Most APIs only tell you what string it is displaying, but don't tell you whether the string is too long for the space it is given or not.

[aslakhellesoy]

@K76154 now that's a use case that makes sense to me. And thinking about it, the same use case makes sense for browser testing like @ShijunK described: Visually inspect that the screens look good.

I realise I was a bit dismissive of the whole idea yesterday - it was Friday and I'd had a long week!

I'm open to adding this now, but like I said, probably after Gherkin3.

[paoloambrosio]

I believe step hooks might make sense. We should implement them when we switch to Gherkin 3.

I remember implementing After Step hooks in Cucumber-CPP to solve a technical issue for a user (cannot remember which one). It can be useful for debugging or even to automate an iteration showcase, introducing delays between steps and running a few scenarios.

I don't like manual acceptance testing, but I understand that sometimes it might be cheaper to visually assess the correctness of a few tricky scenarios than to implement an automated check. However I totally disagree on the value of a screenshot for each step, especially if this is considered "living documentation".

[diabolo]

I'm not convinced by either use case. Both use cases are better addressed by manual testing!!! I don't believe the cost of examining screenshots after every step is in any way cheaper than manually interacting with the application.

In addition this idea will encourage the writing of imperative stories, as the only way to get a screen shot for each significant UI change is for each significant UI change to be implemented by its own step definition

Fundamentally visual examination of artefacts produced by Cucumber will:

1. Detect less errors than actual interaction with the application
2. Take more time than interacting with the application

Overall it might be better to use Cucumber to do a little bit less really well, and make clear what it can and can't do, rather than oversell it and push it usage outside its comfort zone, in an attempt to make it a silver bullet.

If you really want to visually examine using Cucumber then video would be a better medium (see https://saucelabs.com/home)

[SierraGolf]

up! what is the status for gherkin 3 and the likelihood of this feature being implemented?

I also have a couple of use-cases for which step hooks would make sense, but independent from that I think cucumber implementations should be as symmetric as possible.

fyi, my use-cases would be:

• doing global assertions (cross-cutting-concerns) like
  
  
  
  • expect that there was no error being logged during the step execution
  
  
  • expect that a step did not cause memory problems (memory utilisation should not spike over a certain threshold)
  
  
  • expect that a step did not exceed a specific execution time
  
  
• artificially slowing down test-execution for debugging purposes by being able to add waits between each step (running my android cucumber tests with espresso is like drinking from a fire hose)

[cdewangan]

@aslakhellesoy
I am also looking for this implementation, I am into mobile automation. I am developing framework where the screen shots will be compared from baseline on every steps, I am using cucumber for detailed and nice reporting, that integrates well with Jenkins.

When is plan to implement Gherkin3 or after step hook.

[aslakhellesoy]

@cdewangan Gherkin3 is ongoing. I've seen enough arguments to add After Step hooks, so we'll probably add that.

I'd welcome a pull request for After Step hooks, even before Gherkin3 is ready.

[Evg3n]

Hello,

I need steps hooks too. So, I forked the repo and will work on java implementation. I doesn't look too hard to add but will see. I just started digging in the code but let me know if you have any cons.

Best regards,
Eugene.

[aslakhellesoy]

Go ahead!

[Evg3n]

Pull request #838.

Best regards,
Eugene.

[Evg3n]

Hey @aslakhellesoy could you please provide timeline when pull request #838 will be merged and released? I really need in my project.

Best regards,
Eugene.

[aslakhellesoy]

Hi @Evg3n I don't commit to timelines. I try to work on Open Source on Fridays - that's all I can say.

[IntelloScooby]

I am working on a mobile automation framework and use external reporting API. With this API, I would like to log a PASS, FAIL or INFO status after each step along with the step/step definition name. Right now, I have to add this line of code at the end of each step definition for logging in the external report. This is not a good practice and once our framework and team starts growing, it will be difficult to maintain. I am waiting eagerly for an afterstep hook in cucumber-jvm and a method similar to Scenario.getName (something like Step.getName) to handle this issue.

[IntelloScooby]

Is there any plan for adding step hooks any time soon? I see this conversation was started in 2013 and there also seems to be code already written for this change.

[brasmusson]

@IntelloScooby Speaking for me personally, I'm focusing the time I have available to work on upgrading Cucumber-JVM to use Gherkin v4.0.0. From that perspective I would not want to add step hooks before there is an Cucumber-JVM v2.0.0-SNAPSHOT available that uses Gherkin v4.0.0.

[npsullia]

I would like to add reasons supporting need for step level screenshots. For Pharmaceutical industry the evidence needs to be captured for every single step if it is part of validation checkpoint. This is part of CFR-11 requirement. We are trying to save manual testing effort for auditory and compliance requirements by using automation reports.

[Saieph]

What @npsullia said is absolutely correct an is EXACTLY what I'm trying to do right now, which lead me to this post.

[SanjeevKumarmn]

do we have after step or still under consideration ?

[mpkorstanje]

There currently is no after-step hook but I don't believe the the concept it self has any opposition. There is however no usable pull request. An attempt was made with #938 but this was done against an older version of Gherkin and is not usable.

Keep in mind that Cucumber is mostly developed by its community. If you want something the best way to do so is to implement this functionality and send a PR. If you are unable to do this you'll have to find someone to do this for you.

[eskignax]

After-step hook will be really useful with allure-report.

[mpkorstanje]

@eskignax for reporting purposes try to create a custom formatter instead. They get access to way more events and you won't have to put it in your glue code.

You can use the JSONFormatter as an example.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

[stale[bot]]

This issue has been automatically closed because of inactivity. You can support the Cucumber core team on opencollective.

[Lupeipei]

@ShijunK I am sure you must have solved the problem of getting screenshot after each step. I would like to get screenshot after each step and use each step's description to generate a hash string, the problem I have now is I have no idea how to get a step's description. I have googled but haven't got any effective answers, do you have any suggestion? I am using ruby2.4, Thanks in advance.

[bouillonpi]

Step hooks are not implemented in cucumber 5.0.2 (latest version at this time).
I need it to take screenshots after each failed step.
So far the only solution I found to achieve this is to include the screenshot function in each step which is painful.
Any news about step hooks or workaround? Thanks!

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.