Cucumber-jvm: Cucumber JVM is hanging when trying to generate suggestions on how to write undefined step

[ericalofstrom]

Summary

If you have defined own parameter types for Cucumber Expressions, cucumber hangs forever (and in the process eats up all your memory) when you have too many own defined parameters in union with an undefined step that contains approximately 8 or more parameters.
It seems to be related to the functionality that prints out a suggestion on how you can write the undefined step, because if the step is defined than the test runs fine. It also seems to be related to how many self-defined parameter types because if you decrease the number of self-defined parameter types then you successfully get suggestions and the test runs fine.

The suggestion I am referring to is the one that comes in the output when you have an undefined step, which contains the line: Write code here that turns the phrase above into concrete actions

Expected Behavior

Cucumber should run the test.

Current Behavior

The JVM eats up all the memory on the computer and never finishes. When I check the process that runs the test I can see that it is just GC:ing all the time.

Possible Solution

Not really a solution for the issue, but it would solve the issue if you could turn of the suggestion feature, for example an if case that does not run the code that generates the suggestions.
https://github.com/cucumber/cucumber-jvm/blob/8273780e19e22463c8458ca3dded5790b8c8f670/core/src/main/java/cucumber/runner/Runner.java#L75-L83

Steps to Reproduce (for bugs)

1. Define about 25+ own parameter types in the type registry, with the generic regexp: [A-Z]
2. Define an undefined step with 8 possible parameter values in all caps
3. Run the test

Context & Motivation

The main issue seems to be too many self-defined parameter types combined with at least one undefined steps with many possible parameters, that generates way to many possibilities for the suggestion feature. I think it would be good to be able to turn of the hint possibility because firstly it causes the problem, and secondly it is not really a relevant feature when I have to scroll through 50+ suggestions (due to how many self-defined parameters we have)

Your Environment

• Version used: 4.2.6
• Operating System and version: Linux Arch version 4.20.13
• Link to your project: Proprietary so not possible to link

[mpkorstanje]

You've found a combinatorial explosion in an aptly named CombinatorialGeneratedExpressionFactory class. When I ran this on my machine I ran out of memory rather quickly while generating the permutations.

https://github.com/cucumber/cucumber/blob/81d16a9c7cf0473a3af6e1b442a22b5083db2c13/cucumber-expressions/java/src/main/java/io/cucumber/cucumberexpressions/CombinatorialGeneratedExpressionFactory.java#L24-L44

If anything it could be more memory efficient by using a stack for currentParameterTypes and poping the elements after the call to generatePermutations.

[ericalofstrom]

Cool! I'm guessing that might take a while to implement?
Apart from the issue, is it possible to add the possibility to turn off the suggestions anyhow? With as many self-defined parameter types as we have, the suggestions printed are way to many to be relevant.

[mpkorstanje]

Cool! I'm guessing that might take a while to implement?

Yes. The project is mostly driven by volunteers myself included. We can however release quite rapidly so once a good fix has been implemented it should be available on short notice.

Apart from the issue, is it possible to add the possibility to turn off the suggestions anyhow? With as many self-defined parameter types as we have, the suggestions printed are way to many to be relevant.

For the time being you could build your own version of cucumber-core and comment out the snippet generating code in Runner.

[mpkorstanje]

Another contributing factor to the lock up, we don't immediately rethrow OOM.

https://github.com/cucumber/cucumber-jvm/blob/master/core/src/main/java/cucumber/runner/PickleStepDefinitionMatch.java#L53-L54

https://github.com/cucumber/cucumber-jvm/blob/master/core/src/main/java/cucumber/runner/TestStep.java#L50-L52

[mpkorstanje]

The main issue seems to be too many self-defined parameter types combined with at least one undefined steps with many possible parameters, that generates way to many possibilities for the suggestion feature. I think it would be good to be able to turn of the hint possibility because firstly it causes the problem, and secondly it is not really a relevant feature when I have to scroll through 50+ suggestions (due to how many self-defined parameters we have)

Given the predefined types the system works quite well. It's only when you introduce a parameter type with a rather generic regex that the amount and non-specificity of the suggested snippets becomes a problem. So I would rather see a solution that address this then one that simply turns the suggestions off altogether.

[mpkorstanje]

@ericalofstrom a much faster work around would be exclude your custom parameter types from snippet generation altogether. You have to use the overloaded constructor. E.g:

new ParameterType<>(
        "myType",
        "[a-z]",
        String.class,
        new Transformer<String>() {
            @Override
            public String transform(String arg) {
                return arg;
            }
        },
        false, // useForSnippet
        false  // preferForRegexpMatch
);

[ericalofstrom]

@mpkorstanje Cool, I will check that out! Thank you so much for the help, I really appreciate it!

The main reason why I use such a broad regexp is actually that when I use the defineParameterType(ParameterType.fromEnum I get an undefined step error if the Enum is misspelled. However, if I instead define it myself with a broader regexp that is a match I then get an error when creating the Enum instead. So rather than getting an undefined step I get an error saying that MISSPELED is not a valid enum value of class words, which I think is a more informative error on what went wrong. But yeah, I do realize that it might not be how the regexp part was meant to be defined :)

[mpkorstanje]

That's kinda neat.

Thanks for the clear problem description. It made it rather easy to find the problem.

[ericalofstrom]

Thank you for your help! Adding false on my self-defined parameters did the trick without building my own version of cucumber core :)

Also I would like to say that I really appreciate everything you guys do with cucumber. It is really cool to see how the framework evolves, so thank you for you hard work :)

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.