Karate: Gatling karateFeature seems to be blocking

[gvermoen]

We are seeing some unexpected behaviour when running long-lasting user simulation scenarios with karate-gatling.

With karate, we have defined a user scenario where a user logs in, waits a few seconds, navigates, waits again, opens a document, spends some time reading, etc etc. The scenario takes about 2minutes and works as expected. However, the load model does not work as expected when defining more than 8 concurrent users. For example, when using atOnceUsers(20), we would expect a straight line up, a steady line of 20 active users during 2 minutes, and then a straight line down to zero users. The test should not take much more than 2 minutes.
However, we get the following behaviour:

As you can see, the 20 scenarios are started at the same time, but only 8 of them are executed in parallel. Also, the test takes about 6 minutes.

This lead to our conclusion that when using karate-gatling, the number of concurrent scenarios seem to be limited by the clients logical cpus.

We have tried with a simple Scala scenario, and then everyting works as expected and the scenario is executing in parallel for all users:

We copied our setup from https://github.com/ptrthomas/karate-gatling-demo, only the scenario feature files and feeders are different.

Any ideas on this? We certainly hope it can be fixed, we very happy with the readable and reusable way our performance test is setup now with karate features.

[ptrthomas]

thanks for the excellent analysis. I’m confident we can solve this. can you do us a favor and create a quickstart project that simulates this - and infra setup instructions if needed

Sent from my iPhone

On 25-Jul-2019, at 7:00 AM, gvermoen [e-mail] wrote:

We are seeing some unexpected behaviour when running long-lasting user simulation scenarios with karate-gatling.

With karate, we have defined a user scenario where a user logs in, waits a few seconds, navigates, waits again, opens a document, spends some time reading, etc etc. The scenario takes about 2minutes and works as expected. However, the load model does not work as expected when defining more than 8 concurrent users. For example, when using atOnceUsers(20), we would expect a straight line up, a steady line of 20 active users during 2 minutes, and then a straight line down to zero users. The test should not take much more than 2 minutes.
However, we get the following behaviour:

As you can see, the 20 scenarios are started at the same time, but only 8 of them are executed in parallel. Also, the test takes about 6 minutes.

This lead to our conclusion that when using karate-gatling, the number of concurrent scenarios seem to be limited by the clients logical cpus.

We have tried with a simple Scala scenario, and then everyting works as expected and the scenario is executing in parallel for all users:

We copied our setup from https://github.com/ptrthomas/karate-gatling-demo, only the scenario feature files and feeders are different.

Any ideas on this? We certainly hope it can be fixed, we very happy with the readable and reusable way our performance test is setup now with karate features.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[gvermoen]

karate-gatling-demo.zip

Hi Peter, thanks for your quick response!

I took the karate-gatling-demo and tweaked it a bit. Please find it attached. It has a Karate and a Gatling simulation that perform the same 2 requests with a 5 second wait in between, but the reports are totally different.
CatsKarateSimulation:

CatsGatlingSimulation:

It took me a while to reproduce the behaviour at home, before I realized my Macbook has many more logical cpu's than the VDI I unfortunately have to work with at the office... So if you cannot reproduce, first thing to do is to increase the number of users.

BTW, we also experimented with other simulation configuration setups like rampUsers, rampUsersPerSec, constantUsersPerSec, splitUsers, and while a bit different they also show the same blocking behaviour but it is best visible and understandable with atOnceUsers.

HTH,
Gerben

[ptrthomas]

@gvermoen awesome, this is perfect - thanks, especially the hint about the users. which gives me an idea, tweaking a docker container may be a way to simulate that. I'll take a look

[ptrthomas]

@gvermoen I think I have some good news ^_^
actually your test approach was partly to blame because you were using a Thread.sleep() within a feature file with obvious consequences. but I've added a way in which you can pause() in a Gatling compliant way from within a feature file. best is for me to paste the updated documentation here:

next step is please follow the instructions here: https://github.com/intuit/karate/wiki/Developer-Guide
update your pom.xml to use version 1.0.0 and please test locally and let me know how it goes !

[PetMou]

Hi Peter, i'm a collegue of @gvermoen.
We succesfully applied your solution!
We used:

* def pause = karate.get('__gatling.pause')
* pause(2000)

instead of 'java.lang.Thread.sleep' and now we get a good rampup and progress graph!

We did find out however that 'pause' only accepts integers, and because all numeric variables in Karate are double, something like:

* def aNum = 1000
* pause(aNum)

is not possible. In this example

* print aNum

will return "[print] 1000.0"
So could you adapt the 'pause' function so it will accept variables?

The reason we would like to use variables is to be able to use randomized 'user thinktime'.
So perhaps an idea for the backlog would be a function like "pauseBetween(min, max)"

Thanks in advance!
Peter M

[ptrthomas]

@PetMou thanks for quickly confirming this, I was hoping it would work !!

sure, I can make the change you request ! for now as a workaround, you can do:

* pause(~~aNum)

[PetMou]

Thanks Peter!
This really helps us enormously.
I would never have guessed the ~~ workaround.

[ptrthomas]

@PetMou I actually could not simulate the problem :| I did make one change to the code, can you confirm that it works ? the exported pause() function should accept any kind of number now as the arg

[PetMou]

Hi @ptrthomas ,
Your change works. The "pause()' function now accepts variables for input without the ~~.
I don't know why you were not able to repoduce the issue. Perhaps because we work in a Windows environment?
Anyway, for us you can close the issue.
Thanks!
Peter M

[ptrthomas]

@PetMou thank you for confirming ! I will keep this open until the formal 0.9.5 release

[ptrthomas]

for anyone landing here, this fix should be available in 0.9.5.RC1

[ginessan]

Sorry but not sure if it is working fine.
I am doing a test with the Closed workload model version running my scenario (karate version 0.9.5.RC4) with the next configuration:

scenario1.inject(incrementConcurrentUsers(5).times(5).eachLevelLasting(10 seconds).separatedByRampsLasting(10 seconds).startingFrom(10))
getting the next results:

The point is that, if i modify the "startingFrom(10)" to "startingFrom(100)", the results i get are:

so, my questions are:
1- Why one test took 90 seconds and the second one about 3 minutes?
2- Why in the second test it took 30 seconds to achieve the 100 active users running?
3- If, instead of startingFrom(100) I add startingFrom(10000), the test seems never finish, so; is karate+gatling a suitable tool for running rates of 560 requests/second?

Thanks in advance

[ptrthomas]

@ginessan oh it is quite likely that the CWM is not compatible with Karate, really don't know unless we look at it. would be great if you contribute :) at the very least, do consider submitting a way to replicate: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue

[ptrthomas]

so @ginessan are you planning to help us or not ? see the third post in this thread for an example of how the OP created a way for us to replicate - which led to the fix

[ginessan]

Sure @ptrthomas , defect 961 raised.
Thank in advance!

[ptrthomas]

released 0.9.5