K6: If the response takes longer than duration, test should fail or should give an error

[gunesmes]

I had a case to test the thresholds so I added a sleep in the application code which is longer than the thresholds. In this case, the test should have failed but it passed. I realized that the sleep is also longer than --duration by mistake, which means that the response for some requests did not complete.

Case:

1. Add sleep for handling requests in application code (sleep 100 seconds)
2. Add thresholds in your scripts like (in milliseconds), thresholds: {"RTTGet": ["max<400", "avg<200"]}
3. Run the test with the option --duration 60s

For this case, exit code should be different than 0 so it should fail or give an error

[MStoykov]

Hi @gunesmes from what I am understanding you have something like

export default function() {
  http.get("https://example.com/something");
  sleep(100);
}

and you are expecting that the sleep is part of the request ?
It isn't :). The request as a whole happens in the http.get parts and nothing around it counts towards the time it takes ...
If you provide more information about your use case and what you are trying to do (ideally with code samples) I might be able to help you :D

[gunesmes]

@MStoykov sleep should he in the application that you are testing. If the response time of any request is longer than the test duration, the result of the test should be different than the pass.

[MStoykov]

So you have sleep in the server? Does that mean that in your case no requests finish for the duration of the test?
If the request isn't finished no metrics are emitted for it, which is by design as in the usual case you will have thousands upon thousands of requests and you don't want to get the incomplete data for the requests that are being done when the test finishes and are stopped halfway through.

This is somewhat connected to #1053 as this will also happen for not custom metrics (I just tested using http_reqs : ["count>0"], but if I don't do any requests it isn't failed as well ... as we just don't look at thresholds with no data for some reason >.>

[gunesmes]

Yes, intentionally I added some sleep to test some cases, but this sleep was even longer than the test duration and I did not get any response about these requests but tests were passed. However, it can happen without sleep either. Handling this case is hard, there may be some requests are sent just before the test duration ends.

If it is by design it is ok. Is there a way to see if any requests having no responses.

[MStoykov]

sorry for the slower response, I didn't see yours up until now.

If you use duration and/or stages you are extremely likely to have requests that are "inflight" when the test finishes or when there is ramp-down due to stages. At which point the request(s) will be stopped and no metrics will be emitted.
I suppose this is an interesting case. You, in particular, seem to be interested when that happens. But in general in load testing, at least in my opinion, where there are thousands upon thousands of request you are interested in the ones that are finished during the body of the test, not the ones that are started and not finished as they are in the minority and will make the analysis of the data harder.

In your case, I think k6 never finishes a single iteration in which case this warning will be logged

WARN[0008] No data generated, because no script iterations finished, consider making the test duration longer

Maybe we should also exit with a non-zero status code, as I don't think this is something that anyone wants.

[gunesmes]

@MStoykov thank you for the explanation.