Locust: Exception in task method is handled but not reported as a failure

[O1O1O1O]

Description of issue

If an unhandled exception occurs in a task method it is handled by the framework and
tasks continue to be executed by the Locust but not reported as a failure. If you are
only looking at the final success / failure rate - perhaps programmatically - it is easier
to miss the exception. From the UI you can see exceptions more easily, and download
them as a CSV, but when running in --no-web mode they are not logged.

Expected behavior

My proposal is

• an exception should be treated as a failure and logged as such.
• exceptions should be exported as a CSV when using --no-web

Actual behavior

Exceptions are not treated as a failure - in fact tasks appear to have succeeded, and they are not logged to in the CSV outputs on --no-web mode.

Environment settings

N/A

Steps to reproduce (for bug reports)

Write any task where the code after making the web request fails - perhaps asserting
something on the output, or manipulating data it thought would be returned but wasn't.

[WayneKeenan]

This would be a great, clean, alternative to https://docs.locust.io/en/stable/writing-a-locustfile.html?highlight=raise#manually-controlling-if-a-request-should-be-considered-successful-or-a-failure

As Tasks that use helper functions which make HTTP calls using self.client would be a lot simpler (applies to both the helper and the Task)

[WayneKeenan]

Also, if I have a body of existing code used for testing support being able to just rely on standard python exceptions failing the test is a big plus.

Other than this little niggle Locust is perfect for our needs, I found it yesterday (JMeter casualty) and up and running in no time. Amazing work, a shining example of following the path of least surprise.

Thank you!
Wayne

[O1O1O1O]

This would be a great, clean, alternative to https://docs.locust.io/en/stable/writing-a-locustfile.html?highlight=raise#manually-controlling-if-a-request-should-be-considered-successful-or-a-failure

As Tasks that use helper functions which make HTTP calls using self.client would be a lot simpler (applies to both the helper and the Task)

My tests are for a JSON based API. So I have helpers/wrappers for each API call to do request setup like adding auth headers, building a valid URL and basic result parsing (parsing JSON) and they use the with ... as response pattern. But then my tests using those helpers typically need to do additional test case specific processing eg. to take results from one call and use in another to simulate typical API usage and ensure the requests being made are using correct parameters values and returning valid data, without that requests would just fail. Really they are half regression test, half load test.

I'm a bit of a Python n00b but I guess that I could have my helpers implement their own with mechanism that catches all exceptions not just request ones and fail those. And instead of chaining several API calls together in one task method I could have a nested task sequence for each one that makes more than one call and communicate data between them via instance specific fields - that feels unnatural and contrived to me. But you still have the problem that success/failure is managed and reported at the HTTP request-response level and not the task level even though scheduling and execution within Locust is all about tasks.

[heyman]

Problem with this is that you are not limited to a single HTTP-request per task (you're not even limited to HTTP). So how would we know which endpoint to "fail"? It's very common to make multiple requests within a single task.

[O1O1O1O]

I get your point. So long as you are not going to support statistics aggregated at the task level that's a problem. But if you did support aggregated stats it would be making a task a first class citizen you could report stats and errors on.

However I rather think I'm pushing the limits of what you intended Locust for and maybe I should be grouping my requests into TaskSequence, and not into a single task. With only one request per task there would no longer be any ambiguity about who an unhandled exception was associated with right? This then begs the question - could we get aggregated stats for task sequences and other nested task sets?

[heyman]

This then begs the question - could we get aggregated stats for task sequences and other nested task sets?

You should be able to implement it within your own test scripts by overriding the TaskSet.execute_task method, and firing locust.events.request_success events. However I'm -1 on including it in Locust itself.