Robotframework: Enhanced keyword timeout support

[ailjushkin]

Problem

Architecture of Robot Framework does not provide a functionality to stop the execution of a keyword, running for a long time, waiting while other process get some data.

For instance, cx_Oracle library. Some SQL requests can run for hours and they won't be forcibly stopped even though the Timeout argument is set to ten minutes.

This problem blocks execution of many other tests and make test run process much longer than it is expected to be corresponding to Test and Keyword Timeouts defined in test suites.

Proposal:

Create some sort of robot.run --parallel mode for the native test runner.
Add a thread manager to control threads from outside.

Parallel run of tests can automatically divide slow tests and fast ones. And we can break test execution of each test if it could run in a separate thread (correct me if I'm wrong). But each test must run independently to not block other test cases.

Pabot library could do this, but it needs in refactoring: it now runs processes on operating system level, but not using python threads natively, it run many tests in one process, thus we cannot kill slow o/s/ process because then we break execution of tests that were not started yet.

And, on the other hand, pabot, on my point of view, should be a part of robot framework's core.

[pekkaklarck]

Could you clarify what you are proposing? Notice that adding orchestration functionality into the core is not going to happen.

[ailjushkin]

I thought,

1. why not make the robot.run itself to be able to run tests in parallel, based on python threads?
2. And if we could make it happen, then each test could be ran in parallel.
3. And then, If we run each test in parallel thread, we can set test timeout for a whole thread and stop it after a timeout and if we cannot stop execution inside a thread, we can stop in from a thread manager from the outside to force it.

It's not orchestration. Orchestration in my point of view is some automated process of managing tools on operating system level.
But that could be a feature of the test runner itself.

[pekkaklarck]

Without thinking would this be a useful feature or not, I can tell it's very unlikely anything like this could be implemented with the current development resources. First of all, Robot has some global state and due to that it'a not thread safe. Hopefully we can remove all such state in the future, but it's not going to happen in near future. Another problem is that logging would need to be reimplemented pretty much totally. Finally, we'd need some syntax to control this behavior.

[ailjushkin]

So it's better now to optimize code of user keywords rather than start global refactoring of Robot Framework.

[ailjushkin]

What about logging.

Seems like this could be made by storing results not in xml, but in some local nosql database and then create formatters that get data from this database and output xunit.xml files and other log formats, like html.
But nosql database should be simple and also thread-safe.
Or, we can create independent logging library based in queues that could asyncronouy process log requests to it.

[pekkaklarck]

Everything is possible. The question is who does it. And also what kind of maintenance burden it brings.

[pekkaklarck]

This issue doesn't seem to contain any concrete enhancements that would belong to the core framework and could be implemented in the foreseeable future. Closing.