tox --parallel can "time out" on CI providers

[schinckel]

Firstly, thank for tox, I love it.

I've been using detox for some time to run tests in parallel, and was very excited to see the capability added in to tox itself. However, there is one place where the detox behaviour was nicer.

We have a bunch of tests that take about half an hour to run in one test environment, and another process that takes a similar amount of time to run in a different environment (that tests a part of our build process).

Codeship, the CI provider we use will kill a process that has not produced any output in the last 10 minutes, and determine this is a failure.

tox --parallel auto can trigger this, so we turned on --parallel-live, which outputs the data from the various test cases, but (as expected) the data is interleaved.

detox used to display the environment names, and an ASCII spinner while the tests were running, to show that output was being processed. Then, if any tests in an environment failed, it output the test results for that environment, otherwise it was a (much shorter) message about success.

This is problematic because having all of the data from all test runs is currently required to stop the timeout, but also makes it really hard to see which test cases failed (and could possibly result in interleaved tracebacks in the case of failures happening at the same time in different environments).

[gaborbernat]

tox has the same functionality, we'll print the spinner and outcome of test results. Why is the spinner ignored?

[obestwalter]

Maybe unicode woes? The old spinner was ASCII.

[schinckel]

Yes, running tests locally I can see it is a much prettier unicode spinner.

When I watch the tests run on codeship (without the --parallel-live setting), then indeed it doesn't show the spinner.

I'll chase it up with them.

[schinckel]

Here's a screenshot showing what it looks like when tests are executing.

[gaborbernat]

could be the spinner gets ignored, which makes it seem static, I'll look into a way to enable some ASCI fallback🤔

[schinckel]

I've got a fork I'm trying that with: I just have a spinner that uses + and x for now ;)

Once I get that build onto my CI server, I'll report back.

[schinckel]

Okay, it looks like it's got something to do with the mechanism being used to update the spinner.

I wonder how the detox spinner was updating the value, compared to how the new spinner is implemented.

A support person from Codeship said:

yeah unfortunately the way our system captures logs from the build machines doesn't really work with the fancy output from some tools. When tools are continually updating the same exact line, rather than printing a new one, it is especially problematic.

But, again, it used to work in detox.

[gaborbernat]

create a plugin that prints a new line every 10 second?😅

[schinckel]

Actually, that's probably a better idea than writing a command in codeship that does it - then I could install it, rather than having to duplicate the command line in every project I have...

[schinckel]

FWIW, bitbucket pipelines handles it in an interestingly different way: https://bitbucket.org/schinckel/django-keyset-pagination/addon/pipelines/home#!/results/23

Bitbucket

[themr0c]

Hi, on bamboo the new UTF-8 spinner is also causing trouble. It is producing lots of garbage output: each move of the spinner is added to the log.

It would be very nice to have a way to disable the spinner.

[gaborbernat]

We need to add an environment variable that disables it, which one could set for some CIs; as it's slightly different than the scope of this PR I've created #1184 to handle your ask.