Tox: speed up running unit test suit

[gaborbernat]

I find the unit tests way too slow; for comfort and speed of development I would like them to be <30s for one environment; we should investigate how we can achieve that! Here's what I get now locally:

tox -re py27  74.53s user 7.68s system 1% cpu 1:16:00.14 total

[obestwalter]

I was also wondering and had a look recently. slowest tests are the "systemtests" in test_z_commandline and most time is burnt in os.waitpid. Running with pytest-xdist -n 8 speeds the thing up and we can add that to the tox env also.

In the long run it might be a good idea to have a very good look at how tox tests itself and see how can streamline things and speed them up. I have a few ideas already and would love to tackle that soonish.

[obestwalter]

It 's also not strictly necessary to run your tests through tox:

pip install -e <tox path>
pip install pytest-xdist pytest

pytest -n 8 tests  

========================== test session starts ===========================
platform linux -- Python 3.6.2, pytest-3.1.3, py-1.4.34, pluggy-0.4.0
tox comes from: '/home/oliver/work/tox/tox/tox/__init__.py'
rootdir: /home/oliver/work/tox/tox, inifile: tox.ini
plugins: xdist-1.20.0, timeout-1.2.0, forked-0.2, cov-2.5.1
gw0 [300] / gw1 [300] / gw2 [300] / gw3 [300]
scheduling tests via LoadScheduling
...............................................................................................................................................................ss..................................................................................................x........................................
======================== short test summary info =========================
SKIP [1] tests/test_config.py:2290: condition: sys.platform != 'win32'
SKIP [1] tests/test_interpreters.py:14: condition: sys.platform != 'win32'
XFAIL tests/test_config.py::TestCmdInvocation::()::test_colon_symbol_in_directory_name
  Upstream bug. See #203
=========== 297 passed, 2 skipped, 1 xfailed in 47.88 seconds ============

235.40s user 14.84s system 533% cpu 46.899 total

so only 15 seconds to shave of to get there :)

[gaborbernat]

I think new people will tend to use just the default for their contributions; so from that point of view I would prefer to have it run fast by default, so it's encouraged to run it.

[obestwalter]

pytest-xdist -n 8 speeds the thing up and we can add that to the tox env also.

[gaborbernat]

also for me it takes longer, will need to investigate how come :+1: and some tests fail, maybe because I'm behind a proxy:

3 failed, 293 passed, 3 skipped, 1 xfailed in 72.41 seconds

[obestwalter]

hmm, we have to play with that a bit then and find a good value that works reliably or make it configurable with a sane default. I tried -n 4 on mine now and it is just as fast.

[gaborbernat]

I think it depends on the amount of cores, can use python -n auto to use the number of cores automatically; for me the same three tests fail locally: test_env_variables_added_to_pcall, TestCmdInvocation.test_version, TestToxRun.test_json

also here's the test duration report (no test should take more than 10 seconds at worst!):

28.03s call     tests/test_z_cmdline.py::test_test_usedevelop[.]                                                                                              
27.99s call     tests/test_z_cmdline.py::test_test_usedevelop[src]                                                                                  
25.26s call     tests/test_z_cmdline.py::test_usedevelop_mixed                                                                                                 
24.07s call     tests/test_z_cmdline.py::test_test_piphelp         
21.64s call     tests/test_z_cmdline.py::TestToxRun::test_toxuone_env                                                                                          
17.49s call     tests/test_z_cmdline.py::TestToxRun::test_json                                                                                                 
17.29s call     tests/test_z_cmdline.py::TestToxRun::test_different_config_cwd 
13.73s call     tests/test_z_cmdline.py::test_notest                                                                                                           
12.92s call     tests/test_z_cmdline.py::test_venv_special_chars_issue252                                                                            
12.81s call     tests/test_z_cmdline.py::test_envtmpdir                                                                                                        
12.75s call     tests/test_z_cmdline.py::test_package_install_fails                                                                                 
12.37s call     tests/test_z_cmdline.py::test_PYC                                                                                                              
12.19s call     tests/test_z_cmdline.py::test_alwayscopy_default                                                                                               
12.15s call     tests/test_z_cmdline.py::test_separate_sdist                                                                                                   
12.10s call     tests/test_z_cmdline.py::test_usedevelop                                                                                                       
11.91s call     tests/test_z_cmdline.py::test_alwayscopy                                                                                                       
11.90s call     tests/test_z_cmdline.py::test_develop                                                                  
11.62s call     tests/test_z_cmdline.py::test_env_VIRTUALENV_PYTHON             
11.61s call     tests/test_z_cmdline.py::test_verbosity[-vv]                   
11.55s call     tests/test_z_cmdline.py::test_verbosity[-v]  

[obestwalter]

pytest -n auto: 53.93 seconds

But everything succeeds. Interesting.

[obestwalter]

TestToxRun.test_json reuses example123 fixture together with the other tests in that class. Every bet that they get in each others way. Needs cleaning up.

[gaborbernat]

things get a lot slower on Windows, with all the virtualenv creations we're talking about 15s just the base virtualenv creation 👎 for each test within test_z_cmdline.py

major pain points include 4 second to install setuptools, and then 9 for pip 🤔 (we should definitely add the --no-pip, e.g flags for tests not needing those) and also see why it takes so long

80.08s setup    tests/test_config.py::TestVenvConfig::test_config_parsing_minimal
68.74s call     tests/test_z_cmdline.py::test_test_piphelp
61.35s call     tests/test_z_cmdline.py::test_usedevelop_mixed
46.57s call     tests/test_z_cmdline.py::test_test_usedevelop[.]
42.80s call     tests/test_z_cmdline.py::TestToxRun::test_toxuone_env
42.13s call     tests/test_z_cmdline.py::test_test_usedevelop[src]
40.36s call     tests/test_z_cmdline.py::TestToxRun::test_different_config_cwd
40.25s call     tests/test_z_cmdline.py::test_notest
38.78s call     tests/test_z_cmdline.py::test_envsitepackagesdir
38.40s call     tests/test_z_cmdline.py::TestToxRun::test_json

[obestwalter]

Do you have an idea how to speed things up without loosing coverage?

[gaborbernat]

I have a few, will need to play around with a few things, but first I need to understand all that's slow in detail. And of course dependent tests should be linked up together with fixtures or something similar. On windows a single environment tests take over 1000 seconds which is just unacceptable. For me personally the high turnaround to validate changes puts me off from refactoring, even two environments takes 15+ minutes at best on Windows (given that I run them on parallel).

[obestwalter]

Looks like, if you have a good go on those levels, you can shave of a lot of time already without having to change the test suite as such.

My feeling at the moment is: we are all a bit in refactoring mode (style and replacement of internal constructs) ... getting the whole thing a bit into shape, so that we have a base to build on for the next years. That's definitely a good thing, but while we are doing that we absolutely have to keep the test suite as such in tact - however slow it is. When that first refactoring phase is done and the refactored builds are out (first one will ship tomorrow - I had other stuff to do today, but I'll definitely ship the current state as rc tomorrow), then we can look at refactoring the test suite. What do you think?

[nicoddemus]

I think it is quite sensible to either do refactoring in the code or refactor the test suite (to speed it up in this case), but not both at the same time.

@gaborbernat keep in mind that you don't need to run the entire test suite while developing, running two environments and linting/flake8 locally is enough IMHO (and what I do). The real test is on CI, and it is OK to let all environments be checked there.

[gaborbernat]

@obestwalter as long as we think things through of the changes we should be good; me personally I would prefer the other way around, a fast test suite would greatly help with refactoring. But I'll not be the obstacle for changes in the meantime, I expect speeding up things to take a few weeks (with my availability for this), so no rush here.

[gaborbernat]

I've played around with this last few days, and now I consider this a really hard one. I'll keep doing this, for me the tests are still unreliable locally on a VM :face_with_head_bandage:
Some problems that popped up:

• accessing the network (e.g. for virtualenv creation) makes the tests not really deterministic and unreliable as far as evaluation time goes
• the usage of many subprocess.popen to invoke things, make debugging troublesome - PyCharm also messes up this on its own when rewriting subprocess invocations to allow debugging (e.g. for pip)
• print functions in tox often and somehow inexplicably for now fails with write only accepts byte (under Python 3) (pytest -vvv -s -x --pdb e.g. sometime crashes without any error message under Python 3, this probably is some kind of pytest bug )
• when something fails the messages aren't always the most readable
• some tests step on each others toe
• some test modules are by now kinda long, we should probably make them packages (would help with selecting tests parts probably)
• we should probably add pytest marks for long running tests to allow disabling them for quick testing

All in all working on this felt like exploring a mine field.

[obestwalter]

Thanks for exploring this. I am not surprised that this is tricky.

[cryvate]

@gaborbernat, on your first point, we could change this by invoking virtualenv with the --no-download option (also settable by environment variable).

[RonnyPfannschmidt]

also there should be a easy way to do '-m not acceptance' for the casual testing

[gaborbernat]

A significant time is spent on virtual environment creation. An idea to speed up this is what pip does, instead of re-creating virtualenvs - create a session fixture and copy virtual environments. :+1:

[obestwalter]

instead of re-creating virtualenvs - create a session fixture and copy virtual environments.

That sounds great!

[obestwalter]

... but will lock is into keep using virtualenv for our own tests, right? venv does not support portable environments, or am I understanding this wrong entirely?

[gaborbernat]

Ignore this for now, we'll get back to this later.

[gaborbernat]

I think we can close this for now. It's going to be an on-going effort.

[gaborbernat]

A heads up on this; the new faster virtualenv roughly cut our test suite runtime in half.