tox startup is very slow due to setup.py and .tox virtualenvs

[pytoxbot]

• Bitbucket: https://bitbucket.org/hpk42/tox/issue/293
• Originally reported by: spookylukey
• Originally created at: 2015-12-06T16:53:43.040

The issue is described here:

https://bitbucket.org/pypa/setuptools/issues/450/egg_info-command-is-very-slow-if-there-are

It could be considered a performance bug in setuptools, but fixing that is looking quite hard. I'm wondering if fixing it via tox would be better - after all, tox is triggering the really poor performance of setup.py by putting thousands of files in the working directory, and then suffering from it.

Solution 1: specify a toxworkdir outside the package directory - http://tox.readthedocs.org/en/latest/config.html#tox-global-settings

The problem with this is that it is difficult to find a place which would be appropriate when you consider multiple people working on a project, possibly on different platforms. Also, it needs to be applied in every project that notices the problem.

Solution 2: have a different default value for toxworkdir, something like ~/.cache/tox/virtualenvs/{project} , where {project} gets substituted by something appropriate, like a mangled version of the working directory, perhaps combined with a platform name or something. This might also address issue #44 https://bitbucket.org/hpk42/tox/issues/44/cant-share-a-tox-directory-between-os-x

This is a significant pain point for me on multiple projects, I'd be willing to work on a solution if at acceptable approach were suggested.

[pytoxbot]

Original comment by lucaswiman

Take a look at this tox plugin. Seems to work quite well for implementing solution 1, without the cross-platform headaches of specifying a setting in the tox.ini itself: https://bitbucket.org/hpk42/tox/issues/44/cant-share-a-tox-directory-between-os-x#comment-26780734

[cladmi]

It looks like the setuptools issue has been closed: https://github.com/pypa/setuptools/issues/450 and released in setuptools version 28.5 on October 2016 https://github.com/pypa/setuptools/commit/3e3d83ad49e7f45cc8d2f343590f774c41069b74

I have not looked at the impact.

@spookylukey you may be interested

[obestwalter]

Thanks @cladmi for pointing this out. Let's be optimistic then and resolve this :)

[spookylukey]

pypa/setuptools#450 was closed without a patch being merged, but https://github.com/pypa/setuptools/pull/764 was merged, which hopefully addresses this.

[spookylukey]

Using a similar test to the one I used to get metrics for my patch (https://github.com/pypa/setuptools/pull/610 ) I've tested against the new setuptools (34.3.1), which uses a similar method, and I'm getting very similar performance speedups.

With my test case, instead of tox having a startup overhead of:

• 10 minutes (cold start) or 40 seconds (warm start)

...it now has a startup overhead of:

• 40 seconds (cold start) or 1-2 seconds (warm start).

Yay! This makes a world of difference.

To see these benefits, you will need setuptools 28.5 or later in the virtualenv that has tox in it, and I think also in the virtualenvs that tox builds - so you may need to use tox --recreate to force the new version of setuptools to be installed. You also need to avoid use of global_include in your MANIFEST file, which will force all files to be scanned - and is probably not what you wanted anyway, because it will include files from hidden directories like .git, .hg, .tox etc.

The tests aren't exact for a bunch of reasons, but should give a rough idea. The speedup I'm seeing is due to tox having a large number of virtualenvs, so in other cases it will be less extreme. The point to note is that you are no longer punished for having a large number of virtualenvs, which were being unnecessarily scanned by setuptools before (due to them being stored within .tox in the project directory).

[obestwalter]

@spookylukey thanks for verifiying this.

You also need to avoid use of global_include in your MANIFEST file, which will force all files to be scanned - and is probably not what you wanted anyway, because it will include files from hidden directories like .git, .hg, .tox etc.

Just want to highlight this, in case folks end up here wondering why their start up times are still horrendous.

[boompig]

Are there any tips or tricks to speed up tox? From cold or warm start, I am getting a ~40s speed penalty for using tox on top of running mypy and pytest manually in a virtualenv.

Would it be possible for the maintainers to create a single reference page explaining how to speed up tox? For example, locating and purging old/slow dependencies, or the other suggestions on this GH issue?

Perhaps there is (or should be) a tox --profile-perf switch which will explain where tox spends all its time?

For me, this is a serious annoyance with tox and my team is looking at alternatives for this reason. I am including this not to disparage the maintainers, but to let them know that this is a major issue for my team, and possibly other teams as well.

[gaborbernat]

Can you detail what do you compare it against?