Tox: Provide a way to reduce the diagnostic output

[pytoxbot]

• Bitbucket: https://bitbucket.org/hpk42/tox/issue/256
• Originally reported by: @nedbat
• Originally created at: 2015-06-08T04:03:50.359

When I run my tests, tox prints a number of lines of information. This is helpful when things are going wrong, but I don't need to see them on every test run.

$ tox -epy27-django14
py27-django14 develop-inst-noop: /Users/ned/coverage/django_coverage_plugin
py27-django14 installed: -f /Users/ned/Downloads/local_pypi,coverage==4.0a6,Django==1.4.20,-e [email protected]:nedbat/django_coverage_plugin.git@b337234512edc50f3962871bd03998da20d26446#egg=django_coverage_plugin-master,six==1.9.0,wheel==0.24.0
py27-django14 runtests: PYTHONHASHSEED='471377262'
py27-django14 runtests: commands[0] | /Users/ned/coverage/django_coverage_plugin/.tox/py27-django14/bin/python -c import tests.banner
CPython 2.7.6; Django 1.4.20
py27-django14 runtests: commands[1] | /Users/ned/coverage/django_coverage_plugin/.tox/py27-django14/bin/python -m unittest discover -b
...........................s.........
----------------------------------------------------------------------
Ran 37 tests in 0.301s

OK (skipped=1)

Here there are 11 lines of output. Five are from tox, and I would like to suppress them.

[pytoxbot]

Original comment by @jayvdb

Hmm. BitKeeper doesnt cross link, so ... #304 is a similar issue.

[pytoxbot]

Original comment by @jayvdb

I would also appreciate a quieter tox for linter tasks, as the errors are drowned in noise. ( https://phabricator.wikimedia.org/T117570 )
A command line flag would be ok; it could also be a flag in each tox testenv section.
Sort of related, I would also like the option to _NEVER_ print out the env, as it can contain secrets which shouldnt be dumped to publicly accessible logs as is usually the case in open source CI. Currently the env appears if there was any exception.

[pytoxbot]

Original comment by @AndreLouisCaron

I'm willing to give a go at this. Any suggestions for the implementation?

[pytoxbot]

Original comment by @The-Compiler

I agree this would be useful - I also run some linters via tox, and the output for those currently looks rather cluttered (33/58 lines by tox):

misc inst-nodeps: /home/florian/proj/qutebrowser/git/.tox/dist/qutebrowser-0.2.1.zip
misc runtests: PYTHONHASHSEED='678912664'
misc runtests: commands[0] | /home/florian/proj/qutebrowser/git/.tox/misc/bin/python scripts/misc_checks.py git

misc runtests: commands[1] | /home/florian/proj/qutebrowser/git/.tox/misc/bin/python scripts/misc_checks.py vcs

misc runtests: commands[2] | /home/florian/proj/qutebrowser/git/.tox/misc/bin/python scripts/misc_checks.py spelling

pep257 inst-nodeps: /home/florian/proj/qutebrowser/git/.tox/dist/qutebrowser-0.2.1.zip
pep257 runtests: PYTHONHASHSEED='678912664'
pep257 runtests: commands[0] | /home/florian/proj/qutebrowser/git/.tox/pep257/bin/python -m pep257 scripts tests qutebrowser --ignore=D102,D103,D209,D402 --match=(?!resources|test_content_disposition).*\.py
pyflakes inst-nodeps: /home/florian/proj/qutebrowser/git/.tox/dist/qutebrowser-0.2.1.zip
pyflakes runtests: PYTHONHASHSEED='678912664'
pyflakes runtests: commands[0] | /home/florian/proj/qutebrowser/git/.tox/pyflakes/bin/python scripts/link_pyqt.py --tox /home/florian/proj/qutebrowser/git/.tox/pyflakes
pyflakes runtests: commands[1] | /home/florian/proj/qutebrowser/git/.tox/pyflakes/bin/python -m py.test -q --flakes -m flakes
sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss
=========================================================== 1554 tests deselected by "-m 'flakes'" ===========================================================
177 skipped, 1554 deselected in 2.49 seconds
pep8 inst-nodeps: /home/florian/proj/qutebrowser/git/.tox/dist/qutebrowser-0.2.1.zip
pep8 runtests: PYTHONHASHSEED='678912664'
pep8 runtests: commands[0] | /home/florian/proj/qutebrowser/git/.tox/pep8/bin/python scripts/link_pyqt.py --tox /home/florian/proj/qutebrowser/git/.tox/pep8
pep8 runtests: commands[1] | /home/florian/proj/qutebrowser/git/.tox/pep8/bin/python -m py.test -q --pep8 -m pep8
sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss
============================================================ 1554 tests deselected by "-m 'pep8'" ============================================================
177 skipped, 1554 deselected in 2.55 seconds
mccabe inst-nodeps: /home/florian/proj/qutebrowser/git/.tox/dist/qutebrowser-0.2.1.zip
mccabe runtests: PYTHONHASHSEED='678912664'
mccabe runtests: commands[0] | /home/florian/proj/qutebrowser/git/.tox/mccabe/bin/python scripts/link_pyqt.py --tox /home/florian/proj/qutebrowser/git/.tox/mccabe
mccabe runtests: commands[1] | /home/florian/proj/qutebrowser/git/.tox/mccabe/bin/python -m py.test -q --mccabe -m mccabe
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss
=========================================================== 1554 tests deselected by "-m 'mccabe'" ===========================================================
178 skipped, 1554 deselected in 2.52 seconds
pylint inst-nodeps: /home/florian/proj/qutebrowser/git/.tox/dist/qutebrowser-0.2.1.zip
pylint runtests: PYTHONHASHSEED='678912664'
pylint runtests: commands[0] | /home/florian/proj/qutebrowser/git/.tox/pylint/bin/python scripts/link_pyqt.py --tox /home/florian/proj/qutebrowser/git/.tox/pylint
pylint runtests: commands[1] | /home/florian/proj/qutebrowser/git/.tox/pylint/bin/pylint scripts qutebrowser --rcfile=.pylintrc --output-format=colorized --reports=no --expected-line-ending-format=LF
pylint runtests: commands[2] | /home/florian/proj/qutebrowser/git/.tox/pylint/bin/python scripts/run_pylint_on_tests.py --rcfile=.pylintrc --output-format=colorized --reports=no --expected-line-ending-format=LF
pyroma inst-nodeps: /home/florian/proj/qutebrowser/git/.tox/dist/qutebrowser-0.2.1.zip
pyroma runtests: PYTHONHASHSEED='678912664'
pyroma runtests: commands[0] | /home/florian/proj/qutebrowser/git/.tox/pyroma/bin/python scripts/link_pyqt.py --tox /home/florian/proj/qutebrowser/git/.tox/pyroma
pyroma runtests: commands[1] | /home/florian/proj/qutebrowser/git/.tox/pyroma/bin/pyroma .
------------------------------
Checking .
Found qutebrowser
<string>:47: (WARNING/2) Bullet list ends without a blank line; unexpected unindent.
<string>:80: (WARNING/2) Bullet list ends without a blank line; unexpected unindent.
<string>:95: (WARNING/2) Bullet list ends without a blank line; unexpected unindent.
<string>:185: (WARNING/2) Bullet list ends without a blank line; unexpected unindent.
<string>:223: (WARNING/2) Bullet list ends without a blank line; unexpected unindent.
------------------------------
Final rating: 10/10
Your cheese is so fresh most people think it's a cream: Mascarpone
------------------------------
check-manifest inst-nodeps: /home/florian/proj/qutebrowser/git/.tox/dist/qutebrowser-0.2.1.zip
check-manifest runtests: PYTHONHASHSEED='678912664'
check-manifest runtests: commands[0] | /home/florian/proj/qutebrowser/git/.tox/check-manifest/bin/python scripts/link_pyqt.py --tox /home/florian/proj/qutebrowser/git/.tox/check-manifest
check-manifest runtests: commands[1] | /home/florian/proj/qutebrowser/git/.tox/check-manifest/bin/check-manifest --ignore qutebrowser/git-commit-id,qutebrowser/html/doc,qutebrowser/html/doc/*,*/__pycache__
lists of files in version control and sdist match

How do you think the UI for this should look? Just a -q/--quiet option on the commandline?

If nobody else steps up, I might have time next week to submit a PR.

[bittner]

I would need to have a --quiet option (or something similar) for tox.

This is because I have a use case that runs tox as part of a test. I would need to have no output at all if everything goes well, and full diagnostic output if tox fails.

[ssbarnea]

Any updates on this? Somehow I find weird the fact that adding install_command = pip install -q {opts} {packages} does not make the output less verbose.

[obestwalter]

Hi @ssbarnea a PR implementing this would very likely be merged, but nobody is working on this.

[ssbarnea]

@obestwalter I can understand why this --quiet was not implemented as nobody raised a PR. Still, I don't understand why adding -q to install command does not address the same problem.

I have few projects that do have like 6-20 tox targets and most of them are re-used in order speedup testing but for each of them I am forced to see the result of pip freeze. I think that 80% of the console output is taken only by the installed redundant output.

[obestwalter]

Good point. This might actually be much easier to fix then ans should be treated differently. Could you open a new issue outlining this? Thanks.

[ssbarnea]

Added as #601

[nedbat]

@gaborbernat I don't think this issue should have been closed. I'm not sure exactly what #601 does, but it doesn't address the original request, which was to have tox be less verbose in general. Unless I'm misunderstanding something...