Tox: Log/Export environment details to machine-readable file

[danwos]

Proposal
Create a json/xml file for each used environment, which stores installed python packages and system information (os, cpu, ...)

A xml file could look like this:

<environments>
  <environment name="python2.7_sphinx1.6">
    <python version="2.7">
      <path>...</path>
      <packages>
         <package name="sphinx" version="1.6" />
        ...
     </packages>
     <system>
       <operating_system type="linux" name="ubuntu" />
       <cpu vendor="amd" ... />
       <memory size="8GB" />
     ...
  </environment>
  ...
</environments>

Motivation
I'm working for an automotive company and on my current project I have to archive test results for the next 20+ years. For test results most libraries support the junit export format, which makes it easy to reuse this information automatically in other tools like sphinx, word, ...
But beside test results I also need to document and archive the used test environment itself.
For instance it is very important to know, if tests have been run on the final target system and final os. Or if some "old" versions were used to get needed results.

Current situation
Currently this kind of information is spread around several log messages and most of them have different formats, which are not easily parsable.
You could also execute pip freeze and write own scripts to collect system information for each used environment.
Maybe also the usage of the hook tox.hookspecs.tox_runenvreport is an option.
However, all of this needs a bunch of time to develop own solutions for collecting already available information.

[gaborbernat]

This sounds like a great candidate for a plugin.

[vlcinsky]

Few comments:

• there are at least two environment situations: before commands are run and after that. The "before" seems to be more relevant (possible environment modifications by running commands will be probably determined by initial environment).
• regarding format: JSON would be my preference.
• regarding initial intentions for installed packages Pipfile would serve well.
• regarding really installed things, Pipfile.lock would be perfect report (incl. SHA for package files).
• regarding environmental variables, I would think of .env file as used e.g. in pipenv command (wold make it really easy to reuse later).

In pipenv issue regarding pipenv (anti)patterns is somewhere in the middle "Mode: Generate and Seal", serving you with Pipfile.lock.

Warning: when using pipenv, be careful. It may help, but one must understand, what is really happening, otherwise it is easy to mess up the test and cause more harm than good.

[danwos]

pipenv looks good, but I only get it running by manipulating my tox.ini and add pipenv related commands like:

• pip install pipenv
• pipenv install -r requirements.txt
• pipenv run py.test tests

And it also needs to create its own venv, because if it reuses the venv created by tox, it does not update Pipfile with already installed packages in this tox venv.
Also new installed packages via pip install ... are not recognized by pipenv.

My goal is to get such a function without any need for the user to reconfigure something.
So a pip install tox-magic-plugin should be enough and new files get created automatically in the related log-folder of each tox-env.
A Pipfile would be great, but I don't see any way to get it with the above requirements.
Please correct me, if I'm wrong.

Maybe there is a way to get a Pipfile from a venv without initiating und using pipenv.
But I haven't found one yet.

However, my current plan for such a plugin looks like this:

• pip install tox-magic-plugin
• plugin uses hook tox_runtest_pre to gather following information
  
  
  
  • system: os, cpu, ...
  
  
  • python: installed packages (name, version, hash)
  
  
• plugin creates env_report.json inside log folder.

It's quite simple and as tox already provides the hook tox_runenvreport, it does information gathering already. Only storing it to a file is missing.
So to make it even easier for the user: Why not making it available in tox itself?

[obestwalter]

I don‘t know much about pipenv yet, but I am pretty sure that it will cause a lot of pain in conjunction with vanilla tox. If you wanna go down that road you will have to throw https://pypi.org/project/tox-pipenv/ into the mix.

[vlcinsky]

@danwos I guess, proposed plugin would serve well.

My comment is only reminder, there are already existing formats for information you are going to record somewhere. Using existing formats could simplify further reuse (e.g. having Pipfile.lock would allow recreating exact environment as used during test run).

Personally I have rather reserved position to using pipenv with tox (at least it slows things down; when used wrong, it may hide problems in setup.py and far too often the purpose of using it with tox is simply not clear).

Anyway, I have few ideas and corrections related to your comments:

• installing pipenv: pip is already assumed as part of equipment of tox. pipenv may be alternative one.
• tox theoretically provides means (install_command setting) to use alternative commands for installing python packages, pipenv could be one (even though my quick experiments failed in this regard).
• when pipenv runs under virtualenv of tox:
  
  
  
  • it manipulates current virtualenv and does not create private one
  
  
  • for that reason, there is no need to use e.g. pipenv run pytest, one can directly use pytest.
  
  
  • Pipfile.lock file is created (but currently lives in root of the project: to prevent overwriting by next environment, it has to be moved somewhere, e.g. to .tox/py36 directory).
  
  

Anyway, I completely agree with @obestwalter that (at least currently) using pipenv with vanilla tox could cause a lot of pain.

Regading tox-pipenv, I have currently no idea, how to use it as described in my tox-pipenv issue

If you go for plugin creating env_report.json, I think you would have relatively quickly your requirements fulfilled. I would still prefer creating such report using formats, which would allow their practical usage by existing tool. But my real need for such solution is not very strong, so do not take it as "practice really requires my approach".

[danwos]

Ok, I spend some time to create a first alpha version of a plugin and here we go:
github: https://github.com/useblocks/tox-envreport
readthedocs: http://tox-envreport.readthedocs.io/en/latest/
pipy: https://pypi.org/project/tox-envreport/

It is far away from being stable, bug-free or really helpful.

I tried hard to find a solution for getting a Pipfile. But every code for creating such a file is hardly coupled with pipenv and the spec is also not finalized. So currently I would like to avoid spending time on creating such a Pipfile-creation lib.

However, the current implementation is quite small, because tox internally stores already a lot of information about its venvs. I only needed to jsonifiy the data and store it somewhere.

So currently you get a env_report.json file inside .tox after each run. Example of such a file.
All you need to do is to install the plugin via pip install tox-envreport.

Question: Is there any hook, which gets called only once for a complete tox run? (e.g. tox_runall_post).
Currently my plugin is recreating the file for each single test run and hooking into tox_configure is too early as no venv was created yet.

Next steps (if I find some time):

• Adding more system information (cpu, os name and version, ...)
• Adding sha hashes to pip information (Related to Pipfile.lock)
• Trying to push json-structure more into direction of Pipfile and co.

Beside this plugin, I'm also planning to create a sphinx-extension for documenting test-results( junit format) and test envs (tox-envreport format) inside a sphinx documentation.
So if you have any ideas/wishes on this, drop me a line.

Thanks for all the help and awesome ideas!

[vlcinsky]

@danring it seems like Speedy Gonzales is your slower brother.

Pipfile shall be specified here but there is clear warning, that nothing is in stone. There is https://github.com/kennethreitz/pipenvlib but this is clearly WIP too.

Did you consider placing one json report per virtualenv directory? E.g. in .tox/py36/env_report.json? It is clearly product of running tests within given virtualenv and when using with detox, the report would be less prone to concurrency issue compared to all envs report in one file.

[danwos]

Yes, that's already implemented.
Each venv get its own specific env_report_xxx.json file in the related venv-folder. Where xxx is the name of the current venv.
(See the short documentation for details)

Sorry, I have forgotten to mention it in my prior post.

[vlcinsky]

@danring Good.

I searched for file location in your doc and ended at the 2nd paragraph as it seemed to answer my question. I had no idea there are two types of reports generated.

I would propose one of following:

• coin good terms for those two types of reports and mention them at the very beginning. Like toxenv and envenv report.
• skip creating the global report and keep it simple: just use the env specific one.

[danwos]

@vlcinsky Thanks for your feedback. You are right, 2 different names for the 2 kinds of files are really helpful. I changed the code and reorganized the documentation.

Skipping the global report is not a good idea for my use case.
A single file, which contains everything, can be easily archived and reused by other tools.
Having multiple files in different folders will make it hard to see the big picture.
For me having both types of files is perfect, as it supports so some many different use cases.

I close this tox-related issue now.
Feel free to add thoughts and bug reports to the issue tracker of tox-envreport.

And again, thanks for all the feedback and ideas. It was a pleasure to get a solution in less than 2 days :+1: