Products.cmfplone: PLIP: TOX as new standard test invocation tool in Plone and Plone guidelines

[loechel]

PLIP (Plone Improvement Proposal)

Responsible Persons

Proposer: Alexander Loechel

Seconder: _wanted_

Abstract

The Content Management System Plone and its stack is a large set of source code. We the Plone community believe in quality, security and maintenance. Tests are one primary requirement for maintain that existing code (legacy).

With the move to Python 3 and probably multiple Minor Versions to support, I think it is necessary to move to the standard local Test management tool in the Python world: tox

Motivation

During porting of several Zope packages for Python 3, I did see that tox is at the moment the best tool to invoke multiple tests on different Python versions at the same time. Also tox does harmonize with travis-ci, jenkins and other CI tools. Tox seems to be the de facto standard at the moment in Python for testing multiple environments and versions at a time.

During work on some packages in the Plone continuum I did get the response that it needs a PLIP to discuss the move to tox.

Assumptions

Moving to tox as the recommended tool for test invokation means that we need to have good documentation about it, a discussion how that should look like and a prototype / working branch of bobtemplates.plone or cookiecutter template, older Tools like ZopeSkel and Paster maybe not.

Proposal & Implementation

Move to tox as the preferred tool for local test invocation. Therefor I would document how tox for Plone and Zope Packages work, what are the most important steps and how it would improve also our package layout and setup.

I would also implement a prove of concept template for bobtemplates.plone

Deliverables

• Documentation (--> https://github.com/plone/plone_best_practices_discussion)
• bobtemplates.plone tox.ini template and maybe .travis-ci.yml adaptations

Risks

This PLIP itself has nothing to do with Plone itself, so no risk for the Product Plone. It is about the procedures we use to develop Plone and Plone addons.

Participants

[tisto]

@loechel I am +1 for exploring tox. It is kind of a standard in the Python world and indeed a requirement when working with multiple Python versions at the same time (at least if you want to run tests against multiple versions locally).

Though, there is always some kind of overlap with the CI system (e.g. Travis, Jenkins) in functionality which is not easy to overcome. You want to be able to run the tests locally against multiple versions and at the same time use the scalability capabilities of a CI system (Pipelines, running steps in parallel, reporting, etc.). At the same time, it is essential that the command that you run locally produces the same output as the command that you run on the CI. Otherwise, devs will have a hard time when the two produce different results.

Anyways, I just wanted to mention that there are many things to take into consideration which we might be able to tackle if we start looking into what tox can offer us...

[gforcada]

@loechel should this one be done after (or together with) #2001 (i.e. get rid of bootstrap)?

[loechel]

@gforcada technically it is independent of #2001, but could help to get rid of bootstrap.py

I would suggest doing that together would make a lot of sense. I will work next week (During travel for Plone Midsummer Sprint) on that PLIP.

[stevepiercy]

We use tox for running tests, generating projects from scaffolds, building documentation, and validating much of PEP8 across Python 2.7, 3.4, 3.5, 3.6, and 3.7 (allowing for failures, but still testing it), and PyPy. Developers can install multiple versions of Python locally using a tool like pyenv. We use this for Pyramid, WebOb, and other projects under the Pylons Project.

Example configurations and ideas are in the Pyramid repo and its tox.ini, hacking-tox.ini, and .travis.yml with some instructions in HACKING.txt.

We ask that contributors use tox before submitting a pull request, which saves the core contributors from reviewing contributions that would fail in Travis-CI and Jenkins.

We also test on Windows via appveyor.yml.

[loechel]

During Plone Midsummer Sprint I have started to discuss and write some documentation, that first life in https://github.com/plone/plone_best_practices_discussion and should be moved to docs or training after discussion.

Implementation of skeleton for that is postponed, as bobtemplates.plone are not in shape at the moment to do so. I will wait till @MrTango has finished his https://github.com/plone/bobtemplates.plone/tree/MrTango_subcommands work and that got pulled.

[MrTango]

I think many of the ideas are implemented in bobtemplates.plone meanwhile.
It still needs some love and people testing and using it, but in general I'm all in finishing this soon.
Tox and Travis & co can work fine together, i don't see any problem here. We have this for bobtemplate.plone for quite a while now and also pushed it to the generated packages. This is the place which still needs some love, to sync code-analysis and tox setup. With tox triggering the linting i don't see much use of code-analysis anymore. Correct me if I'm wronge there.

[MrTango]

WE discussed this in the FWT. For the coredev buildout we will keep it as it is, even though there is some tox config too which some people use. But integrating jenkins and the tox setup seems too much work right now. For Add-on's though we fully support the idea and as it is already in bobtemplates.plone we are almost there. It just needs some improvements. I'll close the PLIP, as the addon's are not part of the core and don't need to be acepted by the framework team.