Marshmallow: RFC: Use tox for testing in travis and local dev?

[sloria]

Should we use tox to run tests?

Example: https://github.com/python-attrs/attrs/blob/master/.travis.yml

Pros:

• Better parity between local dev and travis
• Can run tests without having to set up an entire environment

Cons:

• Slightly more config in .travis.yml
• Some learning curve for those of us used to using pytest directly

Strong opinions welcome, esp. those of frequent contributors. @lafrech @deckar01 @taion

[lafrech]

I've been using tox in other projects. Generally, I run pytest on a regular basis, and sometimes tox if I want to check if my changes broke another environment (different python/marshmallow/whatever version). Or I forget and realize it is broken once it fails in CI. Just like flake8 tests.

tox cons:

Multiple environments. Can take space on disk if using huge libs (SciPy,...) but not an issue here. In practice not that much because the dev machine only has two Python versions.

Just like pre-commit cache, tox doesn't update the env everytime. It would be costly to recreate it from scratch. So, sometimes you get strange stuff happening. E.g. you remove a lib from setup.py but call it in the code, tox won't notice until the venv is recreated. (Edit: Not exactly. This is the issue I was referring to: https://github.com/tox-dev/tox/issues/149.)

No strong opinion. Would it replace invoke or would it be yet another layer?

I use tox in flask-rest-api: https://github.com/Nobatek/flask-rest-api/blob/master/.travis.yml.

[justanr]

If you're looking to do this, I'd recommend using the Travis plugin for tox. It'll autodetect the version of python running and run that testenv.

Something else that's useful for speeding up tox runs is separating out checks like flake8, black, mypy, etc. They don't need to run every time so dedicated envs for them is really useful.

flask-allows is setup this way and looks like attrs is as well.

It also can help with the coverage issue discussed in #1017

[sloria]

Another option is to keep the local dev environment the same (including the contributing docs) and only change the way tests are run on Travis. So we'd keep invoke for task running and only use tox for running tests against multiple versions.

Would it replace invoke or would it be yet another layer?

If we go with the above approach, we'd keep invoke. We actually already have a tox config, so it wouldn't really be another layer.

I'd recommend using the Travis plugin for tox.

I've used tox-travis in the past. I've found it to be a bit too magical in how it tries to guess the right TOX_ENV, and it doesn't save that much config. In a recent project I ended up switching from tox-travis back to vanilla tox.

Something else that's useful for speeding up tox runs is separating out checks like flake8, black, mypy, etc.

Agreed.

[taion]

I agree with the above. I use tox for my libraries, but do local testing with pytest unless I'm doing something where I think I'll need to check other environments.

I also haven't found it too onerous to just wire up tox and Travis without tox-travis.

[sloria]

What should we use for running tasks in local development?

👍 invoke (invoke test, invoke syntax, invoke docs)

❤️ Run commands manually (pytest, pre-commit run --all-files, cd docs && make html)

🎉 setup.py commands, a la flask-resty (python setup.py test, python setup.py lint, python setup.py docs)

😄 tox commands (tox -e ..., tox -e lint, tox -e docs)

[sloria]

My $0.02: I could do without invoke for running tests and linting. I do like having the invoke docs -w command while writing docs, though. But perhaps we can replace it either with a tox command (e.g. tox -e watch-docs) or a setup.py command using @taion's handy trick. I've updated the poll above with those options.

[taion]

Do any of the newer tools let you put scripts in pyproject.toml the way you can in package.json for Node projects? I think something like that would be the more principled version of my setup.py hack :p

[sloria]

@taion It looks like poetry supports something like npm scripts via poetry run: https://poetry.eustace.io/docs/cli/#run . The example shows resolving a Python import path ("my_module:main"), but according to https://github.com/sdispater/poetry/issues/99 , you can also run arbitrary scripts.

Switching to poetry would be a much bigger lift than the OP, though...

[taion]

Hah, yeah. Oops!

[sloria]

How about this:

• Use tox for running tests in travis
• Remove invoke
• Get rid of dev-requirements.txt. Use setup.py extras, like in attrs. Dev environment will be set up with pip install -e '.[dev]'
• Run tests locally with pytest. Use tox -e ... when necessary (e.g. reproducing a failure on travis).
• Docs can be built with tox -e docs and tox -e watch-docs.

I'll have a PR for this soon.