Conda-forge.github.io: Add more details on how to use package testing

[bulli92]

Following this discussion I suggest to generate a seaparte page in the documentation that is devoted to describe in more details the different options that are available to use already existing package tests when generating the package conda-forge recipe.

[ChrisBarker-NOAA]

I should upt this in a PR against the docs, but I'm short on time at the moment, so a few notes:

(OK, I spent more time than I expected to -- so here is a starting point)

There are many possible ways to test the built package. A more is better philosophy is a good one, but sometimes is it hard to run the tests, and/o they are time consuming, so you may not want to run everything -- but SOMETHING is critical.

For Python packages:

testing importing

The very minimal test should make sure that the package imports. This can be accomplished with this stanza in the yaml:

test:
imports:
- package_name

Note that that package_name is the name that is imported by python, not necessarily the name of the conda package (they are sometimes different)

testing for an import will catch the bulk of the packaging errors, generally including presence of dependencies. However, it does not assure that the pacakge works correctly articularly if it works correctly with the particular versions of dependencies used.

running unit tests

The trick here is that there are multiple ways to run unit tests in Python, including nose, pytest, etc.

Also, some packages install the tests with the package, and thus they can be run in place. while others keep the tests with the source code, and thus can not be run straight from an installed package.

test requirements

sometimes there are pacakges required to run the tests tha are not required to simiply use teh package. This is usually a test-running framework, such as nose or pytest. YOu can ensure that it is included by adding it to the test stanza:

test:
  imports
    - package_name 
...
  requires:
    - pytest

built-in tests

Some packages have testing built-in. In this case, you can put a test command directly in the test stanza:

test:
  ...
  commands:
     python -c "import package_name; package_name.tests.runall()"

Alternatively, you can add a file called run_test.py in the recipe that will be run at test time. This allows an arbitrarily complicated test script.

pytest tests:

if the tests are installed with the package, pytest can find and run them for you with the following command:

test:
  requires:
    - pytest
  commands:
    - py.test --pyargs package_name 

nose tests

Someone please add these.

command line utilities

If a package installs command line utilities, you probably want to test that they were properly installed:

test:
  commands:
    - util_1 --help

If the utility actually has a test mode, great, otherwise simply invoking "--help" or "--version" or something will at least test that it is installed and can run.

tests outside of package

Note that conda-build runs the tests in an isolated environment after installing the package -- thus, at this point it does not have access to the original source tarball. This makes it very hard to run tests that are not installed with the package.

NOTE if anyone has good ideas as to how to do that, please put it here!

non python packages

If a package isn't python, it is all the more important to test. At least try to test that it installs and can be run -- some libraries come bundles with command-line tools -- running those is a good way to assure that the lib is installed.

[jakirkham]

This looks great @ChrisBarker-NOAA! Any thoughts on doing a PR of these notes to the docs? 😄

[bulli92]

Yes, looks very neat already. I spotted a few typos. As I have now an idea how it works, I could also fill in the nosetest section once you open a PR.

[ChrisBarker-NOAA]

Done -- see PR: #349