Conda-build: noarch packages are incompatible with pinned Python versions

[meawoppl]

When you specify the Python version in the meta.yaml file, this causes noarch packages to fail to build.

Here is a demo of the behavior I am seeing:
https://github.com/3Scan/conda-noarch-package-bug/tree/master

Here is the associated traceback:

meawoppl@lena:~/repos/3scan/conda-noarch-package-bug$ git pull origin master
remote: Counting objects: 3, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (3/3), done.
From github.com:3Scan/conda-noarch-package-bug
 * branch            master     -> FETCH_HEAD
   c774d64..5776e8a  master     -> origin/master
Updating c774d64..5776e8a
Fast-forward
 README.md | 2 ++
 1 file changed, 2 insertions(+)
 create mode 100644 README.md
meawoppl@lena:~/repos/3scan/conda-noarch-package-bug$ conda build .
Removing old build environment
Removing old work directory
BUILD START: 3scan-shared-0.0.1-py34_1
Fetching package metadata: ........
Solving package specifications: .
The following NEW packages will be INSTALLED:

    openssl:    1.0.1k-1     
    patchelf:   0.6-0        
    pip:        7.1.0-py34_1 
    python:     3.4.0-0      
    readline:   6.2-2        
    setuptools: 18.0.1-py34_0
    sqlite:     3.8.4.1-1    
    system:     5.8-2        
    tk:         8.5.18-0     
    wheel:      0.24.0-py34_0
    zlib:       1.2.8-0      

Linking packages ...
[      COMPLETE      ]|###################################################| 100%
Removing old work directory
Copying /home/meawoppl/repos/3scan/conda-noarch-package-bug to /home/meawoppl/anaconda3/conda-bld/work
Package: 3scan-shared-0.0.1-py34_1
source tree in: /home/meawoppl/anaconda3/conda-bld/work
+ python setup.py install
running install
running bdist_egg
running egg_info
creating BugDemo.egg-info
writing top-level names to BugDemo.egg-info/top_level.txt
writing dependency_links to BugDemo.egg-info/dependency_links.txt
writing BugDemo.egg-info/PKG-INFO
writing manifest file 'BugDemo.egg-info/SOURCES.txt'
reading manifest file 'BugDemo.egg-info/SOURCES.txt'
writing manifest file 'BugDemo.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib
creating build/lib/psuedopackage
copying psuedopackage/__init__.py -> build/lib/psuedopackage
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/psuedopackage
copying build/lib/psuedopackage/__init__.py -> build/bdist.linux-x86_64/egg/psuedopackage
byte-compiling build/bdist.linux-x86_64/egg/psuedopackage/__init__.py to __init__.cpython-34.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying BugDemo.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying BugDemo.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying BugDemo.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying BugDemo.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating dist
creating 'dist/BugDemo-0.0.1-py3.4.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing BugDemo-0.0.1-py3.4.egg
Copying BugDemo-0.0.1-py3.4.egg to /home/meawoppl/anaconda3/envs/_build/lib/python3.4/site-packages
Adding BugDemo 0.0.1 to easy-install.pth file

Installed /home/meawoppl/anaconda3/envs/_build/lib/python3.4/site-packages/BugDemo-0.0.1-py3.4.egg
Processing dependencies for BugDemo==0.0.1
Finished processing dependencies for BugDemo==0.0.1
found egg: /home/meawoppl/anaconda3/envs/_build/lib/python3.4/site-packages/BugDemo-0.0.1-py3.4.egg
number of files: 2
Fixing permissions
Dist: 3scan-shared-0.0.1-py34_1
An unexpected error has occurred, please consider sending the
following traceback to the conda GitHub issue tracker at:

    https://github.com/conda/conda-build/issues

Include the output of the command 'conda info' in your report.


Traceback (most recent call last):
  File "/home/meawoppl/anaconda3/bin/conda-build", line 5, in <module>
    sys.exit(main())
  File "/home/meawoppl/anaconda3/lib/python3.4/site-packages/conda_build/main_build.py", line 173, in main
    args_func(args, p)
  File "/home/meawoppl/anaconda3/lib/python3.4/site-packages/conda_build/main_build.py", line 400, in args_func
    args.func(args, p)
  File "/home/meawoppl/anaconda3/lib/python3.4/site-packages/conda_build/main_build.py", line 358, in execute
    channel_urls=channel_urls, override_channels=args.override_channels)
  File "/home/meawoppl/anaconda3/lib/python3.4/site-packages/conda_build/build.py", line 433, in build
    noarch_python.transform(m, sorted(files2 - files1))
  File "/home/meawoppl/anaconda3/lib/python3.4/site-packages/conda_build/noarch_python.py", line 68, in transform
    assert 'py_' in m.dist()
AssertionError

The assertion:
assert 'py_' in m.dist()
has something to so with the package naming. In my assessment, the normal naming is something like package-version_py32_0 or some such for which noarch packages seem to expect a non-existent version number such as package-version_py_0

Possible workaround is to include "py_" in the package build-string.

[ilanschnell]

The problem is that when you specify a Python version in meta.yaml is it not a python_noarch package anymore (at least not in it's original sense). python_noarch packages are both platform _and_ Python version number independent, which is why the build string for a Python noarch package has py_ in it, instead of say py34_ for a normal Python package which depends on Python 3.4.

[meawoppl]

Hmmm, the term "arch" here seems a bit overloaded. I guess I thought that the platform-bits convention was what was referred to here as the "architecture" given that was the folder delineations chosen vs the noarch folder.

It is also a bit confusing that I can make a noarch package that supports all versions of python explicitly (python >=0.0.0) and that also fails.

It all just seems really odd to me that a noarch package _must_ support all platforms and versions of Python, when the sat solver can be used to explicitly bracket the ones it does not. An example of how to achieve this currently, is simply making a parent package which calls out the python version, and noarch package together.

[sbourdeauducq]

+1 there are lots of packages that do not contain code for a particular architecture or OS, but only support certain Python versions. I would argue most packages actually won't work with e.g. Python 2.0. If noarch means both platform and Python version independent, it becomes rather useless.

[sbourdeauducq]

As a stopgap solution, can we just patch conda to remove the assert in our build system? Or will that cause later problems when installing the package?

[whitequark]

You can set the build number explicitly as a workaround:

build:
  noarch_python: true
  string: py_{{ environ.get("GIT_DESCRIBE_NUMBER", 0) }}

As far as I understand, conda install doesn't actually look at the build number when installing, only at dependencies, so no issues should arise. You could probably even do several builds for different Python versions, e.g.:

build:
  noarch_python: true
  string: py_actually{{ environ.get("PY_VER") }}_{{ environ.get("GIT_DESCRIBE_NUMBER", 0) }}

which should result in a build string like py_actually3.5_1271.

[meawoppl]

There are a slew of things that are possible in noarch packages that are somewhat disabled by the current config.

If you just remove the assert, there are also some subtle bugs that involve how setuptools puts its version stamp on packages, which conda won't link properly. For example, conda will install a package built with 3.5 (and stamped by setuptools as such), into a 3.4 env. The 3.4 env won't find the package b/c of setuptools .pth hackery. _sighs_

Notably these can still issued can still be evoked in certain situations but the assertion catches at least some of the obvious ones.

Our current workaround is to call out the version in a otherwise empty package which the package of interest depends on. Similarly, you must call out the same version in the build deps, lest possibly experience the setuptools glitch mentioned above. Slightly janky, but I can confirm this works!

I can post a demo repo of both this bug and solution if there is any desire for that.

[msarahan]

@meawoppl if you have time, please post your solution as a PR. I'd be very happy to review and merge it when we get it into the best shape possible. If you have test cases that capture the negative behavior, that would make it all easier.

[meawoppl]

Its not really a solution. Here is the short workaround version.
Package A - noarch pure python code.
Package B - linux64/osx64/win64 (no actual content to the package, just deps)

Dependencies:
A -> Python 3.4 (fails)
A -> B -> Python 3.4 (works)

[ChrisBarker-NOAA]

It seems clear that work-arounds or no -- a python-specific, but architecture non-specific pacakge would be really useful!

I can understand that platform-independent packages that have nothing to do with python make sense, but calling the tag: noarch_python -- really seems to be attempting so support python, and py2-3 differences are a ccore part of the python ecosystem for a while to go :-(

[ChrisBarker-NOAA]

Oh, and in the meantime, a helpful error message would be nice :-)

[meawoppl]

We have decided to stick to the workaround mentioned above. The assertion should at least be more descriptive IMHO.

[jakirkham]

Imagine the original issue here is fixed with noarch: python.