Pants: Unable to build/test a python 3.6.0 project using jsonschema lib due to pants bringing in a 2.7 dependency

[jessehersch]

Symptom: cannot use the jsonschema lib in a python project targeting 3.6.0. pants tries to bring in functools32 which raises an error on python versions greater than 2.7.

The reason it's bringing in this dependency appears to be this line in jsonschema's setup.py: https://github.com/Julian/jsonschema/blob/master/setup.py#L25

This dependency should not be brought in since the project is targeting 3.6.0.

I created a simple repro for this here: pants_bug.zip

To repro:

1. extract the zip above.
2. cd pants_bug
3. note the pants.ini specifies interpreter_requirement: CPython==3.6.0
4. attempt to run tests: ./pants test .:

Output:

09:15:45 00:00 [main]
               (To run a reporting server: ./pants server)
09:15:45 00:00   [setup]
09:15:45 00:00     [parse]fatal: Not a git repository (or any of the parent directories): .git

               Executing tasks in goals: bootstrap -> imports -> unpack-jars -> deferred-sources -> gen -> jvm-platform-validate -> resolve -> compile -> resources -> test
09:15:45 00:00   [bootstrap]
09:15:45 00:00     [substitute-aliased-targets]
09:15:45 00:00     [jar-dependency-management]
09:15:45 00:00     [bootstrap-jvm-tools]
09:15:45 00:00     [provide-tools-jar]
09:15:45 00:00   [imports]
09:15:45 00:00     [ivy-imports]
09:15:45 00:00   [unpack-jars]
09:15:45 00:00     [unpack-jars]
09:15:45 00:00   [deferred-sources]
09:15:45 00:00     [deferred-sources]
09:15:45 00:00   [gen]
09:15:45 00:00     [antlr-java]
09:15:45 00:00     [antlr-py]
09:15:45 00:00     [jaxb]
09:15:45 00:00     [protoc]
09:15:45 00:00     [ragel]
09:15:45 00:00     [thrift-java]
09:15:45 00:00     [thrift-py]
09:15:45 00:00     [wire]
09:15:45 00:00   [jvm-platform-validate]
09:15:45 00:00     [jvm-platform-validate]
09:15:45 00:00   [resolve]
09:15:45 00:00     [ivy]
09:15:45 00:00   [compile]
09:15:45 00:00     [compile-jvm-prep-command]
09:15:45 00:00       [jvm_prep_command]
09:15:45 00:00     [scalafmt]
09:15:45 00:00     [compile-prep-command]
09:15:45 00:00     [compile]
09:15:45 00:00     [zinc]
09:15:45 00:00     [jvm-dep-check]
09:15:45 00:00   [resources]
09:15:45 00:00     [prepare]
09:15:45 00:00     [services]
09:15:45 00:00   [test]
09:15:45 00:00     [test-jvm-prep-command]
09:15:45 00:00       [jvm_prep_command]
09:15:45 00:00     [test-prep-command]
09:15:45 00:00     [test]
09:15:45 00:00     [pytest]
09:15:45 00:00       [run]

09:15:45 00:00         [chroot]**** Failed to install functools32-3.2.3-2 (caused by: NonZeroExit("received exit code 1 during execution of `[u'/home/jhersch/.virtualenvs/gemini/bin/python3.6', '-', 'bdist_wheel', '--dist-dir=/tmp/tmpomPs2n']` while trying to execute `[u'/home/jhersch/.virtualenvs/gemini/bin/python3.6', '-', 'bdist_wheel', '--dist-dir=/tmp/tmpomPs2n']`",)
):
stdout:

stderr:
This backport is for Python 2.7 only.



09:15:46 00:01   [complete]
               FAILURE
Exception caught: (<class 'pex.resolver.Untranslateable'>)

Exception message: Package SourcePackage(u'file:///home/jhersch/pants_bug/.pants.d/python-setup/resolved_requirements/CPython-3.6.0/functools32-3.2.3-2.zip') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)

[kwlzn]

fwiw, pex seems to handle the extras resolve just fine on 2.7 vs 3.6:

2.7:

[omerta ~]$ pex --disable-cache jsonschema -o /tmp/throwaway.pex -vvvvvvvvvvvvv 2>&1 |grep "Translating" |grep -v "Building"
pex:     Translating /private/var/folders/bb/msvj_4_s741g93nzgrj19twc0000gn/T/tmpnjcWsf/jsonschema-2.6.0-py2.py3-none-any.whl into distribution: 2.7ms
pex:     Translating /private/var/folders/bb/msvj_4_s741g93nzgrj19twc0000gn/T/tmpzeyOVs/functools32-3.2.3-2.zip into distribution: 312.6ms
[omerta ~]$ /tmp/throwaway.pex
Python 2.7.13 (default, Mar  2 2017, 16:22:01)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import jsonschema
>>>

3.6:

[omerta ~]$ pex --disable-cache jsonschema -o /tmp/throwaway.pex -vvvvvvvvvvvvv 2>&1 |grep "Translating" |grep -v "Building"
pex:     Translating /private/var/folders/bb/msvj_4_s741g93nzgrj19twc0000gn/T/tmp0t56idwp/jsonschema-2.6.0-py2.py3-none-any.whl into distribution: 3.1ms
[omerta ~]$ /tmp/throwaway.pex
Python 3.6.0 (default, Feb 13 2017, 11:02:15)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import jsonschema
>>>

I think this points to some form of an interpreter mismatch during resolve.

[jessehersch]

We are having to use an old json schema library due to this which is no longer maintained. Any chance this could get some love?

[anikiforovopensource]

Hit the same issue with pants 1.5.0 trying to use plotly==2.5.1 which requires jsonschema.

[bjorn-spire]

same here with pants 1.7.0rc0 trying to use plotly==2.6.0

[bjorn-spire]

I ended up working around this by:

1. Create a local build target with provides=py_setup named functools32
2. Building it with ./pants setup-py <my target>
3. Converting it to a python3 wheel by pip3 wheel dist/functools32-<version>tar.gz
4. Uploading it to a private python "find links repo"

It works by the silly fact that there's something for it to download, and then it just places a silly file called functools32_hack.py explaining why it's there.

So if you get stuck on this then there's always this horrid hack until one of us have time to look into fixing it all proper like. :)

[shuoli84]

Should be fixed by #5906
Verified on 1.9.0.dev0.

[stuhood]

Thanks @shuoli84!