Tensorboard: version 1.5.0 install failure in win10+pyhon3.6.1

[gdyxml2000]

version 1.5.0 install failure in win10+pyhon3.6.1,version 0.4 is normal。
version 1.5.0 install error info:
Unknown requires Python '>=2.6, <3' but the running Python is 3.6.1

that led me to install tensorflow1.5 also failed！！

[manselmi]

Expanding on https://github.com/tensorflow/tensorboard/issues/916#issue-292172148 with some more detail.

There appears to be multiple problems:

1. It appears that setup.py's intention is to not require futures for Python < 3.2, but it's still being pulled in as a requirement when using installing from tensorflow_tensorboard-1.5.0-py3-none-any.whl, even when using Python >= 3.2 (in my case, Python 3.6.3). Unzipping that wheel and looking at tensorflow_tensorboard-1.5.0.dist-info/METADATA, you can see that when installing from this wheel, futures >= 3.1.1 is required, regardless of the Python 3 minor version being used.

   [fortran-favors-the-old@RHEL7-host] ~ % grep -e futures tensorflow_tensorboard-1.5.0.dist-info/METADATA
   Requires-Dist: futures (>=3.1.1)
   

2. futures 3.2.0 (the latest version) is incompatible with Python 3. Since setup.py requires futures >= 3.1.1, this makes successfully installing the tensorflow-tensorboard wheel non-deterministic: sometimes pip install tensorflow-tensorboard==1.5.0 fetches futures 3.1.1 and installation succeeds (version 3.1.1 doesn't blacklist Python 3), and sometimes the same command fetches futures 3.2.0 and installation fails (see code block at the end of this comment).

3. I see that Travis builds are currently disabled for Python 3, but noticed that futures is pinned to version 3.1.1 in .travis.yml. Because of this, when Python 3 Travis builds are re-enabled, I imagine that the futures 3.2.0+ issue I just described would not be detected. (I didn't look at the Travis build code too closely, but that's my suspicion.)

As https://github.com/tensorflow/tensorboard/issues/916#issue-292172148 alludes to, and as the code block at the end of this comment shows, items 1 and 2 above can break installation of tensorflow-tensorboard and tensorflow, which IMO is a critical issue.

Here's what happens when I attempt to install tensorflow-tensorboard==1.5.0 on a RHEL 7 host with Python 3.6.3:

[fortran-favors-the-old@RHEL7-host] ~ % /opt/rh/rh-python36/root/usr/bin/python -m venv pyvenv
[fortran-favors-the-old@RHEL7-host] ~ % source pyvenv/bin/activate
(pyvenv) [fortran-favors-the-old@RHEL7-host] ~ % python --version
Python 3.6.3
(pyvenv) [fortran-favors-the-old@RHEL7-host] ~ % pip install --upgrade pip setuptools
…
(pyvenv) [fortran-favors-the-old@RHEL7-host] ~ % pip freeze --all
pip==9.0.1
setuptools==38.4.0
(pyvenv) [fortran-favors-the-old@RHEL7-host] ~ % pip install tensorflow-tensorboard==1.5.0
Collecting tensorflow-tensorboard==1.5.0
  Downloading https://internal-nexus-server.tld/repository/pypi-proxy/packages/43/69/82e2a368076c94edbba3cd15804103bf1f31486d69e11551b71fa1d1f384/tensorflow_tensorboard-1.5.0-py3-none-any.whl (3.0MB)
    100% |████████████████████████████████| 3.0MB 308kB/s
…
Collecting futures>=3.1.1 (from tensorflow-tensorboard==1.5.0)
  Downloading https://internal-nexus-server.tld/repository/pypi-proxy/packages/1f/9e/7b2ff7e965fc654592269f2906ade1c7d705f1bf25b7d469fa153f7d19eb/futures-3.2.0.tar.gz
futures requires Python '>=2.6, <3' but the running Python is 3.6.3

[manselmi]

You can fix item 1 above by updating setuptools to the latest version before running setup.py. setuptools 36.2.0 fixed a bug that's generating incorrect requirements metadata in your wheel.

Here's an example illustrating the problem:

Example project

% cat > setup.py << 'EOF'
import setuptools

setuptools.setup(
    name='foo',
    version='1.0.0',
    install_requires=[
        'futures >= 3.1.1; python_version < "3"',
    ],
)
EOF


With broken setuptools

% rm -fr foo
% /opt/rh/rh-python36/root/usr/bin/python -m venv foo
% source foo/bin/activate
% python --version
Python 3.6.3
% pip install --upgrade pip setuptools==36.1.1 wheel
% pip freeze --all
pip==9.0.1
setuptools==36.1.1
wheel==0.30.0
% rm -fr build dist foo-1.0.0.dist-info foo.egg-info
% python setup.py bdist_wheel
% deactivate
% unzip dist/foo-1.0.0-py3-none-any.whl
% grep -e '^Requires-Dist:' foo-1.0.0.dist-info/METADATA
Requires-Dist: futures (>=3.1.1)


With fixed setuptools

% rm -fr foo
% /opt/rh/rh-python36/root/usr/bin/python -m venv foo
% source foo/bin/activate
% python --version
Python 3.6.3
% pip install --upgrade pip setuptools==36.2.0 wheel
% pip freeze --all
pip==9.0.1
setuptools==36.2.0
wheel==0.30.0
% rm -fr build dist foo-1.0.0.dist-info foo.egg-info
% python setup.py bdist_wheel
% deactivate
% unzip dist/foo-1.0.0-py3-none-any.whl
% grep -e '^Requires-Dist:' foo-1.0.0.dist-info/METADATA
Requires-Dist: futures (>=3.1.1); python_version < "3"

To fix item 2, I think you'll need to adjust the condition for requiring futures, since futures >= 3.1.1 (specifically, futures == 3.2.0) isn't compatible with Python 3 at all. If you don't mind dropping support for Python 3.0 and 3.1, you can replace line 36 of setup.py with 'futures >= 3.1.1; python_version < "3"',. Then you'd probably also want to pass python_requires='>= 2.7, != 3.0.*, != 3.1.*' to the setup function.

[nfelt]

Thank you for your in-depth report, I will take a look.

[jart]

Note that TensorFlow Python support matrix is officially Python 2.7 and Python 3.4+. If TensorBoard worked on Python 3.[01] with the setup.py fix I'd be surprised.

[manselmi]

Great, in that case it sounds like fixing this will be more straightforward.

[nfelt]

@fortran-favors-the-old What is your pip version? It looks like problem 2 that you identified is addressed in the most recent versions of pip (9.0.0+) by the PEP 503 support here:
https://github.com/pypa/pip/commit/c2101982cb5b28963fe22792cbb30ce50988ab8d

Namely, the newer pip should not try to download and install a version of a package known to be incompatible with the current python version if other versions of that package exist that are compatible. On my machine with pip 9.0.1 this manifests itself as reliably choosing futures 3.1.1 over 3.2.0.

To be clear, I'm not disputing that we should still ensure this works for older pip versions and that issue 1 is a problem in its own right, but just confirming my understanding of the issue.

[nfelt]

Ah, I just saw in your original message that you appear to also be using pip 9.0.1. In that case it's a bit strange you're seeing this - maybe your internal proxy could be affecting this behavior?

Could you try re-running the commands that you did in the "Here's what happens when I attempt to install tensorflow-tensorboard==1.5.0 on a RHEL 7 host with Python 3.6.3" section but use pip install -vvv tensorflow-tensorboard==1.5.0 instead?

My pip shows the following message and I'm a bit surprised yours doesn't:

Skipping link https://pypi.python.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl#md5=cfd62ab6c9852b04bb6048480fefaa75 (from https://pypi.python.org/simple/futures/) (requires-python:>=2.6, <3); it is not compatible with this Python

[manselmi]

Hey @nfelt , thanks for looking at this.

I won’t have access to my computer until the end of the day, so I’ll show you the output I get from running your suggested command later today.

That said, I’m not surprised by your output above. That wheel has Requires-Python metadata correctly populated, which pip picks up on without needing to download the wheel (it’s probably doing this for me too; I’m guessing I would have seen that too if I had passed verbose flags like you).

I suspect that in your case, assuming futures installation eventually succeeds for you, that pip next tries to install the 3.1.1 sdist and succeeds, since 3.1.1 is compatible with Python 3.

In my case, for whatever reason, pip attempts to install the 3.2.0 sdist before the 3.1.1 sdist. Since it’s an sdist, I don’t think pip can see any Python-Requires metadata ahead of time (this is something I can verify later) [EDIT: that was completely wrong], so it attempts installation from the sdist, which eventually fails due to the blacklisting of Python 3 in setup.py . When package installation fails at this point, pip errors out and doesn’t continue searching for the 3.1.1 sdist.

Unfortunately, it appears that PEP 440 only recommends that the latest version of a package be preferred, and doesn’t require it (at least that’s how I interpret the excerpt below):

When multiple candidate versions match a version specifier, the preferred version SHOULD be the latest version as determined by the consistent ordering defined by the standard Version scheme.

The key word there being “SHOULD” (instead of “MUST”). So as annoying as this inconsistency is, it appears it’s not a bug since PEP 440 allows for it, and IMO explains the behaviors that you and I are seeing. [EDIT: despite PEP 440 leaving this ambiguous, pip does the right thing and will attempt to install the most recent candidate version]

(On my Mac, on which pip is configured to use PyPI as the package index, futures 3.1.1 is installed every time. But on my RHEL7 host, on which pip is configured to use a Nexus PyPI mirror as the package index, futures 3.2.0 is always attempted—and fails—every time. Sigh.)

Fortunately, I don’t think any of this will end up mattering for the purposes of this bug report, since the fixes I suggested in a previous comment should ensure that pip will never attempt to install an incompatible version of futures, with one modification. Since Python 3.[0-3] is not supported (thanks @jart !), my python_requires suggestion should be changed to something like python_requires='>= 2.7, != 3.0.*, != 3.1.*, != 3.2.*, != 3.3.*'.

[manselmi]

About 30 minutes ago I accidentally submitted my previous comment halfway though writing it. It’s since been updated. Sorry for any confusion that may have caused.

[manselmi]

@nfelt : Following up on the request you made yesterday:

When PyPI is the package index, Python-Requires metadata is available even for sdists (my initial guess about this was thankfully incorrect), and pip knows to skip the futures 3.2.0 sdist:

The package https://pypi.python.org/packages/1f/9e/7b2ff7e965fc654592269f2906ade1c7d705f1bf25b7d469fa153f7d19eb/futures-3.2.0.tar.gz#md5=d1b299a06b96ccb59f70324716dc0016 (from https://pypi.python.org/simple/futures/) (requires-python:>=2.6, <3) is incompatible with the pythonversion in use. Acceptable python versions are:>=2.6, <3

However, our internal Nexus mirror does not expose this metadata (bug!), so it isn't skipped. pip then prefers 3.2.0 over 3.1.1 since it's the most recent version:

Using version 3.2.0 (newest of versions: 3.1.1, 3.2.0)

Thanks for submitting a pull request!

[pquentin]

Another data point for what it's worth: we have the same issue and we also use a PyPI mirror (devpi). Thanks @fortran-favors-the-old and @nfelt!

[manselmi]

@pquentin : FYI I noticed the same regarding devpi and plan on filing a bug report with them later today.

[nfelt]

Okay, good to know we were able to track down that discrepancy as well - thanks for following up!

The PR to fix the futures dependency to be python2-only has been submitted, and so it should go out in a 1.5.1 bugfix release fairly soon (I anticipate probably no later than end-of-week).

[manselmi]

Great, thanks for the quick fix @nfelt !

[nfelt]

Closing this out, since the fix should now be in the 1.5.1 release: https://github.com/tensorflow/tensorboard/releases/tag/1.5.1

[pquentin]

Thanks! This fixes the issue on our end.