Pytorch-lightning: Install within conda env breaks

[dsuess]

Describe the bug
Installation inside conda environment fails on Mac and Linux with the following message

    ERROR: Command errored out with exit status 1:
     command: /Users/dsuess/Library/Conda/envs/test/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/ky/6gq9n6md77n0y3hsf37_b6qr0000gn/T/pip-install-8izjxcqp/pytorch-lightning/setup.py'"'"'; __file__='"'"'/private/var/folders/ky/6gq9n6md77n0y3hsf37_b6qr0000gn/T/pip-install-8izjxcqp/pytorch-lightning/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
         cwd: /private/var/folders/ky/6gq9n6md77n0y3hsf37_b6qr0000gn/T/pip-install-8izjxcqp/pytorch-lightning/
    Complete output (3 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'setuptools'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Running python -c 'import setuptools' does not fail, so setuptools is installed. Also, installing directly from github is fine. I think it's due to the site directory inside the tarball from pypi breaking python

To Reproduce

conda create --name test python=3
conda activate test
python -m pip install pytorch-lightning

Expected behavior
The package should be installed succesfully

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: MacOS 10.14.6 and Ubuntu 18.04
  
  
  
  • conda 4.7.11
  
  
  • pip 19.2.3
  
  

[williamFalcon]

@borda any idea?

[Borda]

I feel really strange about instaling package via pip in conda ecosystem...
see the section about instaling non-conda packages but it should be possible according to Using Pip to install packages to Anaconda Environment
Anyway, the error message complains about missing setuptools pls could you list local pip packages python -m pip list

[dsuess]

I thought conda would be supported since it's listed as an alternative to virtualenv in the README.md.

Listing packages yield:

> python -m pip list
Package    Version
---------- ---------
certifi    2019.6.16
pip        19.2.3
setuptools 41.2.0
wheel      0.33.6

So setuptools is there. Also installing pytorch-lighting via

pip install git+https://github.com/williamFalcon/pytorch-lightning

works fine. Finally, If I download the package manually from pypi using pip download and running python setup.py install manually in the extracted archive gives the same error. However, if I delete the site module from the directory, it installs fine. The leads to be believe that the package itself works fine with conda, there's just sth weird going on with the packaging.

[williamFalcon]

i use conda with lightning. to install in conda I just use:

pip install pytorch-lightning

that’s it.

[williamFalcon]

oh, i think i see it. lightning only supports python 3.6+ (it’s in the readme)

so, change your command from
python=3

to
python=3.6

[dsuess]

Hmm, it's python 3.7, see the output of conda env export:

  - bzip2=1.0.8=h01d97ff_0
  - ca-certificates=2019.6.16=hecc5488_0
  - certifi=2019.6.16=py37_1
  - libcxx=8.0.1=0
  - libcxxabi=8.0.1=0
  - libffi=3.2.1=h6de7cb9_1006
  - ncurses=6.1=h0a44026_1002
  - openssl=1.1.1c=h01d97ff_0
  - pip=19.2.3=py37_0
  - python=3.7.3=h93065d6_1
  - readline=8.0=hcfe32e1_0
  - setuptools=41.2.0=py37_0
  - sqlite=3.29.0=hb7d70f7_1
  - tk=8.6.9=h2573ce8_1002
  - wheel=0.33.6=py37_0
  - xz=5.2.4=h1de35cc_1001
  - zlib=1.2.11=h01d97ff_1005

[williamFalcon]

(base) user-mbp:~ user$ conda create --name test python=3
Collecting package metadata: - failed

CondaError: KeyboardInterrupt

(base) user-mbp:~ user$
(base) user-mbp:~ user$
(base) user-mbp:~ user$ conda create --name test2 python=3.7
Collecting package metadata: done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.6.14
  latest version: 4.7.11

Please update conda by running

    $ conda update -n base -c defaults conda



## Package Plan ##

  environment location: /Users/user/miniconda3/envs/test2

  added / updated specs:
    - python=3.7


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    pip-19.2.2                 |           py37_0         1.9 MB
    python-3.7.4               |       h359304d_1        21.6 MB
    ------------------------------------------------------------
                                           Total:        23.5 MB

The following NEW packages will be INSTALLED:

  ca-certificates    pkgs/main/osx-64::ca-certificates-2019.5.15-1
  certifi            pkgs/main/osx-64::certifi-2019.6.16-py37_1
  libcxx             pkgs/main/osx-64::libcxx-4.0.1-hcfea43d_1
  libcxxabi          pkgs/main/osx-64::libcxxabi-4.0.1-hcfea43d_1
  libedit            pkgs/main/osx-64::libedit-3.1.20181209-hb402a30_0
  libffi             pkgs/main/osx-64::libffi-3.2.1-h475c297_4
  ncurses            pkgs/main/osx-64::ncurses-6.1-h0a44026_1
  openssl            pkgs/main/osx-64::openssl-1.1.1c-h1de35cc_1
  pip                pkgs/main/osx-64::pip-19.2.2-py37_0
  python             pkgs/main/osx-64::python-3.7.4-h359304d_1
  readline           pkgs/main/osx-64::readline-7.0-h1de35cc_5
  setuptools         pkgs/main/osx-64::setuptools-41.0.1-py37_0
  sqlite             pkgs/main/osx-64::sqlite-3.29.0-ha441bb4_0
  tk                 pkgs/main/osx-64::tk-8.6.8-ha441bb4_0
  wheel              pkgs/main/osx-64::wheel-0.33.4-py37_0
  xz                 pkgs/main/osx-64::xz-5.2.4-h1de35cc_4
  zlib               pkgs/main/osx-64::zlib-1.2.11-h1de35cc_3


Proceed ([y]/n)? y


Downloading and Extracting Packages
python-3.7.4         | 21.6 MB   | ################################################################################################################################################################# | 100%
pip-19.2.2           | 1.9 MB    | ################################################################################################################################################################# | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate test2
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(base) user-mbp:~ user$ conda activate test2
(test2) user-mbp:~ user$ pip install pytorch-lightning
Collecting pytorch-lightning
  Downloading https://files.pythonhosted.org/packages/a7/f8/69614df76721d2e450b260d47e2dca8a2c3aea44353e9a41045762441e65/pytorch-lightning-0.4.8.tar.gz (50kB)
     |████████████████████████████████| 51kB 228kB/s
Collecting torch==1.2.0 (from pytorch-lightning)
  Using cached https://files.pythonhosted.org/packages/ba/88/7640344d841e97b9a1531385caac39d984b2c6f4abd1376e1ce0de3a0933/torch-1.2.0-cp37-none-macosx_10_7_x86_64.whl
Collecting tqdm>=4.35.0 (from pytorch-lightning)
  Using cached https://files.pythonhosted.org/packages/dc/88/d3213e2f3492daf09d8b41631ad6899f56db17ce83ea9c8a579902bafe5e/tqdm-4.35.0-py2.py3-none-any.whl
Collecting test-tube>=0.6.9 (from pytorch-lightning)
Collecting pandas>=0.20.3 (from pytorch-lightning)
  Downloading https://files.pythonhosted.org/packages/39/73/99aa822ee88cef5829607217c11bf24ecc1171ae5d49d5f780085f5da518/pandas-0.25.1-cp37-cp37m-macosx_10_9_x86_64.macosx_10_10_x86_64.whl (10.1MB)
     |████████████████████████████████| 10.1MB 3.3MB/s
Collecting numpy (from torch==1.2.0->pytorch-lightning)
  Downloading https://files.pythonhosted.org/packages/8d/4b/b6339340169862935ef5757db7e5869af7576f03148d069869edbd523ef2/numpy-1.17.1-cp37-cp37m-macosx_10_9_x86_64.whl (15.0MB)
     |████████████████████████████████| 15.0MB 1.1MB/s
Collecting tb-nightly==1.15.0a20190708 (from test-tube>=0.6.9->pytorch-lightning)
  Using cached https://files.pythonhosted.org/packages/b1/98/fa38deb0d26db3c88fc3c3267c622508648df9d67242d8a4d15c2f6bab08/tb_nightly-1.15.0a20190708-py3-none-any.whl
Collecting imageio>=2.3.0 (from test-tube>=0.6.9->pytorch-lightning)
  Using cached https://files.pythonhosted.org/packages/af/0a/943c965d372dae0b1f1482677d29030ab834351a61a9a632fd62f27f1523/imageio-2.5.0-py3-none-any.whl
Collecting future (from test-tube>=0.6.9->pytorch-lightning)
Collecting pytz>=2017.2 (from pandas>=0.20.3->pytorch-lightning)
  Using cached https://files.pythonhosted.org/packages/87/76/46d697698a143e05f77bec5a526bf4e56a0be61d63425b68f4ba553b51f2/pytz-2019.2-py2.py3-none-any.whl
Collecting python-dateutil>=2.6.1 (from pandas>=0.20.3->pytorch-lightning)
  Using cached https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl
Collecting six>=1.10.0 (from tb-nightly==1.15.0a20190708->test-tube>=0.6.9->pytorch-lightning)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting protobuf>=3.6.0 (from tb-nightly==1.15.0a20190708->test-tube>=0.6.9->pytorch-lightning)
  Using cached https://files.pythonhosted.org/packages/6f/67/c9461f62a46b083602276afe20b3a26c19608f8f86e9b2683dd1de001e88/protobuf-3.9.1-cp37-cp37m-macosx_10_9_intel.whl
Requirement already satisfied: setuptools>=41.0.0 in ./miniconda3/envs/test2/lib/python3.7/site-packages (from tb-nightly==1.15.0a20190708->test-tube>=0.6.9->pytorch-lightning) (41.0.1)
Collecting markdown>=2.6.8 (from tb-nightly==1.15.0a20190708->test-tube>=0.6.9->pytorch-lightning)
  Using cached https://files.pythonhosted.org/packages/c0/4e/fd492e91abdc2d2fcb70ef453064d980688762079397f779758e055f6575/Markdown-3.1.1-py2.py3-none-any.whl
Collecting werkzeug>=0.11.15 (from tb-nightly==1.15.0a20190708->test-tube>=0.6.9->pytorch-lightning)
  Downloading https://files.pythonhosted.org/packages/b7/61/c0a1adf9ad80db012ed7191af98fa05faa95fa09eceb71bb6fa8b66e6a43/Werkzeug-0.15.6-py2.py3-none-any.whl (328kB)
     |████████████████████████████████| 337kB 15.6MB/s
Requirement already satisfied: wheel>=0.26; python_version >= "3" in ./miniconda3/envs/test2/lib/python3.7/site-packages (from tb-nightly==1.15.0a20190708->test-tube>=0.6.9->pytorch-lightning) (0.33.4)
Collecting grpcio>=1.6.3 (from tb-nightly==1.15.0a20190708->test-tube>=0.6.9->pytorch-lightning)
  Using cached https://files.pythonhosted.org/packages/6b/f3/a73b5d2e328284054c0d4c2121672bca81277e44a7117bac1c0d348a7ae5/grpcio-1.23.0-cp37-cp37m-macosx_10_9_x86_64.whl
Collecting absl-py>=0.4 (from tb-nightly==1.15.0a20190708->test-tube>=0.6.9->pytorch-lightning)
  Downloading https://files.pythonhosted.org/packages/3c/0d/7cbf64cac3f93617a2b6b079c0182e4a83a3e7a8964d3b0cc3d9758ba002/absl-py-0.8.0.tar.gz (102kB)
     |████████████████████████████████| 112kB 26.1MB/s
Collecting pillow (from imageio>=2.3.0->test-tube>=0.6.9->pytorch-lightning)
  Downloading https://files.pythonhosted.org/packages/8f/f3/c6d351d7e582e4f2ef4343c9be1f0472cb249fb69695e68631e337f4b6e9/Pillow-6.1.0-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (3.8MB)
     |████████████████████████████████| 3.9MB 330kB/s
Building wheels for collected packages: pytorch-lightning, absl-py
  Building wheel for pytorch-lightning (setup.py) ... done
  Created wheel for pytorch-lightning: filename=pytorch_lightning-0.4.8-cp37-none-any.whl size=60520 sha256=405e0d5930623f061c6ff6752097a171e43df42695e531e5c2d761222ae73abf
  Stored in directory: /Users/user/Library/Caches/pip/wheels/c2/10/61/5f634fd0568104a10f44d671c6e1a33c054a77a498343c65d2
  Building wheel for absl-py (setup.py) ... done
  Created wheel for absl-py: filename=absl_py-0.8.0-cp37-none-any.whl size=120986 sha256=d95c0b244ef7882bd03e1e8d0ea7f0ff7bc85b9fb67c2e1316b62fe781e50d8a
  Stored in directory: /Users/user/Library/Caches/pip/wheels/9a/1e/7a/456008eb5e47fd5de792c6139df6d5b3d5f71d51c6a0b94799
Successfully built pytorch-lightning absl-py
Installing collected packages: numpy, torch, tqdm, six, protobuf, markdown, werkzeug, grpcio, absl-py, tb-nightly, pytz, python-dateutil, pandas, pillow, imageio, future, test-tube, pytorch-lightning
Successfully installed absl-py-0.8.0 future-0.17.1 grpcio-1.23.0 imageio-2.5.0 markdown-3.1.1 numpy-1.17.1 pandas-0.25.1 pillow-6.1.0 protobuf-3.9.1 python-dateutil-2.8.0 pytorch-lightning-0.4.8 pytz-2019.2 six-1.12.0 tb-nightly-1.15.0a20190708 test-tube-0.7.0 torch-1.2.0 tqdm-4.35.0 werkzeug-0.15.6
(test2) user-mbp:~ user$ python
Python 3.7.4 (default, Aug 13 2019, 15:17:50)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytorch_lightning as pl
>>>

[williamFalcon]

can't replicate on this machine...

maybe update your conda install? try installing other packages too?

https://superuser.com/questions/1402260/error-when-updating-conda-packages-removeerror-setuptools-is-a-dependency-of

conda install -c anaconda setuptools

[Borda]

I do not have much experience with macOS, I would conclude adding iOS to Travis testing...

[williamFalcon]

in my experience if it works in linux it works in osx (ds packages, conda, etc...)

the above print out is from a mbp

[williamFalcon]

@Borda merged. tests are passing. should we call this user error then? @dsuess
Maybe try again after upgrading your conda and following the links I posted?

We can reopen if it continues

[dsuess]

Thanks for your help, turns out it was a PYTHONPATH issue. I had set PYTHONPATH to some/directory:. Due to the : at the end, the current working directory is added to sys.path, which in turn breaks the installation due to the site directory. This was on both, my Mac and Linux box.

[s-rog]

I just had the same error in a kaggle kernel from importing lightning as a private dataset and running:
!cd /kaggle/input/pytorchlightning/ && pip install -e . --no-deps

  ERROR: Command errored out with exit status 1:
   command: /opt/conda/bin/python /opt/conda/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-_q8x_x53/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel
       cwd: None
  Complete output (7 lines):
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fcba3938710>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/setuptools/
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fcba39cd3c8>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/setuptools/
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fcba39cd400>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/setuptools/
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fcba39cd208>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/setuptools/
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fcba39cd518>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/setuptools/
  ERROR: Could not find a version that satisfies the requirement setuptools (from versions: none)
  ERROR: No matching distribution found for setuptools
  ----------------------------------------
ERROR: Command errored out with exit status 1: /opt/conda/bin/python /opt/conda/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-_q8x_x53/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel Check the logs for full command output.

This is the python path input/output:

> import os
> os.environ['PYTHONPATH'].split(os.pathsep)
['/kaggle/lib/kagglegym', '/kaggle/lib', '/kaggle/input/bengaliai-cv19']

This is ran in an offline kernel, but the error shows pip trying to install setuptools even though it is installed.

[s-rog]

!pip install ../input/pytorchlightning/test_tube-0.7.5-py3-none-any.whl
!pip install ../input/pytorchlightning/pytorch_lightning-0.5.3.2-py3-none-any.whl

Found this snippet used in this notebook, useful?

Edit: I tried running the snippet and it installs with the provided wheels

[dsuess]

I had a good time installing from git directly. Pretty hacky, but worked:

!pip install git+https://github.com/williamFalcon/pytorch-lightning

[williamFalcon]

I install lightning in a conda environment by doing:

pip install pytorch-lightning

Why did that not work?

[dsuess]

Back then the issue for me was a wrong PYTHONPATH setup + the site directory in the pip package. Maybe on colab there's a simliar bad combination of environment varialbes

[Borda]

you may check Using Pip to install packages to Anaconda Environment
there is also a task about supporting conda packaging #794

[s-rog]

Right now I'm converting the package to a wheel with no dependencies then installing in the kaggle kernel, seems to work fine.

[williamFalcon]

i use pip install pytorch-lightning on conda and colab without problems

[Borda]

@hadim may you comment on the conda packaging, do you have advice on how put lightning there?

[hadim]

I install all the packages available in conda-forge via conda and only the ones not available there with pip. I also usually like to use pip --no-deps to be sure only missing packages are installed with pip.

If pip packages are present in a conda env file then --no-deps won't be used. In 99% of the time, it's fine.

Here is an example of conda env file you could use:

# Usage: `conda env update -f environment.yml`
channels:
    - conda-forge
    - pytorch

dependencies:
    - python==3.7.6
    - pip==20.0.2
    - tqdm>=4.35.0
    - numpy>=1.16.4
    - pytorch>=1.1
    - tensorboard>=1.14
    - future>=0.17.1

    # For dev and testing
    - torchvision>=0.4.0, < 0.5  # the 0.5. has some issues with torch JIT
    - tox
    - coverage
    - codecov
    - pytest>=3.0.5
    - pytest-cov
    - pytest-flake8
    - flake8
    - autopep8
    - check-manifest
    - twine==1.13.0
    - pillow<7.0.0

    - pip:
        - test-tube>=0.7.5
        - mlflow
        - comet_ml
        - wandb
        - sacred
        - neptune-client

Then:

conda create -n my_env
conda activate my_env
conda env update -f environment.yml

# To get the latest PL version (adapt to your needs)
python -m pip install https://github.com/PyTorchLightning/pytorch-lightning/archive/master.zip

[Borda]

Great! side question @hadim would it make sense for you have lightning also on conda and if so would you interested in helping us to push the release there? :]

[hadim]

Since PL depends on pytorch, we can't push PL to conda forge because pytorch is available via a separate conda channel called pytorch.

[Borda]

I see, so let's keep it as it is... :]

[hadim]

See https://github.com/PyTorchLightning/pytorch-lightning/pull/898