Google-cloud-python: tox dies with "could not install deps" ?

[jgeewax]

I just wanted to test out the docs we have in CONTRIBUTING.rst, so I did the following:

$ git clone [email protected]:GoogleCloudPlatform/gcloud-python.git
Cloning into 'gcloud-python'...
remote: Counting objects: 24123, done.
remote: Compressing objects: 100% (158/158), done.
remote: Total 24123 (delta 113), reused 0 (delta 0), pack-reused 23963
Receiving objects: 100% (24123/24123), 33.61 MiB | 53.43 MiB/s, done.
Resolving deltas: 100% (18898/18898), done.
Checking connectivity... done.

$ cd gcloud-python/

$ sudo /usr/bin/pip install tox --upgrade
Downloading/unpacking tox from https://pypi.python.org/packages/2.7/t/tox/tox-2.1.1-py2.py3-none-any.whl#md5=c358c3c64e3ed21de9f0c5184c959097
  Downloading tox-2.1.1-py2.py3-none-any.whl
Downloading/unpacking virtualenv>=1.11.2 from https://pypi.python.org/packages/py2.py3/v/virtualenv/virtualenv-13.1.2-py2.py3-none-any.whl#md5=6e7b25033b4aadb262598b76b12edf3a (from tox)
  Downloading virtualenv-13.1.2-py2.py3-none-any.whl (1.7MB): 1.7MB downloaded
Downloading/unpacking py>=1.4.17 from https://pypi.python.org/packages/2.7/p/py/py-1.4.30-py2.py3-none-any.whl#md5=d0faee5774a42313943d4ceaf61b930b (from tox)
  Downloading py-1.4.30-py2.py3-none-any.whl (81kB): 81kB downloaded
Downloading/unpacking pluggy<0.4.0,>=0.3.0 (from tox)
  Downloading pluggy-0.3.0-py2.py3-none-any.whl
Installing collected packages: tox, virtualenv, py, pluggy
  Found existing installation: tox 1.6.0
    Not uninstalling tox at /usr/lib/python2.7/dist-packages, owned by OS
  Found existing installation: virtualenv 1.11.6
    Uninstalling virtualenv:
      Not removing or modifying (outside of prefix):
      /usr/bin/virtualenv
      Successfully uninstalled virtualenv
  Found existing installation: py 1.4.20
    Not uninstalling py at /usr/lib/python2.7/dist-packages, owned by OS
Successfully installed tox virtualenv py pluggy
Cleaning up...

So far so good.... And then...

$ /usr/bin/tox -e py27
GLOB sdist-make: /usr/local/google/home/jjg/gcloud-python/setup.py
py27 create: /usr/local/google/home/jjg/gcloud-python/.tox/py27
py27 installdeps: nose, unittest2, protobuf==3.0.0-alpha-1
ERROR: could not install deps [nose, unittest2, protobuf==3.0.0-alpha-1]
____________________________________________________________________________________ summary ____________________________________________________________________________________
ERROR:   py27: could not install deps [nose, unittest2, protobuf==3.0.0-alpha-1]

$ /usr/bin/tox -e py27 -v
using tox.ini: /usr/local/google/home/jjg/gcloud-python/tox.ini
using tox-1.6.0 from /usr/lib/python2.7/dist-packages/tox/__init__.pyc
GLOB sdist-make: /usr/local/google/home/jjg/gcloud-python/setup.py
  /usr/local/google/home/jjg/gcloud-python$ /usr/bin/python /usr/local/google/home/jjg/gcloud-python/setup.py sdist --formats=zip --dist-dir /usr/local/google/home/jjg/gcloud-python/.tox/dist >/usr/local/google/home/jjg/gcloud-python/.tox/log/tox-0.log
py27 create: /usr/local/google/home/jjg/gcloud-python/.tox/py27
  /usr/local/google/home/jjg/gcloud-python/.tox$ /usr/bin/python2.7 /usr/lib/python2.7/dist-packages/virtualenv.py --setuptools --python /usr/bin/python2.7 py27 >/usr/local/google/home/jjg/gcloud-python/.tox/py27/log/py27-0.log
py27 installdeps: nose, unittest2, protobuf==3.0.0-alpha-1
ERROR: could not install deps [nose, unittest2, protobuf==3.0.0-alpha-1]
____________________________________________________________________________________ summary ____________________________________________________________________________________
ERROR:   py27: could not install deps [nose, unittest2, protobuf==3.0.0-alpha-1]

Any idea what's going on here? The screams of some permission error (like it needs to install something in a directory it doesn't have access to write to...), but I thought the whole point of tox was to dynamically create virtual environments without me needing to ask... no?

[jgeewax]

The following seems to fix it (scarily):

$ source .tox/py27/bin/activate
$ pip install nose unittest2 protobuf==3.0.0-alpha-1
$ deactivate
$ tox -e py27

Any idea why I need to manually install these inside the tox venv ?

[dhermes]

You don't want to do that by hand. That's what tox does when you use it.

[jgeewax]

Yea -- when I try to let tox do it... it fails saying "could not install deps"... When I do it myself, everything works... Any chance this is a "my machine is fucked" situation? I'd be bummed if others are trying to follow our guide (which seems to work awesome on Travis...) and hitting this same ridiculous error...

[dhermes]

I just checked on my Desktop with a fresh checkout. Will check Ubuntu laptop and Mac laptop. I'm 90% sure it's your machine (but I'd still like to figure out the issue).

[jgeewax]

I can't seem to get tox to give me anything more verbose. The logs didn't have anything about this in there, and -v on the command-line doesn't seem to do much...

[dhermes]

The logs might be pip logs, try looking at logs in .tox/py27/log? (It'll probably be easier if you just wipe the .tox directory and start new.)

[dhermes]

@jgeewax Anything in the logs that shine some light on this?

[jgeewax]

Nope -- those were the logs I was saying were empty. Only showing success and some environment variables:

actionid: py27
msg: getenv
cmdargs: ['/usr/bin/python', '-m', 'virtualenv', '--python', '/usr/bin/python2.7', 'py27']
env: {'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:', 'XDG_GREETER_DATA_DIR': '/var/lib/lightdm-data/jjg', 'GNOME_DESKTOP_SESSION_ID': 'this-is-deprecated', 'XDG_CURRENT_DESKTOP': 'Unity', 'QT_IM_MODULE': 'ibus', 'LOGNAME': 'jjg', 'USER': 'jjg', 'NVM_DIR': '/usr/local/google/home/jjg/.nvm', 'P4MERGE': '/google/src/files/head/depot/eng/perforce/mergep4.tcl', 'XDG_VTNR': '7', 'GNOME_KEYRING_CONTROL': '/run/user/114505/keyring-UJZ0D4', 'PS1': '\\[\\033[31m\\]$(get_branch)\\[\\033[37m\\]\\[\\033[00m\\]\\[\\033[38m\\]\\u@\\h:\\w$ ', 'DISPLAY': ':0', 'SSH_AGENT_PID': '4529', 'LANG': 'en_US.UTF-8', 'TERM': 'xterm-256color', 'SHELL': '/bin/bash', 'XDG_SESSION_PATH': '/org/freedesktop/DisplayManager/Session0', 'XAUTHORITY': '/usr/local/google/home/jjg/.Xauthority', 'LANGUAGE': 'en_US:', 'SESSION_MANAGER': 'local/jjg1.nyc.corp.google.com:@/tmp/.ICE-unix/4726,unix/jjg1.nyc.corp.google.com:/tmp/.ICE-unix/4726', 'SHLVL': '1', 'MANDATORY_PATH': '/usr/share/gconf/ubuntu.mandatory.path', 'CLUTTER_IM_MODULE': 'xim', 'GDMSESSION': 'ubuntu', 'VIRTUAL_ENV': '/usr/local/google/home/jjg/gcloud-python/.tox/py27', 'TEXTDOMAIN': 'im-config', 'JOB': 'dbus', 'WINDOWID': '10485983', 'SESSIONTYPE': 'gnome-session', 'PARINIT': 'rTbgqR B=.?_A_a Q=_s>|:', 'PYTHONHASHSEED': '4274346326', 'GPG_AGENT_INFO': '/tmp/gpg-q4Kykn/S.gpg-agent:4698:1', 'HOME': '/usr/local/google/home/jjg', 'AUTO_PROXY': 'https://proxyconfig.corp.google.com/proxy.pac', 'QT4_IM_MODULE': 'xim', 'SELINUX_INIT': 'YES', 'CODE_ROOT_DIRS': '/usr/local/google/home/jjg/projects/:/usr/local/google/home/jjg/playground/', 'XDG_RUNTIME_DIR': '/run/user/114505', 'GTK_IM_MODULE': 'ibus', 'PYTHONPATH': '/usr/local/buildtools/current/sitecustomize', 'COMPIZ_CONFIG_PROFILE': 'ubuntu', 'SSH_AUTH_SOCK': '/tmp/ssh-ccfidRfQ9r6X/agent.4528', 'VTE_VERSION': '3409', 'MAILCHECK': '604800', 'PATH': '/usr/local/google/home/jjg/gcloud-python/.tox/py27/bin:/usr/local/google/home/jjg/google-cloud-sdk/bin:/opt/dart-sdk/bin:/usr/local/google/home/jjg/node_modules/less/bin:/usr/local/go/bin:/opt/android/tools:/opt/android/platform-tools:/usr/local/google/home/jjg/bin:/var/lib/gems/1.8/bin:/usr/lib/google-golang/bin:/usr/local/buildtools/java/jdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'XMODIFIERS': '@im=ibus', 'DEFAULTS_PATH': '/usr/share/gconf/ubuntu.default.path', 'TEXTDOMAINDIR': '/usr/share/locale/', 'P4CONFIG': '.p4config', 'XDG_DATA_DIRS': '/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/', 'XDG_SEAT_PATH': '/org/freedesktop/DisplayManager/Seat0', 'XDG_CONFIG_DIRS': '/etc/xdg/xdg-ubuntu:/usr/share/upstart/xdg:/etc/xdg', 'LESSOPEN': '| /usr/bin/lesspipe %s', 'XDG_SESSION_ID': 'c2', 'DBUS_SESSION_BUS_ADDRESS': 'unix:abstract=/tmp/dbus-Zzptw2ITmg', '_': '/usr/local/bin/tox', 'P4DIFF': '/google/src/files/head/depot/google3/devtools/scripts/p4diff -w', 'SESSION': 'ubuntu', 'DESKTOP_SESSION': 'ubuntu', 'IM_CONFIG_PHASE': '1', 'LESSCLOSE': '/usr/bin/lesspipe %s %s', 'GNOME_KEYRING_PID': '4511', 'UBUNTU_MENUPROXY': 'libappmenu.so', 'XDG_SEAT': 'seat0', 'OLDPWD': '/usr/local/google/home/jjg', 'GDM_LANG': 'en_US', 'GTK_MODULES': 'overlay-scrollbar', 'INSTANCE': '', 'PWD': '/usr/local/google/home/jjg/gcloud-python', 'CVS_RSH': 'ssh', 'COLORTERM': 'gnome-terminal', 'NODE_PATH': '/usr/lib/nodejs:/usr/lib/node_modules:/usr/share/javascript', 'MAIL': '/dev/null', 'RSYNC_RSH': 'ssh', 'XDG_MENU_PREFIX': 'gnome-', 'COMPIZ_BIN_PATH': '/usr/bin/', 'UPSTART_SESSION': 'unix:abstract=/com/ubuntu/upstart-session/114505/4513'}

New python executable in py27/bin/python2.7
Also creating executable in py27/bin/python
Installing setuptools, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2.7

[dhermes]

It's doubtful but https://github.com/pypa/virtualenv/issues/539 may have something to do with it.

Can you

sudo -H pip install --upgrade pip tox virtualenv

and try again?

[jgeewax]

OK so.... I did the following:

$ sudo apt-get remove python-pip
$ sudo easy_install pip
$ sudo ln -s /usr/local/bin/pip /usr/bin/pip # For whatever reason, it didn't do this for me...
$ sudo -H pip install --upgrade pip tox virtualenv
$ git clone [email protected]:GoogleCloudPlatform/gcloud-python.git
$ cd gcloud-python
$ tox -e py27

Great success!

Hopefully this helps someone else when they run into this.........

[dhermes]

Yeah getting the right pip installed can be a bit funky.

Great to hear it's resolved!

[Ritvik-Sapra]

I ran into same issue @jgeewax I don;t think it was your machine. just installed by typing pip install tox and I stll got the error. However, I'm on a widows machine. Any idea how I can fix it?