Deeplabcut: Pip install deeplabcut command causes Google Colab to continually crash

[Jonathan-Oehley]

I am using a Google Colab GPU Runtime with Deeplabcut 2.0.9
OS: Ubuntu
DeepLabCut Version: 2.0.9

On any Google Colab Runtime the command '!pip install deeplabcut' executes no differently to in the past weeks. However, as of the morning of 01 October, when restarting the runtime after executing the above command, the Runtime refuses to reconnect and repeatedly crashes. I've encountered the same issue with the notebooks available on this GitHub namely 'latest_Colab_TrainNetwork_VideoAnalysis' and 'Colab_DEMO_mouse_openfield'.

A standout warning (now noticed in the context of this error) was the below message that is printed out after executing the above command:

"WARNING: Upgrading ipython, ipykernel, tornado, prompt-toolkit or pyzmq can
cause your runtime to repeatedly crash or behave in unexpected ways and is not
recommended. If your runtime won't connect or execute code, you can reset it
with "Reset all runtimes..." from the "Runtime" menu."

I've noted that the logging prints out the following snippet:

"Building wheels for collected packages: pyyaml
Building wheel for pyyaml (setup.py) ... done
Created wheel for pyyaml: filename=PyYAML-5.1.2-cp36-cp36m-linux_x86_64.whl size=44104 sha256=56f41498a76bb84a397f714fc0617a016f8eec9363b0307a6cdbd453e8facf04
Stored in directory: /root/.cache/pip/wheels/d9/45/dd/65f0b38450c47cf7e5312883deb97d065e030c5cca0a365030
Successfully built pyyaml
....
ERROR: google-colab 1.0.0 has requirement ipython~=5.5.0, but you'll have ipython 6.0.0 which is incompatible."

I cannot recall if the warning message was present before I encountered these difficulties, but perhaps it has something to do with the ipython package. I've attached the complete printout of the install command as well.

DLC_install_log.txt

Steps to reproduce the behaviour:

1. On any Google Colab Runtime run !pip install deeplabcut
2. Restart the runtime
3. Note that the runtime continues to crash and cannot be reconnected to.

[Jonathan-Oehley]

I've found what I believe to be the source of the error after getting hold of the Colab crash log

Colab Crashlog.txt

Google Colab requires six=1.12.0 (see install log in my original post) and it seems recent changes require version 1.12 of six within the setup function of Google Colab.
This is due to the fact that Colab requires use of the six.ensure_str function which is not present in version 1.11 of six. This is what seemed to be causing Colab to crash.

Removing the six~=1.11.0 requirement from DLC setup.py seems to have solved the issue for me (which forced an installation of version 1.11). Although a more permanent fix could be a change in the requirement to six~=1.12.0.

However, I am not familiar with the DLC code to understand the possible implications of using six 1.12 over 1.11.

Would someone be able to tell me if there are any possible groundbreaking issues with using version 1.12 of six? My code appears to be running fine at the moment with version 1.12.

[AlexEMG]

As far as I can tell the six update is fine!

[Jonathan-Oehley]

Great! Seems to be fine from what I can see. Just wanted to let you guys know in terms of getting the Colab example notebooks to work again if others have the same issue. (also it seems tensorflow 1.14 may recently require a different gast package requirement which could cause some model issues)

[MMathisLab]

awe yes, the nightly backend update of colab finally caught us ;). we will be releasing 2.1 in a few days and will be sure to rest this, thanks!