Deeplabcut: Installion from git creates a 'dlc' command that does not work

[JanneM]

Issue:

After installation, the installer creates a "dlc" command bin/ that tries to import from a "dlc" module that does not exist. Replacing it with the code from "dlc.py" in the top-level source directory makes it functional. This happens both when installing it from pipy through pip or when installing a stable release from the git source repo.

The manual doesn't refer to the command at all. So is this a real problem, or is the command a leftover from an earlier version or something like that?

Symptom:

$ dlc
Traceback (most recent call last):
  File "/home/xxxxxx/.local/bin/dlc", line 5, in <module>
    from dlc import main
ModuleNotFoundError: No module named 'dlc'

Reproduce:

# Python 3.6.7 installed into a cluster
$ module load python/3.6.7

$ pip3 install --upgrade --user tensorflow-gpu==1.13.1
$ pip3 install --user -f https://extras.wxpython.org/wxPython [...] wxpython

# either:
$ git clone https://github.com/DeepLabCut/DeepLabCut.git dlc
$ cd dlc
$ git checkout v2.1.8.2
$ pip3 install --user .

# or:
$ pip3 install --user deeplabcut

$ dlc

System

• CentOS 7.6
• DeepLabCut 2.1.8.2, 2.2b8

Fix:

Replace installed 'dlc':

#!/apps/free76/python/3.6.7/bin/python3.6
# -*- coding: utf-8 -*-
import re
import sys
from dlc import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

With contents of 'dlc.py' (copyright text elided for brevity):

#!/apps/free76/python/3.6.7/bin/python3.6
# -*- coding: utf-8 -*-
from deeplabcut import cli

def main():
    cli.main()

if __name__ == '__main__':
    main()

[MMathisLab]

Could you maybe explain your use case/what you are hoping to achieve?

the cli interface is not documented, you are correct ;)

[JanneM]

I'm not trying to use the cli interface at all. I am just worried that this issue is an effect of a failed or incomplete installation on my part.

I'm not a user myself; I'm installing it on a shared system for others to use, and I want to try to make sure it is correctly installed before I ask any of them to test it.

If it's supposed to be broken then all is well :-)

[MMathisLab]

Hey @JanneM makes sense! If helpful, I would run just this testscript then: https://github.com/DeepLabCut/DeepLabCut/blob/master/examples/testscript.py

(for the cli, I'll let @AlexEMG dig future into your Q)

[JanneM]

@MMathisLab I tried the script, and it sort of works. As in, it initializes and runs the training part. That's probably good enough test that I can let a few users on to it.

But FYI the test fails later, telling me it doesn't find some of the input data:

Using snapshot-5 for model /home/j/jan-moren/tmp/dlc/examples/TEST-Alex-2020-10-19/dlc-models/iteration-0/TESTOct19-trainset80shuffle1
Initializing ResNet
No video/s found. Please check your path!
CREATE VIDEO
Making plots
Plots created! Please check the directory "plot-poses" within the video directory
EXTRACT OUTLIERS
RELABELING
Traceback (most recent call last):
File "./testscript.py", line 138, in
DF=pd.read_hdf(file,'df_with_missing')
File "/apps/test/jan-moren/deeplabcut/2.1.8.2/lib/python3.6/site-packages/pandas/io/pytables.py", line 387, in read_hdf
raise FileNotFoundError(f"File {path_or_buf} does not exist")
FileNotFoundError: File /home/j/jan-moren/tmp/dlc/examples/TEST-Alex-2020-10-19/labeled-data/reachingvideo1short/machinelabels-iter0.h5 does not exist
There is no directory "reachingvideo1short", only "reachingvideo1/"

Also, if you rerun the test you need to manually delete the TEST-Alex-2020-10-19/ directory first, or it fails with an unhelpful error.

Finally, I note that even with a batch-oriented example like this, it still needs an X server; this makes DeepLabCut difficult to run on a shared cluster where the norm is to submit batch jobs, then log out and do something else. I realize this is not the normal use-case of course, so please don't take it as criticism. Just a note to consider in case you re-architect the system in the future.

[MMathisLab]

Hi @JanneM yes, agreed! :) and feedback always welcomed. We are in fact splitting the package; and I don’t use this on our cluster I use the deeplabcut-core package which doesn’t require xserver. I use this through Docker: https://github.com/DeepLabCut/Docker4DeepLabCut2.0

And here is headless dlc: https://github.com/DeepLabCut/DeepLabCut-core