Deeplabcut: Misleading error message or lax failure behaviour leads to (test) failures down the line
See https://github.com/NixOS/nixpkgs/pull/61253#issuecomment-605446102 ,
If OpenCV fails to do something at https://github.com/AlexEMG/DeepLabCut/blob/bcef8c961ed27a5fc8f947ef97b29a466e3f371a/deeplabcut/create_project/new.py#L156 video_sets will be set to None and execution will continue as if nothing happened. Later, tests error when cfg["videos"] is empty.
deliciouslytyped
All 15 comments
hi @deliciouslytyped we would ask that you fill out the template completely in the future.
Your issue is that if one creates a project incorrectly (i.e. user error), it doesn't throw a warning/error message. i.e. it does this:
Python 3.7.0 (default, Jun 28 2018, 07:39:16)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import deeplabcut
deeplabcut.create_new_projec>>> deeplabcut.create_new_project('mwm', 'test', ['/home/fakenews/video.mp4'])
Created "/Users/mwmathis/Documents/mwm-test-2020-03-28/videos"
Created "/Users/mwmathis/Documents/mwm-test-2020-03-28/labeled-data"
Created "/Users/mwmathis/Documents/mwm-test-2020-03-28/training-datasets"
Created "/Users/mwmathis/Documents/mwm-test-2020-03-28/dlc-models"
Creating the symbolic link of the video
Generated "/Users/mwmathis/Documents/mwm-test-2020-03-28/config.yaml"
A new project with name mwm-test-2020-03-28 is created at /Users/mwmathis/Documents and a configurable file (config.yaml) is stored there. Change the parameters in this file to adapt to your project's needs.
Once you have changed the configuration file, use the function 'extract_frames' to select frames for labeling.
. [OPTIONAL] Use the function 'add_new_videos' to add new videos to your project (at any stage).
'/Users/mwmathis/Documents/mwm-test-2020-03-28/config.yaml'
even though it's a fake video.
i.e. in the config.yaml it does not populate a fake video:
Task: mwm
scorer: test
date: Mar28
# Project path (change when moving around)
project_path: /Users/mwmathis/Documents/mwm-test-2020-03-28
# Annotation data set configuration (and individual video cropping parameters)
video_sets: {}
bodyparts:
- bodypart1
- bodypart2
- bodypart3
- objectA
Of course, if one tries to extract then label frames then it does throw an error as no video is found. (but I agree, even here there is a lack of warning at the extract step @AlexEMG @jeylau)
>>> deeplabcut.extract_frames('/Users/mwmathis/Documents/mwm-test-2020-03-28/config.yaml')
Config file read successfully.
Frames were selected.
You can now label the frames using the function 'label_frames' (if you extracted enough frames for all videos).
Would you like it to throw an error on the first badly created project?
MMathisLab
on 28 Mar 2020
Sorry, I was kind of in a flow state and forgot to go through the issue reporting template.
deliciouslytyped
on 28 Mar 2020
So to review:
We're building deeplabcut v2.1.6 .
A more complete description, since I've made some progress debugging the issue, is that if OpenCV is compiled without FFMPEG, testscript.py will fail because of a badly generated configuration.
I've also posted updates to the crosslinked issue.
Here is a more complete backlog of the original failure :
Imported DLC!
CREATING PROJECT
Created "/run/user/1000/tmp.k2PBGUoiP6/source/examples/TEST-Alex-2020-03-28/videos"
Created "/run/user/1000/tmp.k2PBGUoiP6/source/examples/TEST-Alex-2020-03-28/labeled-data"
Created "/run/user/1000/tmp.k2PBGUoiP6/source/examples/TEST-Alex-2020-03-28/training-datasets"
Created "/run/user/1000/tmp.k2PBGUoiP6/source/examples/TEST-Alex-2020-03-28/dlc-models"
Copying the videos
/run/user/1000/tmp.k2PBGUoiP6/source/examples/TEST-Alex-2020-03-28/videos/reachingvideo1.avi
VIDIOC_REQBUFS: Inappropriate ioctl for device
Cannot open the video file!
Generated "/run/user/1000/tmp.k2PBGUoiP6/source/examples/TEST-Alex-2020-03-28/config.yaml"
A new project with name TEST-Alex-2020-03-28 is created at /run/user/1000/tmp.k2PBGUoiP6/source/examples and a configurable file (config.yaml) is stored there. Change the parameters in this file to adapt to your project's needs.
Once you have changed the configuration file, use the function 'extract_frames' to select frames for labeling.
. [OPTIONAL] Use the function 'add_new_videos' to add new videos to your project (at any stage).
EXTRACTING FRAMES
Config file read successfully.
Traceback (most recent call last):
File "testscript.py", line 55, in <module>
deeplabcut.extract_frames(path_config_file,mode='automatic',userfeedback=False)
File "/nix/store/0rqz6aqpp35zl71s1a9ljs2amjv6hz2r-python3.7-deeplabcut-2.1.6/lib/python3.7/site-packages/deeplabcut/generate_training_dataset/frame_extraction.py", line 119, in extract_frames
videos = cfg['video_sets'].keys()
AttributeError: 'NoneType' object has no attribute 'keys'
VIDIOC_REQBUFS: Inappropriate ioctl for device
Is the OpenCV failure, and enabling the OpenCV debug vars OPENCV_LOG_LEVEL=debug OPENCV_VIDEOIO_DEBUG=1 gives:
[ INFO:0] VIDEOIO: Enabled backends(3, sorted by priority): V4L2(1000); CV_IMAGES(990); CV_MJPEG(980)
[ WARN:0] VIDEOIO(cvCreateCameraCapture_V4L(filename.c_str())): trying ...
VIDIOC_REQBUFS: Inappropriate ioctl for device
[ WARN:0] VIDEOIO(cvCreateCameraCapture_V4L(filename.c_str())): result=(nil) ...
[ WARN:0] VIDEOIO(cvCreateFileCapture_Images(filename.c_str())): trying ...
[ INFO:0] Pattern: /run/user/1000/tmp.k2PBGUoiP6/source/examples/TEST-Alex-2020-03-28/videos/reachingvideo%01d.avi @ 1
[ INFO:0] CAP_IMAGES: Stop scanning. Can't read image file: /run/user/1000/tmp.k2PBGUoiP6/source/examples/TEST-Alex-2020-03-28/videos/reachingvideo1.avi
[ WARN:0] VIDEOIO(cvCreateFileCapture_Images(filename.c_str())): result=(nil) ...
[ WARN:0] VIDEOIO(createMotionJpegCapture(filename)): trying ...
[ WARN:0] VIDEOIO(createMotionJpegCapture(filename)): result=(nil) isOpened=-1 ...
This is not very helpful, but issues like https://github.com/opencv/opencv/issues/14721 point to the underlying FFMPEG problem.
On the deeplabcut side:
When create_new_project is called, https://github.com/AlexEMG/DeepLabCut/blob/bcef8c961ed27a5fc8f947ef97b29a466e3f371a/deeplabcut/create_project/new.py#L156 is called on each video that should be copied.
If OpenCV is compiled without FFMPEG, VideoCapture() fails to open the video and you correctly check for this via isOpen. (I haven't used OpenCV before, and this seems like a pretty crappy way to handle this, but I'm not the person that wrote OpenCV...).
I'm not sure I would label this "user error", but given that we both agree that this is a problem, I won't push it. I'm not really sure who is at fault here - arguably the linux distribution providing OpenCV - which is why we are going to fix it by fixing our packaging script. Also this breaks the test, which is good, because that's what tests are for, but it doesn't give any helpful sort of error and I had to go track down where the problem is originating from. Namely; that the project file is being created incorrectly as you said.
I don't know what the best way to handle this would be, but indeed I think the problem should be raised where the problem happens - i.e. at project creation.
deliciouslytyped
on 28 Mar 2020
If you don't mind me asking a mostly unrelated question here; is there a script that runs all your tests or do we need to run all the test*.py scripts individually?
deliciouslytyped
on 28 Mar 2020
To clarify, I don't actually use deeplabcut, maybe cc @tbenst for what users would like here.
But I think you should probably raise some kind of exception?
If you dont want the whole generation to fail, remove the file from the list of videos, or add an ignore list? I don't know.
You could refactor the project generation code into several functions and if a user doesn't like the default version they can write their own.
deliciouslytyped
on 28 Mar 2020
(I can answer more later, but one should definitely be running 2.1.6.3 as the update had several good updates)
MMathisLab
on 28 Mar 2020
Is there a particular commit that corresponds to 2.1.6.3? Right now we鈥檙e pulling the source code from git using the 2.1.6 tag which appears to be the latest, as the test scripts / example data aren鈥檛 distributed with PyPI
tbenst
on 28 Mar 2020
MMathisLab
on 29 Mar 2020
I made a 2.1.6.4 release tag: https://github.com/AlexEMG/DeepLabCut/releases/tag/v2.1.6.4
Also I changed the testscript.py so that it doesn't use tensorpack augmentation on Windows/MacOS by default.
AlexEMG
on 29 Mar 2020
@deliciouslytyped - the testscript.py tests all main functions of DLC; it does not run the 3d tests (which requires the testscript.py to be run first). The others are just sub-tests of specific functions; not strictly required. testscript.py covers the main fxns.
MMathisLab
on 31 Mar 2020
Just making sure; so if we wanted to be complete we would run testscript.py and then the 3d script?
deliciouslytyped
on 31 Mar 2020
Ok, can you check this: https://github.com/AlexEMG/DeepLabCut/pull/642
AlexEMG
on 1 Apr 2020
Testing the failure:
Imported DLC!
CREATING PROJECT
Created "/build/source/examples/TEST-Alex-2020-04-01/videos"
Created "/build/source/examples/TEST-Alex-2020-04-01/labeled-data"
Created "/build/source/examples/TEST-Alex-2020-04-01/training-datasets"
Created "/build/source/examples/TEST-Alex-2020-04-01/dlc-models"
Copying the videos
/build/source/examples/TEST-Alex-2020-04-01/videos/reachingvideo1.avi
VIDIOC_REQBUFS: Inappropriate ioctl for device
Cannot open the video file! Skipping to the next one...
No valid videos found. The project was not created...
Verify the video files and recreate the project.
Traceback (most recent call last):
File "testscript.py", line 52, in <module>
cfg=deeplabcut.auxiliaryfunctions.read_config(path_config_file)
File "/nix/store/i109mas77f63w4j7h22qbykr9nhazwrm-python3.7-deeplabcut-2.1.6/lib/python3.7/site-packages/deeplabcut/utils/auxiliaryfunctions.py", line 127, in read_config
raise FileNotFoundError ("Config file is not found. Please make sure that the file exists and/or that you passed the path of the config file correctly!")
FileNotFoundError: Config file is not found. Please make sure that the file exists and/or that you passed the path of the config file correctly!
builder for '/nix/store/hrqc7v4hl93v3z1awf0cxwvmvq6fyhbj-python3.7-deeplabcut-2.1.6.drv' failed with exit code 1
Success:
Imported DLC!
CREATING PROJECT
Created "/build/source/examples/TEST-Alex-2020-04-01/videos"
Created "/build/source/examples/TEST-Alex-2020-04-01/labeled-data"
Created "/build/source/examples/TEST-Alex-2020-04-01/training-datasets"
Created "/build/source/examples/TEST-Alex-2020-04-01/dlc-models"
Copying the videos
/build/source/examples/TEST-Alex-2020-04-01/videos/reachingvideo1.avi
Generated "/build/source/examples/TEST-Alex-2020-04-01/config.yaml"
A new project with name TEST-Alex-2020-04-01 is created at /build/source/examples and a configurable file (config.yaml) is stored there. Change the parameters in this file to adapt to your project's needs.
Once you have changed the configuration file, use the function 'extract_frames' to select frames for labeling.
. [OPTIONAL] Use the function 'add_new_videos' to add new videos to your project (at any stage).
EXTRACTING FRAMES
Config file read successfully.
Extracting frames based on kmeans ...
[snip]
LGTM I guess?
cc @tbenst
deliciouslytyped
on 1 Apr 2020
nice!
tbenst
on 2 Apr 2020
I guess I'll close. Will you release a new version or should we use the hash?
deliciouslytyped
on 2 Apr 2020
Related issues
phcao101
路
4Comments
Xieyuanting
路
3Comments
guyts
路
3Comments
monajalal
路
4Comments
cvKDean
路
4Comments