Deeplabcut: Unable to crop videos (in manual frame-grabbing mode)

[zhangwujie]

Your Operating system and DeepLabCut version
Windows 10, Anaconda (using the provided environment dlc-windowsGPU.yaml), DeepLabCut 2.0.3

Describe the problem
When I extract frames using
deeplabcut.extract_frames(config_file,mode='manual',crop=True,checkcropping=True)
The GUI pops up and asks whether to crop the video, and if I click yes, I get:

KeyError                                  Traceback (most recent call last)
c:\users\analysis\anaconda3\envs\dlc-windowsgpu\lib\site-packages\deeplabcut\generate_training_dataset\frame_extraction_toolbox.py in browseDir(self, event)
    169                 self.cropping = True
    170                 self.basefolder = 'data-' + self.Task + '/'
--> 171                 MainFrame.CheckCropping(self)
    172             else:
    173                 self.cropping = False

c:\users\analysis\anaconda3\envs\dlc-windowsgpu\lib\site-packages\deeplabcut\generate_training_dataset\frame_extraction_toolbox.py in CheckCropping(self)
     99         filename = self.filename
    100         time = self.start
--> 101         self.x1 = int(self.cfg['video_sets'][videosource]['crop'].split(',')[0])
    102         self.x2 = int(self.cfg['video_sets'][videosource]['crop'].split(',')[1])
    103         self.y1 = int(self.cfg['video_sets'][videosource]['crop'].split(',')[2])

KeyError: 'C:\\Users\\Analysis\\BatVideos\\0104\\Downsampled\\short_video.avi'

And if I then click grab frame, I get:

AttributeError                            Traceback (most recent call last)
c:\users\analysis\anaconda3\envs\dlc-windowsgpu\lib\site-packages\deeplabcut\generate_training_dataset\frame_extraction_toolbox.py in grabFrame(self, event)
    211             img_name = str(output_path) +'/img'+str(self.currFrame).zfill(int(np.ceil(np.log10(self.numberFrames)))) + '.png'
    212             if self.cropping:
--> 213                 crop_img = frame[self.y1:self.y2, self.x1:self.x2]
    214                 cv2.imwrite(img_name, cv2.cvtColor(crop_img, cv2.COLOR_RGB2BGR))
    215             else:

AttributeError: 'MainFrame' object has no attribute 'y1'

If I don't crop, then grabbing frames works fine.

My config.yaml is the following:

batch_size: 4
project_path: C:\Users\Analysis\BatVideos\0104\Downsampled\Results\short-WZ-2019-01-18
iteration: 0
colormap: jet
alphavalue: 0.5
dotsize: 10
y2: 624
y1: 277
x2: 640
x1: 0
move2corner: true
corner2move2: !!python/tuple
- 50
- 50
pcutoff: 0.1
snapshotindex: -1
resnet: 50
TrainingFraction:
- 0.95
date: Jan18
scorer: WZ
bodyparts:
- hand
- Finger1
- Finger2
- Joystick
numframes2pick: 4
stop: 1
start: 0
cropping: true
video_sets:
  C:\Users\Analysis\BatVideos\0104\Downsampled\Results\short-WZ-2019-01-18\videos\short_video.avi:
    crop: 10, 50, 10, 50
Task: short

Thank you very much!
Wujie

[MMathisLab]

if you use this feature with automatic mode, i.e., deeplabcut.extract_frames(config_file,mode='automatic',crop=True,checkcropping=True) do you get the same error?

[zhangwujie]

Automatic mode works fine, no error.

[MMathisLab]

Great, thanks for checking! We will look into it for the manual mode!

[zhangwujie]

Thanks for the super quick response!

[AlexEMG]

This is the expected behavior: At the state you showed the video you want to add ('C:\\Users\\Analysis\\BatVideos\\0104\\Downsampled\\short_video.avi') is not part of the video set:

(from your config file):
video_sets:
C:UsersAnalysisBatVideos0104DownsampledResultsshort-WZ-2019-01-18videosshort_video.avi:
crop: 10, 50, 10, 50

For that reason you get a key-error (and subsequently there are no crop parameters for this video, because it is not part of the video_sets). You can always add a video to the list by:

deeplabcut.add_new_video ...

The cropping works for me, but the behavior is different than for the automatic mode, it creates a file in the labeled data folder to check. If that is cropped as expected than when you click grab_frame the frames will be cropped in the same way.

[MMathisLab]

(ah! good catch, @AlexEMG - strange how this could happen though, as the extract frames only get the video list from the config.yaml file, no?) And, it works with automatic...

[zhangwujie]

Thanks!
I'm confused because the video in the error ('C:\\Users\\Analysis\\BatVideos\\0104\\Downsampled\\short_video.avi') and the video in the config file (C:\Users\Analysis\BatVideos\0104\Downsampled\Results\short-WZ-2019-01-18\videos\short_video.avi) are the same one. The latter is just a copy of the former, created by

task='short'
experimenter='WZ'
video=['C:\\Users\\Analysis\\BatVideos\\0104\\Downsampled\\short_video.avi']
working_directory='C:\\Users\\Analysis\\BatVideos\\0104\\Downsampled\\Results'
deeplabcut.create_new_project(task,experimenter,video,working_directory,copy_videos=False)

So I should add the original video again via deeplabcut.add_new_video?

[AlexEMG]

This is a bit unfortunate, but I guess with the current implementation you would have to click on the one in the project/video folder (i.e. C:\Users\Analysis\BatVideos\0104\Downsampled\Results\short-WZ-2019-01-18\videos\short_video.avi) and then it should work. No need to add the video again.

[zhangwujie]

I was able to get it to work with this:
Create the project with copy_videos=True, then in the extract frames GUI, load the original video (not the one in the project folder).

Before, I had copy_videos=False, and neither loading the original video nor the one in the project folder worked.

Thank you for your help!