Deeplabcut: create_labeled_video_3d [defining skeleton Q]

[mawaishafeez]

Using: Ubuntu 16.04 LTS, with an Anaconda Env, & DLC 2.0.7

Describe the problem
Having issues with labeling 3d videos... couldn't sort it out.. Folder contains videos from both cameras and .h5 file ... can you please help to locate the error

In [59]: dlc.create_labeled_video_3d(config_file_3d, [videospath], trailpoints=3, videotype=".mp4")
Analyzing all the videos in the directory
[['/work/dlc-project/prehen/ Ah T2  D20 W7_DLC_3D.h5', '/work/dlc-project/prehen/camera-1 Ah T2  D20 W7.mp4', '/work/dlc-project/prehen/camera-2 Ah T2  D20 W7.mp4'], ['/work/dlc-project/prehen/ Ah T2  D20 W5_DLC_3D.h5', '/work/dlc-project/prehen/camera-1 Ah T2  D20 W5.mp4', '/work/dlc-project/prehen/camera-2 Ah T2  D20 W5.mp4'], ['/work/dlc-project/prehen/ Ah T3  D40 W5_DLC_3D.h5', '/work/dlc-project/prehen/camera-1 Ah T3  D40 W5.mp4', '/work/dlc-project/prehen/camera-2 Ah T3  D40 W5.mp4'], ['/work/dlc-project/prehen/ Ah T2  D40 W7_DLC_3D.h5', '/work/dlc-project/prehen/camera-1 Ah T2  D40 W7.mp4', '/work/dlc-project/prehen/camera-2 Ah T2  D40 W7.mp4'], ['/work/dlc-project/prehen/ Ah T1  D40 W7_DLC_3D.h5', '/work/dlc-project/prehen/camera-1 Ah T1  D40 W7.mp4', '/work/dlc-project/prehen/camera-2 Ah T1  D40 W7.mp4'], ['/work/dlc-project/prehen/ Ah T3  D40 W7_DLC_3D.h5', '/work/dlc-project/prehen/camera-1 Ah T3  D40 W7.mp4', '/work/dlc-project/prehen/camera-2 Ah T3  D40 W7.mp4'], ['/work/dlc-project/prehen/ Ah T1  D20 W7_DLC_3D.h5', '/work/dlc-project/prehen/camera-1 Ah T1  D20 W7.mp4', '/work/dlc-project/prehen/camera-2 Ah T1  D20 W7.mp4'], ['/work/dlc-project/prehen/ Ah T3  D20 W7_DLC_3D.h5', '/work/dlc-project/prehen/camera-1 Ah T3  D20 W7.mp4', '/work/dlc-project/prehen/camera-2 Ah T3  D20 W7.mp4'], ['/work/dlc-project/prehen/ Ah T2  D40 W5_DLC_3D.h5', '/work/dlc-project/prehen/camera-1 Ah T2  D40 W5.mp4', '/work/dlc-project/prehen/camera-2 Ah T2  D40 W5.mp4'], ['/work/dlc-project/prehen/ Ah T3  D20 W5_DLC_3D.h5', '/work/dlc-project/prehen/camera-1 Ah T3  D20 W5.mp4', '/work/dlc-project/prehen/camera-2 Ah T3  D20 W5.mp4'], ['/work/dlc-project/prehen/ Ah T1  D40 W5_DLC_3D.h5', '/work/dlc-project/prehen/camera-1 Ah T1  D40 W5.mp4', '/work/dlc-project/prehen/camera-2 Ah T1  D40 W5.mp4']]
Creating 3D video from camera-1 Ah T2  D20 W7.mp4 and camera-2 Ah T2  D20 W7.mp4 using  Ah T2  D20 W7_DLC_3D.h5
Trying to find filtered predictions...
Found filtered predictions! I will use these for triangulation.
  0%|                                                                                                                     | 0/296 [00:00<?, ?it/s]
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/home/ubuntu/anaconda3/lib/python3.6/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2521             try:
-> 2522                 return self._engine.get_loc(key)
   2523             except KeyError:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'O'

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
<ipython-input-59-9277253585c4> in <module>()
----> 1 dlc.create_labeled_video_3d(config_file_3d, [videospath], trailpoints=3, videotype=".mp4")

/home/ubuntu/anaconda3/lib/python3.6/site-packages/deeplabcut/pose_estimation_3d/plotting3D.py in create_labeled_video_3d(config, path, videofolder, start, end, trailpoints, videotype, view, xlim, ylim, zlim, draw_skeleton)
    133         # Start plotting for every frame
    134         for k in tqdm(frames):
--> 135             output_folder,num_frames = plot2D(cfg_3d,k,bodyparts2plot,vid_cam1,vid_cam2,bodyparts2connect,df_cam1,df_cam2,df_3d,pcutoff,markerSize,alphaValue,color,path_h5_file,file_name,skeleton_color,view,draw_skeleton,trailpoints,xlim,ylim,zlim)
    136 
    137         # Once all the frames are saved, then make a movie using ffmpeg.

/home/ubuntu/anaconda3/lib/python3.6/site-packages/deeplabcut/pose_estimation_3d/plotting3D.py in plot2D(cfg_3d, k, bodyparts2plot, vid_cam1, vid_cam2, bodyparts2connect, df_cam1_view, df_cam2_view, xyz_pts, pcutoff, markerSize, alphaValue, color, path_h5_file, file_name, skeleton_color, view, draw_skeleton, trailpoints, xlim, ylim, zlim)
    196     df_z = np.empty((len(bodyparts2plot),numberFrames))
    197     for bpindex, bp in enumerate(bodyparts2plot):
--> 198         df_x[bpindex,:]=xyz_pts[scorer_3d][bp]['x'].values
    199         df_y[bpindex,:]=xyz_pts[scorer_3d][bp]['y'].values
    200         df_z[bpindex,:]=xyz_pts[scorer_3d][bp]['z'].values

/home/ubuntu/anaconda3/lib/python3.6/site-packages/pandas/core/frame.py in __getitem__(self, key)
   2135             return self._getitem_frame(key)
   2136         elif is_mi_columns:
-> 2137             return self._getitem_multilevel(key)
   2138         else:
   2139             return self._getitem_column(key)

/home/ubuntu/anaconda3/lib/python3.6/site-packages/pandas/core/frame.py in _getitem_multilevel(self, key)
   2179 
   2180     def _getitem_multilevel(self, key):
-> 2181         loc = self.columns.get_loc(key)
   2182         if isinstance(loc, (slice, Series, np.ndarray, Index)):
   2183             new_columns = self.columns[loc]

/home/ubuntu/anaconda3/lib/python3.6/site-packages/pandas/core/indexes/multi.py in get_loc(self, key, method)
   2069 
   2070         if not isinstance(key, tuple):
-> 2071             loc = self._get_level_indexer(key, level=0)
   2072             return _maybe_to_slice(loc)
   2073 

/home/ubuntu/anaconda3/lib/python3.6/site-packages/pandas/core/indexes/multi.py in _get_level_indexer(self, key, level, indexer)
   2359         else:
   2360 
-> 2361             loc = level_index.get_loc(key)
   2362             if isinstance(loc, slice):
   2363                 return loc

/home/ubuntu/anaconda3/lib/python3.6/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2522                 return self._engine.get_loc(key)
   2523             except KeyError:
-> 2524                 return self._engine.get_loc(self._maybe_cast_indexer(key))
   2525 
   2526         indexer = self.get_indexer([key], method=method, tolerance=tolerance)

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'O'

my videospath is as following:

In [62]: videospath
Out[62]: '/work/dlc-project/prehen/'

Thanks

[MMathisLab]

Did you define the skeleton in the config.yaml file?

https://github.com/AlexEMG/DeepLabCut/blob/master/docs/Overviewof3D.md#4-triangulation----take-your-2d-to-3d

*also, trailpoints is optional, if you don't want them, but the skeleton is not.

[ also it's possible it doesn't work as you should not have spaces in file names; it's very hard to read the file names correctly in this format ]

[meet10may]

Seems it is an issue while setting up the skeleton variable in the config file. The skeleton parameters are passed as a list of list something like this

where the bodypart1 will be connected to the bodypart2, bodypart2 will be connected to the bodypart3 and so and so forth.
Check the docs here:
https://github.com/AlexEMG/DeepLabCut/blob/master/docs/Overviewof3D.md

This is the same issue which is resolved here:
https://forum.image.sc/t/how-can-i-solve-this-problem-with-create-labeled-video-3d-function/26907

Hope this helps!

[mawaishafeez]

that's my config.yaml file and have defined skeleton with 3 bodyparts
already had one network trained on all views from both cameras (3 bodyparts) therefore same path for the config file of both cameras

scorer: jack_250619
date: Jun25

# Project path (change when moving around)
project_path: /work/dlc-project/prehension-jack_250619-2019-06-25-3d

# Plotting configuration
skeleton:     # Note that the pairs must be defined, as you want them linked!
  - - Thumb
    - Wrist
  - - Finger
    - Wrist
  - Object

skeleton_color: black
pcutoff: 0.4
colormap: jet
dotsize: 15
alphaValue: 0.8
markerType: '*'
markerColor: r

# Number of cameras, camera names, path of the config files, shuffle index and trainingsetindex used to analyze videos:
num_cameras: 2
camera_names:
- camera-1
- camera-2
scorername_3d: DLC_3D # Enter the scorer name for the 3D output
trainingsetindex_camera-1: 0
trainingsetindex_camera-2: 0
config_file_camera-1: /work/dlc-project/Reaching-Awais-2019-04-24/config.yaml
config_file_camera-2: /work/dlc-project/Reaching-Awais-2019-04-24/config.yaml
shuffle_camera-1: 1
shuffle_camera-2: 1

[previously have used the same file names for 2D analyzing and labeling videos and it worked]

[meet10may]

You need to delete the - Object from the skeleton variable since you are not using it. I believe you want to connect the thumb to wrist and finger to wrist. So, in this case, you do not need the "Object"

[mawaishafeez]

Thanks a lot! @meet10may @MMathisLab ... Issue resolved

and during triangulate command, i forgot to use 'save_as_csv=True' and how can i go back and save csv for 3D dataset.

[MMathisLab]

I'm not sure we have a back up for the 3d.h5 .... but you could try running deeplabcut.analyze_videos_converth5_to_csv

otherwise, you can just delete only the 3d.h5 and it's very fast to re-compute (~ secs)

Signature: deeplabcut.analyze_videos_converth5_to_csv(videopath, videotype='.avi')
Docstring:
    By default the output poses (when running analyze_videos) are stored as MultiIndex Pandas Array, which contains the name of the network, body part name, (x, y) label position 

    in pixels, and the likelihood for each frame per body part. These arrays are stored in an efficient Hierarchical Data Format (HDF) 

    in the same directory, where the video is stored. If the flag save_as_csv is set to True, the data is also exported as comma-separated value file. However,
    if the flag was *not* set, then this function allows the conversion of all h5 files to csv files (without having to analyze the videos again)!

    This functions converts hdf (h5) files to the comma-separated values format (.csv), which in turn can be imported in many programs, such as MATLAB, R, Prism, etc.

    Parameters
    ----------

    videopath : string
        A strings containing the full paths to videos for analysis or a path to the directory where all the videos with same extension are stored.

    videotype: string, optional
        Checks for the extension of the video in case the input to the video is a directory.
Only videos with this extension are analyzed. The default is ``.avi``

    Examples
    --------

    Converts all pose-output files belonging to mp4 videos in the folder '/media/alex/experimentaldata/cheetahvideos' to csv files. 
    deeplabcut.analyze_videos_converth5_to_csv('/media/alex/experimentaldata/cheetahvideos','.mp4')