Deeplabcut: Key Error at Evaluate Results

[neurowookie]

System: Windows 10 Enterprise
DLC version:2.1 GPU(using environment install)

So I have managed to get almost everything working so far but am really stuck on a key error I get during evaluate results and I am not sure where its coming from as I am a bit new to Python.

This is when running evaluate through the GUI with plotting enabled.

>

[]
Config:
{'all_joints': [[0], [1], [2], [3]],
 'all_joints_names': ['tailbase', 'earL', 'earR', 'tailtip'],
 'batch_size': 1,
 'bottomheight': 400,
 'crop': True,
 'crop_pad': 0,
 'cropratio': 0.4,
 'dataset': 'training-datasets\\iteration-0\\UnaugmentedDataSet_squeaklowOct14\\squeaklow_steven95shuffle1.mat',
 'dataset_type': 'default',
 'deconvolutionstride': 2,
 'deterministic': False,
 'display_iters': 1000,
 'fg_fraction': 0.25,
 'global_scale': 0.8,
 'init_weights': 'D:\\Anaconda\\envs\\dlc-windowsGPU\\lib\\site-packages\\deeplabcut\\pose_estimation_tensorflow\\models\\pretrained\\resnet_v1_50.ckpt',
 'intermediate_supervision': False,
 'intermediate_supervision_layer': 12,
 'leftwidth': 400,
 'location_refinement': True,
 'locref_huber_loss': True,
 'locref_loss_weight': 0.05,
 'locref_stdev': 7.2801,
 'log_dir': 'log',
 'max_input_size': 1500,
 'mean_pixel': [123.68, 116.779, 103.939],
 'metadataset': 'training-datasets\\iteration-0\\UnaugmentedDataSet_squeaklowOct14\\Documentation_data-squeaklow_95shuffle1.pickle',
 'min_input_size': 64,
 'minsize': 100,
 'mirror': False,
 'multi_step': [[0.005, 10000],
                [0.02, 430000],
                [0.002, 730000],
                [0.001, 1030000]],
 'net_type': 'resnet_50',
 'num_joints': 4,
 'optimizer': 'sgd',
 'output_stride': 16,
 'pos_dist_thresh': 17,
 'project_path': 'D:\\DLC\\squeaklow-steven-2019-10-14',
 'regularize': False,
 'rightwidth': 400,
 'save_iters': 50000,
 'scale_jitter_lo': 0.5,
 'scale_jitter_up': 1.25,
 'scoremap_dir': 'test',
 'shuffle': True,
 'snapshot_prefix': 'D:\\DLC\\squeaklow-steven-2019-10-14\\dlc-models\\iteration-0\\squeaklowOct14-trainset95shuffle1\\test\\snapshot',
 'stride': 8.0,
 'topheight': 400,
 'weigh_negatives': False,
 'weigh_only_present_joints': False,
 'weigh_part_predictions': False,
 'weight_decay': 0.0001}
Running  DLC_resnet50_squeaklowOct14shuffle1_250000  with # of trainingiterations: 250000
Initializing ResNet
INFO:tensorflow:Restoring parameters from D:\DLC\squeaklow-steven-2019-10-14\dlc-models\iteration-0\squeaklowOct14-trainset95shuffle1\train\snapshot-250000
Analyzing data...
100it [00:13,  7.31it/s]
Done and results stored for snapshot:  snapshot-250000

KeyError                                  Traceback (most recent call last)
D:\Anaconda\envs\dlc-windowsGPU\lib\site-packages\deeplabcut\gui\evaluate_network.py in evaluate_network(self, event)
    158             plotting = False
    159         print(self.bodyparts)
--> 160         deeplabcut.evaluate_network(self.config,shuffle,trainingsetindex,plotting,show_errors=True,comparisonbodyparts=self.bodyparts,gputouse=None)
    161
    162     def cancel_evaluate_network(self,event):

D:\Anaconda\envs\dlc-windowsGPU\lib\site-packages\deeplabcut\pose_estimation_tensorflow\evaluate.py in evaluate_network(config, Shuffles, trainingsetindex, plotting, show_errors, comparisonbodyparts, gputouse, rescale)
    222                     DataCombined = pd.concat([Data.T, DataMachine.T], axis=0).T
    223
--> 224                     RMSE,RMSEpcutoff = pairwisedistances(DataCombined, cfg["scorer"], DLCscorer,cfg["pcutoff"],comparisonbodyparts)
    225                     testerror = np.nanmean(RMSE.iloc[testIndices].values.flatten())
    226                     trainerror = np.nanmean(RMSE.iloc[trainIndices].values.flatten())

D:\Anaconda\envs\dlc-windowsGPU\lib\site-packages\deeplabcut\pose_estimation_tensorflow\evaluate.py in pairwisedistances(DataCombined, scorer1, scorer2, pcutoff, bodyparts)
     29     else:
     30             Pointwisesquareddistance=(DataCombined[scorer1][bodyparts]-DataCombined[scorer2][bodyparts])**2
---> 31             RMSE=np.sqrt(Pointwisesquareddistance.xs('x',level=1,axis=1)+Pointwisesquareddistance.xs('y',level=1,axis=1)) #Euclidean distance (proportional to RMSE)
     32             return RMSE,RMSE[mask]
     33

D:\Anaconda\envs\dlc-windowsGPU\lib\site-packages\pandas\core\generic.py in xs(self, key, axis, level, drop_level)
   3715         labels = self._get_axis(axis)
   3716         if level is not None:
-> 3717             loc, new_ax = labels.get_loc_level(key, level=level, drop_level=drop_level)
   3718
   3719             # create the tuple of the indexer

D:\Anaconda\envs\dlc-windowsGPU\lib\site-packages\pandas\core\indexes\multi.py in get_loc_level(self, key, level, drop_level)
   2854                 return indexer, maybe_droplevels(indexer, ilevels, drop_level)
   2855         else:
-> 2856             indexer = self._get_level_indexer(key, level=level)
   2857             return indexer, maybe_droplevels(indexer, [level], drop_level)
   2858

D:\Anaconda\envs\dlc-windowsGPU\lib\site-packages\pandas\core\indexes\multi.py in _get_level_indexer(self, key, level, indexer)
   2944                 if not locs.any():
   2945                     # The label is present in self.levels[level] but unused:
-> 2946                     raise KeyError(key)
   2947                 return locs
   2948

KeyError: 'x' 

> 

[MMathisLab]

“ Analyzing data...
100it [00:13, 7.31it/s]
Done and results stored for snapshot: snapshot-250000”

• seems it maybe a permission issue for saving the plots; if plotting is false it works correctly?

• could you launch the terminal “as an admin” and see if that changes the outcome?

• also, how large are your frames?

[neurowookie]

Plotting false has the same result, and I already run it in admin as I get issues with other parts when I don't. Frame size = 722 x 538 . Thanks for the fast response

[MMathisLab]

Did you edit the pose_cfg yaml any body parts/joints? Is your config consistent with reapect to body part names?

[neurowookie]

Yes pose_cfg and config yaml are all matched up.

[MMathisLab]

then your .h5 must be corrupt, as it's not finding the "x" coordinate to compute the pairwise errors.

RMSE=np.sqrt(Pointwisesquareddistance.xs('x',level=1,axis=1)+Pointwisesquareddistance.xs('y',level=1,axis=1)) #Euclidean distance (proportional to RMSE)

[MMathisLab]

Any updates to your issue? @neurowookie

[neurowookie]

Hiya,

I ran it twice and same issue. I'm a bit confused as to how its corrupting each time. If I have time this week I'll try some other videos and if that doesn't work a fresh install and see where I get. Thanks for the help.

[MMathisLab]

going to link this here, as another issue reported: https://forum.image.sc/t/errors-in-network-evaluation/30756

[MMathisLab]

Hey @neurowookie, can you try just using the terminal, i.e delete any files in the evaulation folder directory, then:

ipython
import deeplabcut
cfp  = 'D:\\DLC\\squeaklow-steven-2019-10-14\\config.yaml'
deeplabcut.evalute_network(cfp) 

[neurowookie]

and now it works....

[MMathisLab]

I found an error in the GUI, updated now. Doesn’t effect your network or performance, just if you computed all body parts from the GUI. Fixed in repo, will be in next 2.1.3 update!

[MMathisLab]

fixed on commit:https://github.com/AlexEMG/DeepLabCut/commit/b989c665c72f3b02e871f51c13fc4c610d845a94