Deeplabcut: Copy labels from previous image

[nishbo]

Hi, we label a lot of images, and on some of them the markers barely move or don't move at all during some portions of movements, e.g. shoulder during finger manipulation. For that, I have written a small routine that checks several previous frames and copies the marker position from the previous image if the marker has not moved much.
This was inserted into generate_training_dataset/labeling_toolbox.py:572

        # copy over labels
        is_this_labeled = any([not np.isnan(self.dataFrame[self.scorer][bp]['x'].values[self.iter])
                               for bpindex, bp in enumerate(self.bodyparts)]) 
        num_prev_frames = 3
        pixel_distance_threshold = 10
        if self.iter > 1 and not is_this_labeled:
            start_copy_frame = max(1, self.iter-num_prev_frames)
            prev_labeled_bps = []
            prev_labeled_bpindexes = []
            for bpindex, bp in enumerate(self.bodyparts):
                if not np.isnan(self.dataFrame[self.scorer][bp]['x'].values[self.iter-1]):
                    prev_labeled_bps.append(bp)
                    prev_labeled_bpindexes.append(bpindex)
            for bpindex, bp in zip(prev_labeled_bpindexes, prev_labeled_bps):
                xs = []
                ys = []
                for frame_index in range(start_copy_frame, self.iter):
                    xs.append(self.dataFrame[self.scorer][bp]['x'].values[frame_index])
                    ys.append(self.dataFrame[self.scorer][bp]['y'].values[frame_index])
                if not any(np.isnan(x) for x in xs) and (np.std(xs)**2+np.std(ys)**2)**0.5 < pixel_distance_threshold:
                        self.rdb.SetSelection(bpindex)
                        event2 = wx._core.CommandEvent()
                        event2.xdata = self.dataFrame[self.scorer][bp]['x'].values[self.iter-1]
                        event2.ydata = self.dataFrame[self.scorer][bp]['y'].values[self.iter-1]
                        event2.button = 3
                        self.onClick(event2)

It has several parameters that should be available to the user, like:
num_prev_frames - number of frames to be checked for marker movement;
pixel_distance_threshold - allowed 2D deviation of the marker position for the marker to be considered "not moving".

It calls OnClick event to place the markers, which makes them appear gradually on the screen, and some users liked that as a visual indication of the procedure.

I am not sure of the best way to integrate it into DLC config, so I am just leaving it here as a suggestion.

[MMathisLab]

Thanks for the tip! but I would note, there is no reason to label frames that look identical though, the ideal is to find very diverse postures. More data that looks the same is not helpful, and can be harmful in the sense of overtraining on nearly identical postures...

[AlexEMG]

I actually like this suggestion a lot! It is useful, if e.g. only one bodypart moves across frames and all the others remain stable (or if one labels e.g. the boundary of a box). Then one only needs to move the one point... I think it would be good to work this into the GUI at some point! @jeylau what do you think?

[jeylau]

I would like a lot a simplified version of it: hitting a key (C?) to copy the keypoints from the previous frame on demand. This way we don't rely on the config and two very sensitive parameters, nor do we incur the growing cost of computing distances over many bodyparts. How does that sound?

[AlexEMG]

Sounds good to me; "c" and maybe also a button?

[jeylau]

Ctrl+C?

[AlexEMG]

@nishbo -- Thanks for this suggestion -- please check out the PR!