Deeplabcut: Frame extraction that displays labeled videos to improve predictions

[altear]

Feature Request: Frame extraction that displays labeled data

Problem

When a DLC model does not correctly label a body part, new data should be added to the training set in order to improve the model's prediction. However deeplabcut has no tool to extract frames while viewing the labeled frame.

One major problem is that frames that appear quite similar to humans (and happen near each other) may have differences that cause the model to make very different predictions. Meaning that not saving the exact frame where incorrect predictions are made may cause irrelevant frames to be saved (ones that are predicted correctly). Another is that it's time-consuming to convert seconds from a video player (for viewing the labeled video) to an approximate frame, and then try to find the exact point using code or the DLC frame extractor. For high FPS videos with fast movements, a 1-second resolution may cause you to search up to 120 frames.

Solution

Add the ability to see labeled data when extracting unlabeled frames

Alternatives considered

As described above, the best available alternative right now seems to be viewing the labeled videos and manually converting the time to frames and then attempting to find the frame using the frame-extractor or code. However there are problems with this method and it is inconvenient.

---

as an aside, if I need this feature sooner than others can get around to it and make it myself- would it be useful as a contribution?

[jeylau]

Hello @altear, passing outlieralgorithm='manual' to extract_outlier_frames seems to achieve what you describe. You get a GUI that allows you to navigate through the labeled frames and grab only those you want.