Deeplabcut: How to improve the performace of my labeling?

[EstherXSD]

Is your feature request related to a problem? Please describe.
Hi, I'm a freshman for the DeepLabCut. I'm working to label the forelimb when the mouse is doing the lever-press task.

Previously I've compared multpile networks and hyperparameters to do the labeling. I did 200,000 interations for each, where I labelled 100 frames manually in advance. My best performance is the following.

The net_type is Mobilenet_v2_1.0, augmenter_type is default.
However, when I explore the labeling in the video, I found that when the mouse is moving crazily, points could not follow the movement.

Describe the solution you'd like
So I extract the outlier frames and retrain the network. To my surprise, I found that the test error increased almost 10 folds.

I cannot figure out what's wrong during the processes.

Describe alternatives you've considered
I'm wondering whether the test error around 7 cannot be reduced anymore.
Is it possible to add skeletons to improve the labeling performance? My postdoc suggests that I could turn to MATLAB to exclude those "crazy time" but I haven't equipped myself with that skill. Is it feasible?

Additional context
I found that there is a new paper titled "Real-time, low-latency closed-loop feedback using markerless posture tracking", and I'm wondering whether this new technique could make the improvement.

Sincerely yours,
Confusing Esther 😞

[MMathisLab]

HI Esther, what version of DeepLabCut are you using?

You should train your initial network longer; you can see from the data that it is going down (and with only 5 frames for testing, it's hard to know really what the performance is!); the default is 1M iterations.

For the outlier, that looks like an issue; can you check labels before you train to be sure it looks correct?

Also filter the data after video analysis, and this will help a lot, as well as use a higher pcutoff, like 0.9.

Lastly, looks like you are from TK lab, no? If so, use the pretrained forelimb network; if you need a mobilenet, send me an email.

[EstherXSD]

Hi MMathisLab,
Words couldn't convey my appreciation for your detailed and patient respsonses. Now I have some new ideas about what I can do. The information of my system is as the following:
OS: Ubuntu 18.04.1
DeepLabCut Version: 2.1.8.2
Anaconda env used: GPU
About the iterations, I noticed that in the DLC project documents it mentions that "CRITICAL POINT: It is recommended to train the ResNets or MobileNets for thousands of iterations until the loss plateaus (typically around 200,000) if you use batch size 1." I'm wondering whether running 1M iterations will render my network overfitted. I could try 300,000 iterations or more later. Also, 1M iterations would cost me a lot of time. 😔
I will check the labels,change the pcutoff&train_test_split parameter in the configuration file, and filtered=True for my next training.
Above all, you are so smart and your guess is correct 😉 I'm an undergraduate student from TK lab. If you have pretrained forelimb network, that will help me a lot! (And obviously, I'm not that familiar with loading pretained network, and would explore further about that. If you have some information, I'm grateful.)

Sincerely yours,
A little bit clear-headed Esther 😌

[EstherXSD]

Hi Mackenzie Mathis,
I've also send you a gmail requesting for the pretailed network. Looking forward to your reply!
Best,
Esther

[EstherXSD]

Hi MMathisLab,

This time I tried network (The net_type is Mobilenet_v2_1.0, augmenter_type is default.) with 400,000 iterations, 0.9 pcut-off, 170 frames labelled manually, and 0.9 train_test_split percentage. I got the following results:

Compared with the previous one in the first comment, the test error even increased, even though the loss has been decreased to 0.0013 and lr to 0.02. Although the train error decreased as the number of iterations increased, the test error remained similar, suggesting that the network was overfitted?
I cannot wait to exploit your pretrained forelimb network for another trial. Looking forward to your email!

Sincerely yours
Esther

[MMathisLab]

Hi @EstherXSD will get back to you later this week, but sorry for any confusion; ( gave the network to Bin Yu in your lab already ;). But I will follow up by email as well.

in answer though, the numbers can always be different as it's a different shuffle -- i.e. the split of data is different, so it's only a few frames you are "testing" on, and that can be variable. It's always best to look at the scoremaps and then the actual videos :) too