Deeplabcut: Loading Checkpoint during training

[ForgottenOneNyx]

Hello, I had a question regarding the how we could load the checkpoints if we have paused the training, I understand it could be an issue for Pose-Tensorflow, but I just feel you might know how to solve it!

My last step was use train.py

Thank you in advance!

[MMathisLab]

I am quoting from: https://github.com/AlexEMG/DeepLabCut/blob/master/docs/demo-guide.md

"You can also stop during a training (Ctrl-C), and restart from a snapshot (aka checkpoint): Just change the init_weights term, i.e. instead of "init_weights: ../../pretrained/resnet_v1_50.ckpt" put "init_weights: ./snapshot-insertthe#ofstepshere" (i.e. 10,000). We recommend training for thousands of iterations until the loss plateaus (typically >200,000 - see Fig 2 and Supp Fig 2 in our paper)."

[ForgottenOneNyx]

I apologize, I might have missed it. Thank you so much!

[PerformanceGenetics]

So I am clear, the change in "init_weights:" value in pose_cfg.yaml (found in .../dlc-models/iteration-0/-trainset95shuffle1/train/

Should go from /usr/local/lib/python3.6/dist-packages/deeplabcut/pose_estimation_tensorflow/models/pretrained/resnet_v1_50.ckpt

to (as example)

.../dlc-models/iteration-0/-trainset95shuffle1/train/snapshot-80000.meta

?

[MMathisLab]

Without the ending! models/iteration-0/-trainset95shuffle1/train/snapshot-80000