Caffe: failed to start training from snapshot

[yossibiton]

i run a training for about 40k iterations.
than i stopped it and tried to train again (with lower lr) from snapshot of iteration 17k.

command :
$CAFFE_ROOT/build/tools/caffe train --solver ../../nets/net1/solver.prototxt --snapshot ../../nets-weights/net1/net__iter_17000.caffemodel

i used the same solver.prototxt i used in the first training phase, except for changing base_lr from 0.001 to 0.0001 (lr_policy: "fixed").

I get the following error :
I0519 19:16:09.539479 23277 caffe.cpp:209] Resuming from ../../nets-weights/net1-data02_160x064/net__iter_17000.caffemodel F0519 19:16:09.551967 23277 sgd_solver.cpp:316] Check failed: state.history_size() == history_.size() (0 vs. 14) Incorrect length of history blobs.

my suspect is that caffe can't find the ".solverstate" file, although it's there right next to the ".caffemodel" snapshot file.

[shelhamer]

See our command line interface examples: http://caffe.berkeleyvision.org/tutorial/interfaces.html . The --snapshot arg takes the .solverstate.

From https://github.com/BVLC/caffe/blob/master/CONTRIBUTING.md:

_Please do not post usage, installation, or modeling questions, or other requests for help to Issues._
Use the caffe-users list instead. This helps developers maintain a clear, uncluttered, and efficient view of the state of Caffe.

[yossibiton]

Oops, my mistake.
I will just use this opportunity and ask to add input arguments
documentation when typing 'caffe train'.

On Thu, May 19, 2016, 19:31 Evan Shelhamer [email protected] wrote:

See our command line interface examples:
http://caffe.berkeleyvision.org/tutorial/interfaces.html . The --snapshot
arg takes the .solverstate.

From https://github.com/BVLC/caffe/blob/master/CONTRIBUTING.md:

_Please do not post usage, installation, or modeling questions, or other
requests for help to Issues._
Use the caffe-users list
https://groups.google.com/forum/#!forum/caffe-users instead. This helps
developers maintain a clear, uncluttered, and efficient view of the state
of Caffe.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/BVLC/caffe/issues/4181#issuecomment-220379249