Caffe: caffe-parallel has a bug: Check failed: proto.SerializeToOstream(&output)

[wkal]

Hello, everyone! Today I attempted the mnist example of caffe-parallel, it runs ok but when start to save the snapshot of model then error occurs, the error info say some thing wrong in the io.cpp(line66). Thanks!
The detail of the error is below:

I1104 12:49:05.119057 31209 solver.cpp:322] Snapshotting to examples/mnist/lenet_iter_10000.caffemodel
F1104 12:49:05.126173 31209 io.cpp:66] Check failed: proto.SerializeToOstream(&output)
* Check failure stack trace: *
@ 0x7fc2eef6cdaa (unknown)
@ 0x7fc2eef6cce4 (unknown)
@ 0x7fc2eef6c6e6 (unknown)
@ 0x7fc2eef6f687 (unknown)
@ 0x45e33c caffe::WriteProtoToBinaryFile()
@ 0x4982e1 caffe::Solver<>::Snapshot()
@ 0x4991f4 caffe::Solver<>::Solve()
@ 0x415f7b train()
@ 0x410981 main
@ 0x7fc2eaa57ec5 (unknown)
@ 0x414ab7 (unknown)
@ (nil) (unknown)
Aborted (core dumped)

[mees]

I have the same bug, did anyone solved it?

[shelhamer]

This usually means the snapshot directory doesn't exist -- nothing more.

[mees]

In my case it was related to not having permisions to write in that directory.

[zeakey]

have no permission to the directory or the directory not exist!

[matrlot]

This means that you have problem about writing snapshot.

There may be 2 solutions

1. Solution is to modify src/caffe/layers/data_layer.cpp by setting options.max_open_files = 100 (or any number significantly lower than 1000)
   according to https://github.com/BVLC/caffe/issues/38
2. your disk is full
   try to see the state of your disk by "df -a"

in my case, the solution was second one.

[nageshkuba]

I too faced this issue and the problem was in the naming of snapshot directory in solver file or in other words the directory did not exist.

[thanhnguyentang]

You must make sure you have manually created the snapshot directory before hand. Worked for me.

[IsabelEric]

I want to know the detail about add snapshot directory in pva-faster-rcnn