Caffe: Check failed: data_ MemoryDataLayer needs to be initialized by calling Reset

[Madgeeno]

I have a problem with feeding data to a MemoryDataLayerwhich is used as input for training a very simple network. I provide the data using the function addDatumVector. However, when I run the solver calling the Solve() method, I get the following error:

memory_data_layer.cpp:133] Check failed: data_ MemoryDataLayer needs to be initialized by calling Reset

Googling in the Caffe-user-group, I found that the following threads refer to same problem, which affect both the python and the C++ interface:
link1
link2

So, here is a snippet of my code, the solver.prototxt file is based on the mnist example:

std::string SolverFile = "path-to-solver/solver.prototxt";
  caffe::SolverParameter solver_param;
  caffe::ReadSolverParamsFromTextFileOrDie(SolverFile, &solver_param);
  // set mode
  Caffe::SetDevice(0);
  Caffe::set_mode(Caffe::GPU);
  Caffe::set_solver_count(1);
  // initialize solver with params from file
  boost::shared_ptr<caffe::Solver<float> >
    solver(caffe::SolverRegistry<float>::CreateSolver(solver_param));
  // get a shared_ptr to the MemoryDataLayer
  boost::shared_ptr<caffe::MemoryDataLayer<float> > 
    memoryData = boost::static_pointer_cast<caffe::MemoryDataLayer<float> >(solver->net().get()->layer_by_name("data"));
  if (!memoryData)
    LOG(FATAL) << "the first layer is not a memory layer, the pointer cast made the layer unusable";
  else
  {
    // fill a vector of Datum's with xor example
    std::vector<caffe::Datum> datumVector(memoryData->batch_size()); // batch_size (x num_iter ?)
    std::vector<char> datum0 = { '0', '0' };
    std::vector<char> datum1 = { '1', '0' };
    std::vector<char> datum2 = { '0', '1' };
    std::vector<char> datum3 = { '1', '1' };
    std::vector<std::vector<char> > Data = { datum0, datum1, datum2, datum3 };
    std::vector<int> Labels = { 0, 1, 1, 0 };
    for (size_t i = 0; i != datumVector.size(); ++i)
    {
      datumVector[i].set_channels(memoryData->channels());
      datumVector[i].set_height(memoryData->height());
      datumVector[i].set_width(memoryData->width());
      datumVector[i].set_data(&(Data[i][0]), Data[i].size());
      datumVector[i].set_label(Labels[i]);
    }    
    // add the data to the memory layer
    memoryData->AddDatumVector(datumVector);
  }
  LOG(INFO) << "Starting Optimization";
  solver->Solve(); // here I get the error
  LOG(INFO) << "Optimization Done.";

Error:

I0317 12:11:10.810009  5076 solver.cpp:288] Solving XOR
I0317 12:11:10.810009  5076 solver.cpp:289] Learning Rate Policy: step
I0317 12:11:10.810009  5076 solver.cpp:341] Iteration 0, Testing net (#0)
F0317 12:11:10.810508  5076 memory_data_layer.cpp:133] Check failed: data_ MemoryDataLayer needs to be initialized by calling Reset

Here is my xor.prototxt, called by solver.prototxt

name: "XOR"

layer {
  name: "data"
  type: "MemoryData"
  top: "data"
  top: "label"
  memory_data_param {
    batch_size: 4
    channels: 1
    height: 1
    width: 2
  }
}

I fear it's a bug in the Solve() method, performing a clean up on the initialized layers.
Many thanks in advance for any help!

Mattia

[supriya-k]

I am having the same issue. Were you able to solve this?

[twmht]

I am having the same issue.

Any update?

[willyd]

Have you tried using the InputLayer? This is what I use when doing inference from C++ or python.

[PaulDufort]

@willyd, would you mind posting (or linking to) an example of how to do this? I haven't been able to find much documentation on InputLayer.

[PaulDufort]

@willyd, does this mean just defining a layer of type Input, and then loading it with a new minibatch manually on each iteration? I suppose you would have to use the Step(1) approach with the solver one iteration at a time then instead of calling Solve(), correct?

[cesarsouza]

@PaulDufort did that approach work for you? I am facing the same problem with MemoryDataLayer calling either Step(1) or Solve(). I am about to test with the InputLayer.

[willyd]

@willyd, would you mind posting (or linking to) an example of how to do this? I haven't been able to find much documentation on InputLayer.

Sorry for being so quiet I did not have much time to spend on caffe lately. I will look in my examples and see if I can put something up on github.

@willyd, does this mean just defining a layer of type Input, and then loading it with a new minibatch manually on each iteration? I suppose you would have to use the Step(1) approach with the solver one iteration at a time then instead of calling Solve(), correct?

Yes this is correct. Since I work mostly from python I usually have a python layer that feeds the tops during training so Solve() will just work too and keep the Input layer for inference where my minibatch often contains a single image anyway.

[PaulDufort]

Hi @cesarsouza, I haven't tried it yet and it sounds from @willyd like there might be a better way. Using Input Layer for inference is described already in caffe/examples/00-classification, so it was using it for learning that I really wanted to know about. But @willyd describes above using a (home-cooked?) python layer in that context, and only using InputLayer for inference.

@willyd, is it possible you could share the python layer you've created to feed data in during learning? That would be a big help.

Thanks,
Paul

[willyd]

@PaulDufort I will try to set something up as a gist or git repo ASAP.

[PaulDufort]

Thanks @willyd, that would be great!

[willyd]

@PaulDufort I just uploaded a simple example that uses a python layer to feed data during training and an Input layer during inference. You can get the code here: https://github.com/willyd/caffe-examples/tree/master/linear_regression.

The network performs a linear regression on a set of x, y points. The python layer always feeds the same data but in a real scenario you would load a batch and feed that batch in the forward method. I am using the caffe.NetSpec to generate the prototxt files so if you are not sure how this translates to prototxt files run the code once and then inspect the prototxt files.

Let me know if you have questions.

[PaulDufort]

Hey @willyd, thanks a lot for this! I am still relatively new to both Python and Caffe, so I will be able to learn a lot from it. I'm very happy to see that it all looks pretty straightforward to understand. I've never used yaml so I'll have to have a look at that. @cesarsouza, I would imagine this will help you out as well.

[shelhamer]

Closing according to #5528.