Caffe: Check failed: a <= b (0 vs. -1.19209e-07)

[fnzhan]

I am run ssd to train, but error happened in this step: solver = caffe.SGDSolver(caffe_root + 'models/VGGNet/ICDAR2017/SSD_300x300/solver.prototxt')

the trace stack is :
F0708 18:54:26.167244 20822 math_functions.cpp:250] Check failed: a <= b (0 vs. -1.19209e-07)
* Check failure stack trace: *
@ 0x7f84859db5ad google::LogMessage::Fail()
@ 0x7f84859dd413 google::LogMessage::SendToLog()
@ 0x7f84859db13b google::LogMessage::Flush()
@ 0x7f84859dddfe google::LogMessageFatal::~LogMessageFatal()
@ 0x7f8485fdc0e2 caffe::caffe_rng_uniform<>()
@ 0x7f848600ac36 caffe::SampleBBox()
@ 0x7f848600af80 caffe::GenerateSamples()
@ 0x7f848600b1b4 caffe::GenerateBatchSamples()
@ 0x7f8485f8ec28 caffe::AnnotatedDataLayer<>::load_batch()
@ 0x7f8485e90f2a caffe::BasePrefetchingDataLayer<>::InternalThreadEntry()

I have searched the similar issues and it maybe the precision problem, but it seems the problem is still unresolved. Is there any method? @weiliu89

[IamTechknow]

Hi, I have this problem too. You can get rid of it by uncommenting line 250 in $CAFFE_ROOT/src/caffe/util/math_functions.cpp and then rebuilding, but then I get a message about the "Data layer prefetch queue empty" when I run ssd_pascal.py. That's a separate issue but it's possible the two are correlated.

[IamTechknow]

I found a workaround actually for anyone who stumbles on this:

The fatal log indicates that the check has failed, well, what if we made sure it wouldn't fail? Go to $CAFFE_ROOT/src/caffe/util/math_functions.cpp and change the first five lines of caffe_rng_uniform() to:

void caffe_rng_uniform(const int n, Dtype a, Dtype b, Dtype* r) {
  CHECK_GE(n, 0);
  CHECK(r);

  if(a > b) {
    Dtype c = a;
    a = b;
    b = c;
  }
  CHECK_LE(a, b);

The primary changes are to make a and b not constant and to swap them if the if statement is true.

Rebuild (takes a few seconds thanks to incremental building), run the tests and examples to make sure they still pass before using SSD again, because you might uncover new problems.

[PumayHui]

Hi, I have this problem too. You can get rid of it by uncommenting line 250 in $CAFFE_ROOT/src/caffe/util/math_functions.cpp and then rebuilding, but then I get a message about the "Data layer prefetch queue empty" when I run ssd_pascal.py. That's a separate issue but it's possible the two are correlated.

Hi @IamTechknow
Could you please tell me how to solve the problem about "Data layer prefetch queue empty"? I met the same error……Thanks so much!

[jtdutta1]

I found a workaround actually for anyone who stumbles on this:

The fatal log indicates that the check has failed, well, what if we made sure it wouldn't fail? Go to $CAFFE_ROOT/src/caffe/util/math_functions.cpp and change the first five lines of caffe_rng_uniform() to:

void caffe_rng_uniform(const int n, Dtype a, Dtype b, Dtype* r) {
  CHECK_GE(n, 0);
  CHECK(r);

  if(a > b) {
    Dtype c = a;
    a = b;
    b = c;
  }
  CHECK_LE(a, b);

The primary changes are to make a and b not constant and to swap them if the if statement is true.

Rebuild (takes a few seconds thanks to incremental building), run the tests and examples to make sure they still pass before using SSD again, because you might uncover new problems.

Hi @IamTechknow , I tried doing this but during rebuilding, I got this error

Is there any workaround this?

[jtdutta1]

I found a workaround actually for anyone who stumbles on this:
The fatal log indicates that the check has failed, well, what if we made sure it wouldn't fail? Go to $CAFFE_ROOT/src/caffe/util/math_functions.cpp and change the first five lines of caffe_rng_uniform() to:

void caffe_rng_uniform(const int n, Dtype a, Dtype b, Dtype* r) {
  CHECK_GE(n, 0);
  CHECK(r);

  if(a > b) {
    Dtype c = a;
    a = b;
    b = c;
  }
  CHECK_LE(a, b);

The primary changes are to make a and b not constant and to swap them if the if statement is true.
Rebuild (takes a few seconds thanks to incremental building), run the tests and examples to make sure they still pass before using SSD again, because you might uncover new problems.

Hi @IamTechknow , I tried doing this but during rebuilding, I got this error

Is there any workaround this?

I found my mistake. I didn't remove the const keyword before the Dtype in the parameters space.