Caffe: Blob size exceeds INT_MAX

[PiranjaF]

When running my code I get the following error blob.cpp:29] Check failed: shape[i] <= 2147483647 / count_ (24 vs. 17) blob size exceeds INT_MAX, which I've found in the source code as seen below. However, I am curious as to how to circumvent this limit, which seems rather low. If I e.g. use 200x200 images, then I can only input 2147483647 / 40000 = 53687 images. Is this a bug?

I am currently using HDF5 as my data source. Would it make any difference if I used another data source instead?

template <typename Dtype>
void Blob<Dtype>::Reshape(const vector<int>& shape) {
  CHECK_LE(shape.size(), kMaxBlobAxes);
  count_ = 1;
  shape_.resize(shape.size());
  for (int i = 0; i < shape.size(); ++i) {
    CHECK_GE(shape[i], 0);
    CHECK_LE(shape[i], INT_MAX / count_) << "blob size exceeds INT_MAX";
    count_ *= shape[i];
    shape_[i] = shape[i];
  }
  if (count_ > capacity_) {
    capacity_ = count_;
    data_.reset(new SyncedMemory(capacity_ * sizeof(Dtype)));
    diff_.reset(new SyncedMemory(capacity_ * sizeof(Dtype)));
  }
}

[seanbell]

See #1470 and #2953. This is a limitation of the current implementation of HDF5DataLayer. Other data layers don't have this problem (e.g. DataLayer, ImageDataLayer, or writing a PythonLayer that outputs data).

[PiranjaF]

@seanbell Thanks! I don't why I couldn't find it with a Google search.

[vra]

@seanbell Thanks for your explanation, but I wonder is there any utility to split that big hdf5 file into small ones? I googled it but did not find a good solution.

[lukeyeager]

@vra you can try the DIGITS create_db tool:

https://github.com/NVIDIA/DIGITS/blob/digits-3.0/tools/create_db.py

tools/create_db.py train.txt train_db 28 28 -b hdf5 --hdf5_dset_limit=2147483648

[vra]

@lukeyeager Thanks! that's helpful.

[arasharchor]

@seanbell @vra @lukeyeager @PiranjaF
I am experiencing the same problem. After searching through the internet, I noticed the blob format should be changed to avoid INT_MAX problem. As I am using faster_rcnn algorithm to detect object, DIGIT is not possible to be used. My images are big (5kx3k). I do not have GPU memory problem. How I can solve this problem?

In Faster RCNN they are using python input layer and not HDF5 layer. So why this problem happen?

F1214 18:32:32.298799 27802 blob.cpp:33] Check failed: shape[i] <= 2147483647 / count_ (3744 vs. 663) blob size exceeds INT_MAX

[naibaf7]

@smajida
You can try to use the OpenCL branch with USE_CUDA=1, USE_GREENTEA=0 and USE_INDEX64=1, it allows full 64 bit address space (above INT_MAX) and shapes in all critical areas of Caffe.
Haven't tested though if it will also work with larger HDF5 files, but it might.

[vra]

Hi @smajida ,
Are you using rgb's py-faster-rcnn implementation? If so , you can check the code in lib/fast_rcnn/config.py and resize your image since your image is too large. The related lines are this and this. Hope this helps.

[arasharchor]

@naibaf7

You can try to use the OpenCL branch with USE_CUDA=1, USE_GREENTEA=0 and USE_INDEX64=1, it allows full 64 bit address space (above INT_MAX) and shapes in all critical areas of Caffe. Haven't tested though if it will also work with larger HDF5 files, but it might.

Thanks. I am on it. I will give you feedback as soon as I am done.
BTW, my images are RGB (5616x3744,3) with object sizes of around (20x10x3)

@vra

Are you using rgb's py-faster-rcnn implementation? If so , you can check the code in lib/fast_rcnn/config.py and resize your image since your image is too large. The related lines are this and this. Hope this helps.

Hi and thanks. Yes, I am using rgb implementation. Images are RGB (5616x3744,3) with object sizes of around (20x10x3). First I can not change the size of images as the objects are very small (~ 20x10x3)and resizing would have negative effect. I can not also split the image as the Ground truth has been provided for the original image and I thought it would be less efficient to split the image as I would miss some objects at the splitting border. I had already changed the size of images in the following lines from (600,1000) to (5616,3744):

# Each scale is the pixel size of an image's shortest side
__C.TRAIN.SCALES = (5616,)
# Max pixel size of the longest side of a scaled input image
__C.TRAIN.MAX_SIZE = 3744

When training the network, I can see the original image is being read:

I1214 18:32:32.067646 27802 net.cpp:150] Setting up input-data
I1214 18:32:32.067694 27802 net.cpp:157] Top shape: 1 3 5616 3744 (63078912)
I1214 18:32:32.067699 27802 net.cpp:157] Top shape: 1 3 (3)
I1214 18:32:32.067703 27802 net.cpp:157] Top shape: 1 4 (4)

I think if I can be able to increase the blob size (above INT_MAX), it would be the most efficient way to do so as @naibaf7 has mentioned. I have not started doing that, but as soon as I am done, I will write you back.

BTW, a small typo in the previous post: I wanted to write down "I do not have GPU memory problem"

[vra]

Hi @smajida ,
Thanks for your clear explanation, I notice that you set TRAIN.SCALES larger than TRAIN.MAX_SIZE, however, from the comments above the lines we can find that the former should be smaller than or equal to the latter, because the former one is the target size the shorter side resizes to, while the latter one is the max size that the larger side can resize to.

[arasharchor]

Hi @vra
Thanks. I changed those parameters to the following ones:

# Scales to use during training (can list multiple scales)
# Each scale is the pixel size of an image's shortest side
__C.TRAIN.SCALES = (5616,)

# Max pixel size of the longest side of a scaled input image
__C.TRAIN.MAX_SIZE = 5616

But I see the wrong image size is being read i.e. instead of 5616x3744x3 now 5616x5616 is being read.

I1215 09:05:41.145800 22868 net.cpp:150] Setting up input-data
I1215 09:05:41.145840 22868 net.cpp:157] Top shape: 1 3 5616 5616 (94618368)
I1215 09:05:41.145849 22868 net.cpp:157] Top shape: 1 3 (3)
I1215 09:05:41.145858 22868 net.cpp:157] Top shape: 1 4 (4)

Is 3x5616x5616 a four channel blob with CHW dimension i.e. BGR? If yes, I swapped 5616 with 3744

I1215 09:12:35.132300 26293 net.cpp:150] Setting up input-data
I1215 09:12:35.132341 26293 net.cpp:157] Top shape: 1 3 3744 5616 (63078912)
I1215 09:12:35.132346 26293 net.cpp:157] Top shape: 1 3 (3)
I1215 09:12:35.132349 26293 net.cpp:157] Top shape: 1 4 (4)

In both cases I am still experiencing blob size exceeds INT_MAX problem.

first case: Check failed: shape[i] <= 2147483647 / count_ (5616 vs. 995) blob size exceeds INT_MAX
second case: Check failed: shape[i] <= 2147483647 / count_ (5616 vs. 995) blob size exceeds INT_MAX

In the comments in the beginning of config.py, it says

This file specifies default config options for Fast R-CNN. You should not
change values in this file. Instead, you should write a config file (in yaml)
and use cfg_from_file(yaml_file) to load it and override the default options.

shall I change the cfg_from_file(yaml_file) to modify the size of input image? does it make any difference?

@naibaf7
I have changed the parameters which you said, but how about other parameters. I do not know whether I should leave them unchanged or no. Shall I change the others as well? if yes how please?
here is Makefile.config before USE_CUDNN := 1 which afterwards is the same as caffe version of pyfaster rcnnwhich I am using.

# 32 bit / 64 bit indexing
USE_INDEX_64 := 1

# GreenTea (ViennaCL/OpenCL) backend switch
# Enable the CUDA backend
USE_CUDA := 1

# Enable the OpenCL/Greentea backend
USE_GREENTEA := 0

# Enable the Greentea-LibDNN convolution backend
# USE_LIBDNN := 1

# Enable the Intel spatial convolutions
# USE_INTEL_SPATIAL := 1

# Folder of the ViennaCL header-only library
VIENNACL_DIR = ../ViennaCL

# Override BLAS, use CLBlast instead of ViennacLBLAS.
# USE_CLBLAST := 1

# Custom CLBlast lib and include directories.
# CLBLAST_INCLUDE := /path/to/clblast/include
# CLBLAST_LIB     := /path/to/clblast/lib

# Override BLAS, use clBLAS insead of ViennaclBLAS.
# USE_CLBLAS := 1

# Custom clBLAS lib and include directories.
# CLBLAS_INCLUDE := /path/to/clblas/include
# CLBLAS_LIB     := /path/to/clblas/lib

# Override BLAS, use ISAAC instead of ViennaclBLAS.
# USE_ISAAC := 1

# Uncomment for FFT
# USE_FFT := 1

@naibaf7
I did what you said, but I am getting the following error:

AR -o .build_release/lib/libcaffe.a
LD -o .build_release/lib/libcaffe.so.1.0.0-rc3
CXX examples/cpp_classification/classification.cpp
CXX/LD -o .build_release/tools/caffe.bin
CXX/LD -o .build_release/tools/compute_image_mean.bin
CXX/LD -o .build_release/tools/convert_imageset.bin
CXX/LD -o .build_release/tools/device_query.bin
CXX/LD -o .build_release/tools/extract_features.bin
CXX/LD -o .build_release/tools/finetune_net.bin
CXX/LD -o .build_release/tools/net_speed_benchmark.bin
CXX/LD -o .build_release/tools/test_net.bin
CXX/LD -o .build_release/tools/train_net.bin
CXX/LD -o .build_release/tools/upgrade_net_proto_binary.bin
CXX/LD -o .build_release/tools/upgrade_net_proto_text.bin
CXX/LD -o .build_release/tools/upgrade_solver_proto_text.bin
CXX/LD -o .build_release/examples/cifar10/convert_cifar_data.bin
CXX/LD -o .build_release/examples/siamese/convert_mnist_siamese_data.bin
CXX/LD -o .build_release/examples/mnist/convert_mnist_data.bin
examples/cpp_classification/classification.cpp: In function 'std::vector<int> Argmax(const std::vector<float>&, int64_t)':
examples/cpp_classification/classification.cpp:124:10: error: could not convert 'result' from 'std::vector<long int>' to 'std::vector<int>'
   return result;
          ^
examples/cpp_classification/classification.cpp: In member function 'std::vector<std::pair<std::basic_string<char>, float> > Classifier::Classify(const cv::Mat&, int64_t)':
examples/cpp_classification/classification.cpp:132:46: error: conversion from 'std::vector<int>' to non-scalar type 'std::vector<long int>' requested
   std::vector<int_tp> maxN = Argmax(output, N);
                                              ^
make: *** [.build_release/examples/cpp_classification/classification.o] Error 1

[naibaf7]

@smajida
The other options should be fine. You can enable USE_LIBDNN if you want faster pooling backpropagation. And of course make sure to turn on USE_CUDNN for the fast convolutions.
Okay, that's only the classification binary having an issue, I'll fix that quickly.

Update: fixed.

[arasharchor]

@naibaf7
Thanks for your help. I tried the update version and made sure that your recent commits are included in the downloaded file. Unfortunately, I am still getting the same error:

examples/cpp_classification/classification.cpp: In function 'std::vector<int> Argmax(const std::vector<float>&, int64_t)':
examples/cpp_classification/classification.cpp:124:10: error: could not convert 'result' from 'std::vector<long int>' to 'std::vector<int>'
   return result;
          ^
examples/cpp_classification/classification.cpp: In member function 'std::vector<std::pair<std::basic_string<char>, float> > Classifier::Classify(const cv::Mat&, int64_t)':
examples/cpp_classification/classification.cpp:132:46: error: conversion from 'std::vector<int>' to non-scalar type 'std::vector<long int>' requested
   std::vector<int_tp> maxN = Argmax(output, N);
make: *** [.build_release/examples/cpp_classification/classification.o] Error 1

BTW, I am using Linux Ubuntu 16.04 x64 LTS server version with CUDA 7.5 and Nvidia Titan.

Update1: I removed classification.cpp from examples folder and Caffe was compiled and tested without any errors. I am trying to use Faster_RCNN which is an algorithm to do object detection. When running my algorithm which was running on the previous caffe with small image without INT_MAX problem. Now I am getting the following error using small images:

I1216 08:05:50.066968  7669 solver.cpp:98] Creating training net from train_net file: models/pascal_voc/VGG16/faster_rcnn_end2end/train.prototxt
[libprotobuf ERROR google/protobuf/text_format.cc:245] Error parsing text-format caffe.NetParameter: 470:24: Message type "caffe.LayerParameter" has no field named "smooth_l1_loss_param".
F1216 08:05:50.068133  7669 upgrade_proto.cpp:88] Check failed: ReadProtoFromTextFile(param_file, 
eadProtoFromTextFile(param_file, param) Failed to parse NetParameter file: models/pascal_voc/VGG16/faster_rcnn_end2end/train.prototxt
*** Check failure stack trace: ***
./experiments/scripts/faster_rcnn_end2end.sh: line 61:  7669 Aborted                 (core dumped) ./tools/train_net.py --gpu ${GPU_ID} --solver models/${PT_DIR}/${NET}/faster_rcnn_end2end/solver.prototxt --weights data/imagenet_models/${NET}.v22.caffemodel --imdb ${TRAIN_IMDB} --iters ${ITERS} --cfg experiments/cfgs/faster_rcnn_end2end.yml ${EXTRA_ARGS}

It seems opencl version does not have smooth_l1_loss_param.

Update2: I replaced smooth_l1_loss_layer.cpp and smooth_l1_loss_layer.cu as well as fast_rcnn_layer.hpp to their corresponding folder in OpenCL version of Caffe. This time caffe is having errors in compiling smooth_l1_loss_layer.cpp.

NVCC src/caffe/layers/smooth_L1_loss_layer.cu
src/caffe/layers/smooth_L1_loss_layer.cpp: In member function 'virtual void caffe::SmoothL1LossLayer<Dtype>::LayerSetUp(const std::vector<caffe::Blob<Dtype>*>&, const std::vector<caffe::Blob<Dtype>*>&)':
src/caffe/layers/smooth_L1_loss_layer.cpp:15:3: error: 'SmoothL1LossParameter' was not declared in this scope
   SmoothL1LossParameter loss_param = this->layer_param_.smooth_l1_loss_param();
   ^
src/caffe/layers/smooth_L1_loss_layer.cpp:15:25: error: expected ';' before 'loss_param'
   SmoothL1LossParameter loss_param = this->layer_param_.smooth_l1_loss_param();
                         ^
src/caffe/layers/smooth_L1_loss_layer.cpp:16:13: error: 'loss_param' was not declared in this scope
   sigma2_ = loss_param.sigma() * loss_param.sigma();
             ^
In file included from src/caffe/layers/smooth_L1_loss_layer.cpp:8:0:
./include/caffe/fast_rcnn_layers.hpp: In instantiation of 'class caffe::SmoothL1LossLayer<float>':
src/caffe/layers/smooth_L1_loss_layer.cpp:67:1:   required from here
./include/caffe/fast_rcnn_layers.hpp:72:22: error: conflicting return type specified for 'int caffe::SmoothL1LossLayer<Dtype>::ExactNumBottomBlobs() const [with Dtype = float]'
   virtual inline int ExactNumBottomBlobs() const { return -1; }
                      ^
In file included from ./include/caffe/fast_rcnn_layers.hpp:18:0,
                 from src/caffe/layers/smooth_L1_loss_layer.cpp:8:
./include/caffe/layers/loss_layer.hpp:32:25: error:   overriding 'int64_t caffe::LossLayer<Dtype>::ExactNumBottomBlobs() const [with Dtype = float; int64_t = long int]'
   virtual inline int_tp ExactNumBottomBlobs() const { return 2; }
                         ^
In file included from src/caffe/layers/smooth_L1_loss_layer.cpp:8:0:
./include/caffe/fast_rcnn_layers.hpp:73:22: error: conflicting return type specified for 'int caffe::SmoothL1LossLayer<Dtype>::MinBottomBlobs() const [with Dtype = float]'
   virtual inline int MinBottomBlobs() const { return 2; }
                      ^
In file included from ./include/caffe/fast_rcnn_layers.hpp:15:0,
                 from src/caffe/layers/smooth_L1_loss_layer.cpp:8:
./include/caffe/layer.hpp:248:25: error:   overriding 'int64_t caffe::Layer<Dtype>::MinBottomBlobs() const [with Dtype = float; int64_t = long int]'
   virtual inline int_tp MinBottomBlobs() const {
                         ^
In file included from src/caffe/layers/smooth_L1_loss_layer.cpp:8:0:
./include/caffe/fast_rcnn_layers.hpp:74:22: error: conflicting return type specified for 'int caffe::SmoothL1LossLayer<Dtype>::MaxBottomBlobs() const [with Dtype = float]'
   virtual inline int MaxBottomBlobs() const { return 4; }
                      ^
In file included from ./include/caffe/fast_rcnn_layers.hpp:15:0,
                 from src/caffe/layers/smooth_L1_loss_layer.cpp:8:
./include/caffe/layer.hpp:258:25: error:   overriding 'int64_t caffe::Layer<Dtype>::MaxBottomBlobs() const [with Dtype = float; int64_t = long int]'
   virtual inline int_tp MaxBottomBlobs() const {
                         ^
In file included from src/caffe/layers/smooth_L1_loss_layer.cpp:8:0:
./include/caffe/fast_rcnn_layers.hpp: In instantiation of 'class caffe::SmoothL1LossLayer<double>':
src/caffe/layers/smooth_L1_loss_layer.cpp:67:1:   required from here
./include/caffe/fast_rcnn_layers.hpp:72:22: error: conflicting return type specified for 'int caffe::SmoothL1LossLayer<Dtype>::ExactNumBottomBlobs() const [with Dtype = double]'
   virtual inline int ExactNumBottomBlobs() const { return -1; }
                      ^
In file included from ./include/caffe/fast_rcnn_layers.hpp:18:0,
                 from src/caffe/layers/smooth_L1_loss_layer.cpp:8:
./include/caffe/layers/loss_layer.hpp:32:25: error:   overriding 'int64_t caffe::LossLayer<Dtype>::ExactNumBottomBlobs() const [with Dtype = double; int64_t = long int]'
   virtual inline int_tp ExactNumBottomBlobs() const { return 2; }
                         ^
In file included from src/caffe/layers/smooth_L1_loss_layer.cpp:8:0:
./include/caffe/fast_rcnn_layers.hpp:73:22: error: conflicting return type specified for 'int caffe::SmoothL1LossLayer<Dtype>::MinBottomBlobs() const [with Dtype = double]'
   virtual inline int MinBottomBlobs() const { return 2; }
                      ^
In file included from ./include/caffe/fast_rcnn_layers.hpp:15:0,
                 from src/caffe/layers/smooth_L1_loss_layer.cpp:8:
./include/caffe/layer.hpp:248:25: error:   overriding 'int64_t caffe::Layer<Dtype>::MinBottomBlobs() const [with Dtype = double; int64_t = long int]'
   virtual inline int_tp MinBottomBlobs() const {
                         ^
In file included from src/caffe/layers/smooth_L1_loss_layer.cpp:8:0:
./include/caffe/fast_rcnn_layers.hpp:74:22: error: conflicting return type specified for 'int caffe::SmoothL1LossLayer<Dtype>::MaxBottomBlobs() const [with Dtype = double]'
   virtual inline int MaxBottomBlobs() const { return 4; }
                      ^
In file included from ./include/caffe/fast_rcnn_layers.hpp:15:0,
                 from src/caffe/layers/smooth_L1_loss_layer.cpp:8:
./include/caffe/layer.hpp:258:25: error:   overriding 'int64_t caffe::Layer<Dtype>::MaxBottomBlobs() const [with Dtype = double; int64_t = long int]'
   virtual inline int_tp MaxBottomBlobs() const {
                         ^
make: *** [.build_release/src/caffe/layers/smooth_L1_loss_layer.o] Error 1

[jiayi-wei]

@smajida Hi~ Have you solved your problem? I'm trying to package my Faste-RCNN project to C++. Undoudtedly, I can train or test my net in python. It runs well. In C++. however, i met a error like this
F0511 10:25:19.911154 8752 blob.cpp:33] Check failed: shape[i] <= 2147483647 / count_ (3200 vs. 1553) blob size exceeds INT_MAX
The size of my image is 1600*1200.
Thanks very much for any help！

[enduringstack]

@jiayi-wei Have you solved your problem? I hava the same problem as you.

[naibaf7]

@jiayi-wei If your image is of size 3 (RGB) * 1600 * 1200 then this is obviously too large for standard Caffe, as this exceeds the limit for integers. If you can run it in python, it will likely just cut off the data on what's above the allowed size, because Caffe internally does not handle more data / higher indices.

Note that the OpenCL Caffe version, which also has CUDA, supports the compile option "USE_INDEX_64", in which case the limit will not be on 32bit integers for blob sizes, but 64bit integers. Note however, that the OpenCL or CUDA implementation of your device (BLAS, or otherwise) might still not support such large indices, in which case it will compile and run, but can give faulty results!

My general recommendation is to work with smaller data though, most networks can be adapted in a way that they can be applied equally by repeating them over the input images, in a sliding window style.

[huangbo1221]

I have solved this problem,and just revised something in caffe. Firstly locate caffe-master/src/caffe/blob.cpp. Secondly, delete two lines of code:
CHECK_GE(shape[i], 0);
CHECK_LE(shape[i], INT_MAX / count_) << "blob size exceeds INT_MAX";
Lastly,recompile caffe!

[naibaf7]

@huangbo1221
You absolutely cannot do that. It will result in index overflows, wrong computations, crashes etc. These checks are there for a reason...

If you REALLY want to use blobs exceeding INT_MAX, use a 64 bit operating system, compile OpenCL Caffe with USE_INDEX_64 and make sure your hardware (GPU) and software (BLAS) supports 64 bit pointers and allocations.