Py-faster-rcnn: Training of py-faster-rcnn on ImageNet

[ghost]

I revised the codes in lib/datasets to train on the ImageNet detection data: I create an ilsvrc.py (like the pascal_voc.py) to account for the ImageNet data and modify the corresponding codes in factory.py. Then I run the experiment script ./experiments/scripts/faster_rcnn_alt_opt.sh and everything seems to be correct at first, but then the training seems to get stuck with the following information never moving forward again...

Loading pretrained model weights from data/imagenet_models/ZF.v2.caffemodel
Solving...
I1101 15:18:12.302584 40312 solver.cpp:242] Iteration 0, loss = 1.10525
I1101 15:18:12.302647 40312 solver.cpp:258]     Train net output #0: rpn_cls_loss = 0.785905 (* 1 =     0.785905 loss)
I1101 15:18:12.302659 40312 solver.cpp:258]     Train net output #1: rpn_loss_bbox = 0.319344 (* 1 = 0.319344 loss)
I1101 15:18:12.302670 40312 solver.cpp:571] Iteration 0, lr = 0.001

Could anyone give me some suggestions? Thanks!

[GregorySenay]

Same thing for me. I have reduced the pascal voc dataset to couple of classes.
The data loading is okay, number of classes changed, but it is stacks like you on the Iteration 0.

I suspect some errors in the roidb.

[sunshineatnoon]

@Jianchao-ICT I am also interested in training a model on ImageNet, so all I need to do is to change code in lib/datasets/ilsvrc.py and factory.py ? Do I need to change --imdb voc_2007_trainval \ in ./experiments/scripts/faster_rcnn_alt_opt.sh ?

[ghost]

@senayg Hi, have you tried to run it without changing any code?
Well, now I located the problem by printing some information manually in the SolverWrapper.train_model in ./lib/fast_rcnn/train.py and the training gets stuck at the fourth call of self.solver.step(1). Since it will not print out the information in each iteration, so we only stop at Iteration 0.
Anyway, I am still painstakingly trying to fix it...

[ghost]

@sunshineatnoon Yes, you also need to change the .sh training script to use your dataset. You may also specify the dataset in the command line using --imdb YOUR_DATASET.

[sunshineatnoon]

@Jianchao-ICT Thx. I have trained a fast rcnn model on ImageNet, it seems the method to train faster rcnn is the same, right?

[GregorySenay]

@Jianchao-ICT as I suspected for myself it was due to the data (maybe the same thing for you).
Look the py-faster-rcnn/lib/roi_data_layer/minibatch.py file and line 80 you can print the value for each batch with "blobs"

print blobs

and look the last values before your crash, if some values are empty the crash is normal, you have to check again you dataset.

[ghost]

@sunshineatnoon Well, I guess training the two is basically the same since the core of faster r-cnn is also implemented using fast rcnn. However, I still came across the above problem...

[ghost]

@senayg Oh, GREAT THANKS! I will try it now.

[ghost]

@senayg Hi, I have added print blobs to the line you mention. And the printing information is as follows (sorry that they are so long). But I do not see something like _empty values_. Could you help me check for it? Thanks!

Loading pretrained model weights from data/imagenet_models/ZF.v2.caffemodel
Solving...
{'gt_boxes': array([[ 411.20001221,  254.3999939 ,  534.40002441,  451.20001221,   97.        ],
   [ 136.        ,  246.3999939 ,  280.        ,  465.6000061 ,   97.        ],
   [ 248.        ,  240.        ,  390.3999939 ,  448.        ,   97.        ],
   [ 481.6000061 ,  283.20001221,  638.40002441,  456.        ,   97.        ]], dtype=float32), 'data': array([[[[  4.01989985,   4.45739985,   4.95739985, ...,  -9.16759968,
      -16.29260063, -21.98010063],
     [  5.76990032,   5.82458735,   5.88708735, ...,  -8.75744343,
      -15.11681938, -20.23010063],
     [  7.95740032,   7.4925561 ,   6.9534936 , ...,  -8.08947468,
      -13.55041313, -17.98010063],
     ..., 
     [-50.04259872, -48.97619247, -46.67150497, ..., -38.23009872,
      -33.35119247, -27.85510063],
     [-34.35509872, -37.41759872, -40.98009872, ..., -23.77697372,
      -15.47228622,  -7.98010015],
     [-22.98010063, -29.10509872, -37.04259872, ..., -11.41760063,
       -1.85509968,   6.01989985]],

    [[-22.94650078, -22.50900078, -21.82150078, ..., -43.82149887,
      -43.38399887, -42.94649887],
     [-22.07150078, -22.01681328, -21.84884453, ..., -42.94649887,
      -41.77071762, -40.75899887],
     [-21.00900078, -21.41915703, -21.88790703, ..., -41.85665512,
      -39.79415512, -38.07149887],
     ..., 
     [-46.00899887, -41.44259262, -35.15353012, ..., -47.54806137,
      -30.46603012, -15.94649982],
     [-44.13399887, -37.21603012, -28.86056137, ..., -19.04806137,
       -8.69259262,  -0.57149982],
     [-41.94649887, -34.07149887, -25.19650078, ...,   2.99100018,
        6.86600018,   9.05350018]],

    [[-25.77169991, -25.33419991, -24.77169991, ..., -45.77170181,
      -44.08420181, -42.77170181],
     [-26.20919991, -26.15451241, -26.05685616, ..., -45.85373306,
      -45.31467056, -44.95920181],
     [-26.45919991, -26.89669991, -27.40841866, ..., -45.58810806,
      -46.83810806, -47.95920181],
     ..., 
     [-51.08420181, -36.78342056, -18.31076241, ..., -28.17013931,
      -22.21701431, -15.27169991],
     [-32.64670181, -20.28732681,  -5.38888741, ...,  -6.17013836,
        7.09548569,  18.85329819],
     [-17.77169991,  -7.70919991,   3.41580009, ...,  12.04080009,
       29.79079819,  44.22829819]]]], dtype=float32), 'im_info': array([[ 600.        ,  800.        ,    1.60000002]], dtype=float32)}
I1103 14:39:37.399827 37785 solver.cpp:242] Iteration 0, loss = 1.10525
I1103 14:39:37.400996 37785 solver.cpp:258]     Train net output #0: rpn_cls_loss = 0.785905 (*    1 = 0.785905 loss)
I1103 14:39:37.401373 37785 solver.cpp:258]     Train net output #1: rpn_loss_bbox = 0.319344 (* 1 = 0.319344 loss)
I1103 14:39:37.401665 37785 solver.cpp:571] Iteration 0, lr = 0.001
{'gt_boxes': array([[ 317.64706421,   65.77539825,  625.66845703,  596.79144287,  178.        ],
   [ 630.48126221,  142.78074646,  802.13903809,  595.18719482,  178.        ],
   [   1.60427809,   16.04277992,  449.19787598,  596.79144287,  182.        ]], dtype=float32), 'data': array([[[[  1.39019897e+02,   1.39019897e+02,   1.39019897e+02, ...,
        7.11499634e+01,   7.44115601e+01,   7.70198975e+01],
     [  1.38584900e+02,   1.38584900e+02,   1.38584900e+02, ...,
        7.17175980e+01,   7.21227417e+01,   7.22348938e+01],
     [  1.38078232e+02,   1.38078232e+02,   1.38078232e+02, ...,
        7.25131989e+01,   6.92919464e+01,   6.62532272e+01],
     ..., 
     [  9.64950256e+01,   8.99192886e+01,   8.04756699e+01, ...,
       -6.37412262e+01,  -5.87805023e+01,  -5.58634605e+01],
     [  9.13698730e+01,   9.24639282e+01,   9.25231552e+01, ...,
       -6.10821762e+01,  -5.89387131e+01,  -5.99351120e+01],
     [  8.70198975e+01,   9.39798965e+01,   1.01328232e+02, ...,
       -5.86048241e+01,  -5.88067589e+01,  -6.29800949e+01]],

    [[  1.38053497e+02,   1.38053497e+02,   1.38053497e+02, ...,
        9.13434753e+01,   9.00968323e+01,   8.90534973e+01],
     [  1.37618500e+02,   1.37618500e+02,   1.37618500e+02, ...,
        9.20873337e+01,   9.09668274e+01,   8.99235001e+01],
     [  1.37111832e+02,   1.37111832e+02,   1.37111832e+02, ...,
        9.31118317e+01,   9.21272659e+01,   9.10534973e+01],
     ..., 
     [  1.23878540e+02,   1.20808174e+02,   1.14971565e+02, ...,
       -4.11328888e+01,  -2.67664261e+01,  -1.12380953e+01],
     [  1.26878510e+02,   1.24570831e+02,   1.20317894e+02, ...,
       -3.77918243e+01,  -2.69656506e+01,  -1.60315208e+01],
     [  1.29053497e+02,   1.27313492e+02,   1.24178505e+02, ...,
       -3.52514000e+01,  -2.72498436e+01,  -1.99465008e+01]],

    [[  1.31228302e+02,   1.31228302e+02,   1.31228302e+02, ...,
        1.06938324e+02,   1.07706635e+02,   1.08228302e+02],
     [  1.30793304e+02,   1.30793304e+02,   1.30793304e+02, ...,
        1.07934464e+02,   1.08160484e+02,   1.08228302e+02],
     [  1.30286636e+02,   1.30286636e+02,   1.30286636e+02, ...,
        1.09353424e+02,   1.08810829e+02,   1.08228302e+02],
     ..., 
     [  1.31645111e+02,   1.29676804e+02,   1.25583656e+02, ...,
       -9.91133976e+00,   1.87424850e+00,   1.68200645e+01],
     [  1.29403290e+02,   1.29875275e+02,   1.29425476e+02, ...,
       -7.24253225e+00,   1.09261334e-01,   8.75326347e+00],
     [  1.27228302e+02,   1.29403290e+02,   1.31469971e+02, ...,
       -5.20666695e+00,  -1.42337227e+00,   2.22830009e+00]]]], dtype=float32), 'im_info': array([[ 600.        ,  802.        ,    1.60427809]], dtype=float32)}
{'gt_boxes': array([[ 262.3999939 ,   44.79999924,  712.        ,  480.        ,  182.        ],
   [ 113.59999847,  379.20001221,  350.3999939 ,  596.79998779,  199.        ]], dtype=float32), 'data': array([[[[ -96.98010254,  -96.98010254,  -96.91760254, ...,  -97.98010254,
       -97.98010254,  -97.98010254],
     [ -96.54260254,  -96.54260254,  -96.50744629, ...,  -97.98010254,
       -97.98010254,  -97.98010254],
     [ -95.91760254,  -95.91760254,  -95.91760254, ...,  -97.98010254,
       -97.98010254,  -97.98010254],
     ..., 
     [  -8.35509968,   -8.73791218,   -9.10509968, ...,    3.07849383,
         1.83240008,    0.95739996],
     [  -6.29260015,   -7.22228718,   -8.25744343, ...,    4.83240032,
         3.89880633,    3.26990008],
     [  -4.98010015,   -6.29260015,   -7.79260015, ...,    6.14489985,
         5.45739985,    5.01989985]],

    [[-100.94650269, -100.94650269, -100.88400269, ..., -105.94650269,
      -105.94650269, -105.94650269],
     [-100.50900269, -100.50900269, -100.47384644, ..., -105.94650269,
      -105.94650269, -105.94650269],
     [ -99.88400269,  -99.88400269,  -99.88400269, ..., -105.94650269,
      -105.94650269, -105.94650269],
     ..., 
     [ -17.32150078,  -17.70431328,  -18.07150078, ...,   -9.88790607,
       -11.13399982,  -12.00899982],
     [ -15.25900078,  -16.18868828,  -17.22384453, ...,   -8.13399982,
        -9.06759357,   -9.69649982],
     [ -13.94650078,  -15.25900078,  -16.75900078, ...,   -6.82149982,
        -7.50899982,   -7.94649982]],

    [[ -97.771698  ,  -97.771698  ,  -97.709198  , ..., -113.771698  ,
      -113.771698  , -113.771698  ],
     [ -97.334198  ,  -97.334198  ,  -97.29904175, ..., -113.771698  ,
      -113.771698  , -113.771698  ],
     [ -96.709198  ,  -96.709198  ,  -96.709198  , ..., -113.771698  ,
      -113.771698  , -113.771698  ],
     ..., 
     [ -23.14669991,  -23.52951241,  -23.89669991, ...,  -16.71310616,
       -17.95919991,  -18.83419991],
     [ -21.08419991,  -22.01388741,  -23.04904366, ...,  -14.95919991,
       -15.89279366,  -16.52169991],
     [ -19.77169991,  -21.08419991,  -22.58419991, ...,  -13.64669991,
       -14.33419991,  -14.77169991]]]], dtype=float32), 'im_info': array([[ 600.        ,  800.        ,    1.60000002]], dtype=float32)}
{'gt_boxes': array([], shape=(0, 5), dtype=float32), 'data': array([[[[ 142.01989746,  142.01989746,  142.36988831, ...,  141.77655029,
       143.63317871,  146.01989746],
     [ 141.54989624,  141.32897949,  141.37506104, ...,  141.06529236,
       142.50361633,  144.6098938 ],
     [ 141.13656616,  140.61172485,  140.28045654, ...,  140.43359375,
       141.2769165 ,  143.13656616],
     ..., 
     [ 141.01989746,  141.01989746,  141.01989746, ...,  141.77655029,
       141.61657715,  141.01989746],
     [ 141.01989746,  141.01989746,  141.01989746, ...,  141.77655029,
       141.61657715,  141.01989746],
     [ 141.01989746,  141.01989746,  141.01989746, ...,  141.77655029,
       141.61657715,  141.01989746]],

    [[ 133.05349731,  131.64349365,  130.40348816, ...,  130.05349731,
       130.86013794,  132.05349731],
     [ 132.58349609,  131.39439392,  130.40348816, ...,  129.92662048,
       130.39013672,  131.58349609],
     [ 132.05349731,  131.1683197 ,  130.57460022, ...,  130.15882874,
       130.09347534,  131.28683472],
     ..., 
     [ 131.05349731,  131.05349731,  131.05349731, ...,  131.81015015,
       131.650177  ,  131.05349731],
     [ 131.05349731,  131.05349731,  131.05349731, ...,  131.81015015,
       131.650177  ,  131.05349731],
     [ 131.05349731,  131.05349731,  131.05349731, ...,  131.81015015,
       131.650177  ,  131.05349731]],

    [[ 115.228302  ,  114.28829956,  113.69496918, ...,  114.95834351,
       114.63162231,  115.228302  ],
     [ 114.75830078,  114.26010132,  114.32946777, ...,  116.01271057,
       115.19249725,  115.228302  ],
     [ 114.46163177,  114.57130432,  115.55246735, ...,  117.8072052 ,
       116.26908875,  115.81163025],
     ..., 
     [ 115.228302  ,  115.228302  ,  115.228302  , ...,  115.98495483,
       115.82498169,  115.228302  ],
     [ 115.228302  ,  115.228302  ,  115.228302  , ...,  115.98495483,
       115.82498169,  115.228302  ],
     [ 115.228302  ,  115.228302  ,  115.228302  , ...,  115.98495483,
       115.82498169,  115.228302  ]]]], dtype=float32), 'im_info': array([[ 600.        ,  773.        ,    1.54639173]], dtype=float32)}

[ghost]

@senayg Oh, I see, the last gt_boxes is an empty []. I will check it again. Thanks for your really useful hints!

[GregorySenay]

So clearly the issue is here : {'gt_boxes': array([], shape=(0, 5), dtype=float32),

[GregorySenay]

Your welcome! Good to help you to debug!

[ghost]

@senayg Thanks! You are really nice :+1:

[ghost]

@senayg Oh, the problem is found. It is due to the image ILSVRC2013_val_00003481.JPEG in the fourth batch. This image strangely has no ground truth annotations and causes gt_boxes to be empty...
Oh, damn it, how could such images still appear in the dataset...

[GregorySenay]

Good the issue, can now be close, I think

[ghost]

@senayg Hi, I have closed it.

[sunshineatnoon]

@Jianchao-ICT I am training the model on imagenet, but I have few problems:

1. do I need to change files in /py-faster-rcnn/models/ZF/faster_rcnn_alt_opt/ to specify my own classes?
2. I've encounter a problem:

File "/py-faster-rcnn/tools/../lib/datasets/imdb.py", line 108, in append_flipped_images
    assert (boxes[:, 2] >= boxes[:, 0]).all()
AssertionError

The command I used to train is:

./tools/train_faster_rcnn_alt_opt.py --gpu 0 --net_name ZF --weights data/imagenet_models/ZF.v2.caffemodel --imdb imagenet_train --cfg experiments/cfgs/faster_rcnn_alt_opt.yml 

Did you have the same problem?

[ghost]

@sunshineatnoon

1. Well, I am not sure whether this method will work. To account for the classes of ImageNet, I add a file ilsvrc.py inside ./lib/datasets like the original pascal_voc.py.
2. Haha, I have met exactly the same problem. This is due to the Pascal Voc's annotations are 1-based while the ImageNet's annotations are 0-based in xmin, xmax, ymin, ymax. So you just need to change the code in the function def _load_pascal_annotation(self, index) in your added ilsvrc.py. Specifically, the following four lines need to get the - 1 removed.

x1 = float(get_data_from_tag(obj, 'xmin')) - 1
y1 = float(get_data_from_tag(obj, 'ymin')) - 1
x2 = float(get_data_from_tag(obj, 'xmax')) - 1
y2 = float(get_data_from_tag(obj, 'ymax')) - 1

[sunshineatnoon]

@Jianchao-ICT I've got rid of the -1 operation, but I still got the same error. Then I found this guy: n02958343_19208.xml had wrong annotations. There is only one object in the image, but there are four objects annotated in the .xml file. Changing the annotation file fixed my problem.

[ghost]

@sunshineatnoon Oh, driven crazy by this dataset: it has involved many gray images; some of the images have 0 ground truth object bounding boxes. Both these two have incurred some mistakes to my program. In fact I use the validation set for training since I have still been not very clear with the labels of the training set: there are more than 200 categories labeled in it.

Now I also see your problem. I just want to say this dataset seems not that user-friendly...

[ghost]

@sunshineatnoon Hi, how is your training now? I now come across the following error in the Stage 1 Fast RCNN training. Have you got similar things?

Traceback (most recent call last):
  File "/home/code/lijianchao/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/code/lijianchao/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "./tools/train_faster_rcnn_alt_opt.py", line 196, in train_fast_rcnn
    max_iters=max_iters)
  File "/home/code/lijianchao/py-faster-rcnn/tools/../lib/fast_rcnn/train.py", line 138, in train_net
    model_paths = sw.train_model(max_iters)
  File "/home/code/lijianchao/py-faster-rcnn/tools/../lib/fast_rcnn/train.py", line 104, in train_model
    self.solver.step(1)
  File "/home/code/lijianchao/py-faster-rcnn/tools/../lib/roi_data_layer/layer.py", line 143, in forward
    blobs = self._get_next_minibatch()
  File "/home/code/lijianchao/py-faster-rcnn/tools/../lib/roi_data_layer/layer.py", line 62, in _get_next_minibatch
    return get_minibatch(minibatch_db, self._num_classes)
  File "/home/code/lijianchao/py-faster-rcnn/tools/../lib/roi_data_layer/minibatch.py", line 55, in get_minibatch
    num_classes)
  File "/home/code/lijianchao/py-faster-rcnn/tools/../lib/roi_data_layer/minibatch.py", line 125, in _sample_rois
    roidb['bbox_targets'][keep_inds, :], num_classes)
  File "/home/code/lijianchao/py-faster-rcnn/tools/../lib/roi_data_layer/minibatch.py", line 176, in _get_bbox_regression_labels
    bbox_targets[ind, start:end] = bbox_target_data[ind, 1:]
ValueError: could not broadcast input array from shape (4) into shape (0)

[sunshineatnoon]

@Jianchao-ICT No, I didn't encounter this problem, so it might caused by your dataset. I got other errors (https://github.com/rbgirshick/py-faster-rcnn/issues/13) during Stage 1 Fast RCNN training. I fixed that error and still training now, hope no more errors...

[ghost]

@sunshineatnoon Oh, I train on the ImageNet detections. Well, I will try to figure it out... I also hope no more errors...

[ghost]

@sunshineatnoon Hi, I have figured out the reason... I made a too naive mistake: I forgot to change the num_output in .pt (prototxt) from 21 (PASCAL VOC) to 201 (ImageNet)...

[sunshineatnoon]

@Jianchao-ICT Good for you! You trained on the whole dataset of imagenet? I only trained on two classes on ImageNet and it took a long long time for training.

[ghost]

@sunshineatnoon Yeah, I train on the 201 category (with background as one category) but I use the validation set for training since I notice that the training set has some labels that are not in the 200 categories?

[leefionglee]

Hi all:
I would like to train py-faster-rcnn on my own dataset, but what is exactly the dataformat? i.e. the images, annotations, train, val? Can anyone post an example annotation file here? As I found one post here: https://github.com/zeyuanxy/fast-rcnn/tree/master/help/train, in which the annotation is text, while I also found PASCAL is XML. What exactly it is? Any reference blogs or tutorials will be highly appreciated. Thanks

[jojotata]

@Jianchao-ICT
@sunshineatnoon
Hi, I already successfully trained a model on ImageNet datasets.
I trained on 2 classes, "scissors" and "computer_mouse" on ImageNet, and it worked.
However, when I tried to expand the number of classes, I got the below error.

File "/py-faster-rcnn/tools/../lib/datasets/imdb.py", line 108, in append_flipped_images
    assert (boxes[:, 2] >= boxes[:, 0]).all()
AssertionError

I examined the cause of this error, and I suspect the images of "person" class(http://www.image-net.org/synset?wnid=n00007846) has a problem.
This is because when I trained without "person" class, it worked, but with this class, I always got the assertion error.
According to sunshineatnoon's previous post, you could find the wrong data.
But I can't find which image in "person" class's datasets has a problem.
How did you find the image which caused the error?
And I also suspect the cause of this assertion error is not annotation file(_.xml file_) but image itself(_JPEG file_).
Could you know why this error occurred?
Thanks.

[leejiajun]

@jojotata
1.you can check annotation file.
2.you can print boxes to check value. if oldx1 or oldx2 equal 0, oldx1 or oldx2 minus 1 is equal to 65535.
see the file: lib/datasets/imdb.py
oldx1 = boxes[:, 0].copy()
oldx2 = boxes[:, 2].copy()
boxes[:, 0] = widths[i] - oldx2 - 1
boxes[:, 2] = widths[i] - oldx1 - 1

[ericromanenghi]

@jojotata maybe you should clean the cache folder, in the data directory

[Anubhav7]

Hi,

Has anyone tried to parallelize the calls to im_detect. I am facing issues with Caffe Initialization when I make calls to the network. Exact error details are below

Background: I have pre-loaded the network and am passing the net parameter in pooled calls.

Error Statement:

F0428 16:30:55.227715 19959 syncedmem.hpp:18] Check failed: error == cudaSuccess (3 vs. 0) initialization error

[ericromanenghi]

How are you parallelizing? With multiprocessing module?

My coworker has this issue yesterday trying to parallelize with celery.

[Anubhav7]

Yes, I am parallelizing with the multiprocessing module in python (pool function). Has your co-worker solved the issue?

[ericromanenghi]

No, we couldn't =S

[ericromanenghi]

But now I suspect that is something related with the way that python handle the pooling, and not with celery exclusively. Thank you for your response!

[ednarb29]

@jojotata
I had the same problem and solved it by removing the -1 in these lines, too. (I removed the -1 in the lib/datasets/mydataset.py but forgot this one)

Change:
boxes[:, 0] = widths[i] - oldx2 - 1
boxes[:, 2] = widths[i] - oldx1 - 1

To:
boxes[:, 0] = widths[i] - oldx2
boxes[:, 2] = widths[i] - oldx1

Should fix your problem. :)

[JohnnyY8]

@Jianchao-ICT @sunshineatnoon Hi！
I want to train faster r-cnn on VID dataset, but I am not quite sure how to load data. Because I do not know how to specify the parameter in factory.py like following:

I saw you guys have trained faster r-cnn on ImageNet dataset, could you please tell me what does "imdb" mean? How should I use it ?
In addition, if I finish the code changes, should I use experiments/scripts/faster_rcnn_alt_opt.sh or tools/train_faster_rcnn_alt_opt.py to start training ?
Thank you so much!

[ednarb29]

@JohnnyY8

It should look more like this, if you refer to the original pascal_voc structure:

for year in ['2016']:
for split in ['train', 'test']:
name = 'vid_{}_{}'.format(year, split)
__sets[name] = (lambda split=split, year=year: vid(split, year))

Then you have to create the function, that creates the dataset with the call vid(split, year).

IMDB is actually the image database, where the data is stored for training, filled by your own function.

If you have a look into the shell script, you will note that the shell script just calls the python script. You can do both ways but the shell script might be more comfortable, even for testing later on.

Does this help a bit?

[JohnnyY8]

@ednarb29 Thanks for your time!
The explaination about shell script and python script is good for me. I am confused about two of them because too many parameters in shell script. And I do not know how to set the right path about train_imdb or test_imdb. As for lambda expression, I am a new fisher.
Thanks again! About the questions for training, I will reply on #243.

[ednarb29]

The shell script just defines the parameters and calls the python script for training and testing. That is only for automation, so later you just have to call the shell script and everything is done for you.

[JohnnyY8]

@ednarb29 So if I modify the codes and set training or test data path rightly, just change some parameters for shell script will be ok for training and testing, right ?

[ednarb29]

Yes, that all depends somehow on the other changes you have to do for the new dataset, but the idea is an easy use for later experiments when it is configured.

[JohnnyY8]

@ednarb29 @Jianchao-ICT Hi!
I want to train faster r-cnn on ImageNet DET dataset by following command:
sudo rm -f data/cache/* && sudo ./tools/train_net.py --gpu 0 --iters 100000 --imdb DET_train --weights data/imagenet_models/ZF.v2.caffemodel --cfg experiments/cfgs/faster_rcnn_end2end.yml --solver models/imagenet_det/ZF/faster_rcnn_end2end/solver.prototxt
But I got this error:

I have checked all xml files, if some of them do not have any object or the object's name beyond the 200 classes, I decide to ingore these xml files. What's more, I also remove -1 in mydataset.py or imdb.py, but it does not work.
Any suggestions? Thanks!

[leejiajun]

@JohnnyY8 it seems that caffemodel and prototxt do not match.

[JohnnyY8]

@leejiajun Yes, I solver it by changing
--solver models/imagenet_det/ZF/faster_rcnn_end2end/solver.prototxt to
--solver models/pascal_voc/ZF/faster_rcnn_end2end/solver.prototxt. That means it still load the old train.prototxt under pascal_voc, right?
But I got another error:

Do you know what happened?

[leejiajun]

@JohnnyY8 argmax is an empty sequence. this problem could be comes from too small or too large the size of image of your training set.

BTW: http://blog.csdn.net/jiajunlee/article/details/50470897

[JohnnyY8]

@leejiajun Thanks for your blog.
So that means I need to remove all too small or too large images in traning set by some script, right?
Is the ratio (width / height) range in 0.117-15.500 ?

[leejiajun]

@JohnnyY8
I removed.

[JohnnyY8]

@leejiajun Hi！
I have removed all too small or too large images. All remainings' ratio are in (0.18, 15.000).
But I meet another problem, the iteration just stopped when it started, like following:
I0804 11:28:13.044482 21960 solver.cpp:229] Iteration 0, loss = 6.54452
I0804 11:28:13.044561 21960 solver.cpp:245] Train net output #0: bbox_loss = 0.305254 (* 1 = 0.305254 loss)
I0804 11:28:13.044575 21960 solver.cpp:245] Train net output #1: cls_loss = 5.45421 (* 1 = 5.45421 loss)
I0804 11:28:13.044585 21960 solver.cpp:245] Train net output #2: rpn_cls_loss = 0.614111 (* 1 = 0.614111 loss)
I0804 11:28:13.044595 21960 solver.cpp:245] Train net output #3: rpn_loss_bbox = 0.379346 (* 1 = 0.379346 loss)
I0804 11:28:13.044607 21960 sgd_solver.cpp:106] Iteration 0, lr = 0.001
Is this as your blog said "play dead"?

[leejiajun]

@JohnnyY8 Probably! I remenber that people have discussed your problem in this issue.

[JohnnyY8]

@leejiajun Yes, but I have checked all xml files more than once, all remainings maybe ok.
I still try to print blobs information to find the problem. Do you have any other suggestion please?

[JohnnyY8]

@Jianchao-ICT @GregorySenay Hi!
I meet the same problem that the iteration get stuck when it started. And I try to print blobs information, there seems not empty variable. What's more, I try to check all remaining xml files are ok by python script.
Could you have any suggestion?
Thanks!

[GregorySenay]

Can you print the blobs during the iteration and paste the end of the log ?

[JohnnyY8]

@GregorySenay No problem！
But there is too much, the following is last part of it. Thanks a lot!

{'gt_boxes': array([[ 156.75675964,   82.88288116,  805.40539551,  598.19818115,   26.        ]], dtype=float32), 'data': array([[[[ -45.98009872,  -33.01259995,  -11.36759949, ...,  -98.62988281,
           -87.5300293 ,  -81.98010254],
         [ -30.68510056,  -22.25040817,   -8.17144394, ...,  -94.91837311,
           -84.74119568,  -79.65260315],
         [  -5.15510035,   -4.28644419,   -2.83650637, ...,  -88.7232132 ,
           -80.08614349,  -75.76760101],
         ..., 
         [  32.94458008,   37.6867485 ,   45.60224152, ...,   47.19591141,
            61.69502258,   68.94458008],
         [  47.37518311,   57.46916199,   74.31774902, ...,   57.00600433,
            74.58545685,   83.37518311],
         [  56.01990128,   69.31990051,   91.51989746, ...,   62.8827858 ,
            82.30752563,   92.01989746]],

        [[   9.05350018,   12.04599953,   17.04099846, ...,  -14.27138996,
            -8.7214632 ,   -5.94649982],
         [  15.03849983,   17.47821808,   21.55052948, ...,  -11.77848244,
            -6.78216076,   -4.28399992],
         [  25.0284996 ,   26.54553032,   29.07771873, ...,   -7.61738825,
            -3.5451293 ,   -1.50900006],
         ..., 
         [  52.76559448,   59.74810028,   71.40309143, ...,   63.76848984,
            72.34997559,   76.64071655],
         [  71.08135986,   78.06385803,   89.71885681, ...,   70.52735901,
            83.11328888,   89.40625   ],
         [  82.05349731,   89.03599548,  100.69099426, ...,   74.57629395,
            89.56109619,   97.05349731]],

        [[  21.228302  ,   25.88330078,   33.65329742, ...,  -40.92908096,
           -34.82416153,  -31.77169991],
         [  26.88080025,   30.31968307,   36.05976868, ...,  -38.49187088,
           -33.12509155,  -30.44170189],
         [  36.3157959 ,   37.72476959,   40.07658005, ...,  -34.4237442 ,
           -30.28904915,  -28.22169876],
         ..., 
         [  83.90322876,   89.26475525,   98.21405792, ...,   90.79845428,
            94.50986481,   96.36557007],
         [  87.23336792,   94.44034576,  106.47002411, ...,   94.41313171,
           102.74504852,  106.91101074],
         [  89.228302  ,   97.540802  ,  111.415802  , ...,   96.57852173,
           107.67837524,  113.228302  ]]]], dtype=float32), 'im_info': array([[ 600.       ,  901.       ,    1.8018018]], dtype=float32)}
{'gt_boxes': array([[ 140.11976624,   23.35329247,  810.17962646,  562.27545166,  101.        ]], dtype=float32), 'data': array([[[[ -64.98010254,  -64.64509583,  -64.08843231, ...,  -68.97509766,
           -68.98010254,  -68.98010254],
         [ -64.98010254,  -64.75732422,  -64.38713837, ...,  -67.3000946 ,
           -67.12029266,  -66.9701004 ],
         [ -64.98010254,  -64.94380951,  -64.88350677, ...,  -64.51676178,
           -64.02986145,  -63.63010025],
         ..., 
         [ -94.08843994,  -93.34585571,  -92.11190033, ...,  -97.09790039,
           -96.53677368,  -96.08843994],
         [ -94.64511108,  -93.52955627,  -91.67584229, ...,  -97.65179443,
           -97.09344482,  -96.64511108],
         [ -94.98010254,  -93.64009857,  -91.41343689, ...,  -97.98510742,
           -97.42843628,  -96.98010254]],

        [[ -50.94650269,  -52.95649719,  -56.29649734, ...,  -61.93648911,
           -61.94649887,  -61.94649887],
         [ -52.28649902,  -53.51091766,  -55.54553986, ...,  -61.93816376,
           -61.94649506,  -61.94649506],
         [ -54.51316452,  -54.4322052 ,  -54.2976799 , ...,  -61.94094849,
           -61.94649887,  -61.94649887],
         ..., 
         [ -99.05484009,  -98.04983521,  -96.37984467, ..., -106.06430817,
          -105.56293488, -105.16317749],
         [ -99.61151123,  -98.60650635,  -96.93650818, ..., -106.61819458,
          -106.42671204, -106.27651978],
         [ -99.94650269,  -98.9414978 ,  -97.27149963, ..., -106.95150757,
          -106.94650269, -106.94650269]],

        [[-100.771698  ,  -98.09169769,  -93.6383667 , ..., -103.77670288,
          -103.771698  , -103.771698  ],
         [ -98.42669678,  -96.86894226,  -94.28044891, ..., -104.77835083,
          -104.4070816 , -104.10669708],
         [ -94.53003693,  -94.83711243,  -95.34738922, ..., -106.44277954,
          -105.46289062, -104.66336823],
         ..., 
         [-116.8800354 , -116.54502869, -115.9883728 , ..., -117.88950348,
          -117.20883942, -116.6633606 ],
         [-117.43670654, -117.10169983, -116.54504395, ..., -118.44338989,
          -117.15131378, -116.10668945],
         [-117.771698  , -117.43669128, -116.8800354 , ..., -118.77670288,
          -117.11669922, -115.771698  ]]]], dtype=float32), 'im_info': array([[ 600.        ,  898.        ,    1.79640722]], dtype=float32)}
{'gt_boxes': array([[   3.18302393,  183.02388   ,  558.6206665 ,  792.57293701,  124.        ]], dtype=float32), 'data': array([[[[-102.98010254, -101.21009827,  -99.26343536, ...,   16.09515572,
            22.23242378,   20.01989746],
         [-102.98010254, -101.60171509, -100.02306366, ...,   16.40118408,
            20.6073494 ,   17.80739975],
         [-102.98010254, -102.15779877, -101.12177277, ...,   17.82502937,
            18.45528793,   14.38240051],
         ..., 
         [ -97.47503662,  -88.94907379,  -78.49136353, ...,  -81.52783203,
           -95.53408813, -102.7326355 ],
         [-101.24517822,  -98.55821228,  -94.92288971, ...,  -90.3013382 ,
           -99.48808289, -100.8475647 ],
         [-102.98010254, -102.98010254, -102.48426819, ...,  -94.33868408,
          -101.30761719,  -99.98010254]],

        [[-115.94650269, -115.06150055, -114.08816528, ...,  100.62063599,
           113.70852661,  111.05349731],
         [-115.94650269, -115.45311737, -114.84779358, ...,   98.64407349,
           113.91978455,  114.59349823],
         [-115.94650269, -115.94650269, -115.81487274, ...,   98.12947083,
           113.98478699,  117.84933472],
         ..., 
         [-110.44143677,  -94.97698212,  -76.4625473 , ...,  -85.0682373 ,
          -103.93998718, -112.61654663],
         [-114.21157837, -109.03489685, -102.7697525 , ...,  -97.48362732,
          -109.49004364, -110.1031189 ],
         [-115.94650269, -115.5039978 , -114.87567139, ..., -103.1968689 ,
          -112.04403687, -108.94650269]],

        [[-122.771698  , -121.88669586, -120.9133606 , ...,   96.51211548,
           123.82070923,  132.228302  ],
         [-122.771698  , -122.27831268, -121.67298889, ...,   96.01977539,
           119.43886566,  126.47579956],
         [-122.771698  , -122.771698  , -122.64006805, ...,   98.00818634,
           114.30695343,  118.51997375],
         ..., 
         [-117.26663208, -103.46268463,  -86.97528076, ...,  -93.52438354,
          -113.71115112, -122.60671997],
         [-121.03677368, -116.68645477, -111.44192505, ..., -103.60436249,
          -118.849617  , -121.3500061 ],
         [-122.771698  , -122.771698  , -122.7008667 , ..., -108.24291992,
          -121.21420288, -120.771698  ]]]], dtype=float32), 'im_info': array([[ 796.        ,  600.        ,    1.59151196]], dtype=float32)}
{'gt_boxes': array([[  36.79999924,   57.59999847,  600.        ,  593.59997559,  124.        ]], dtype=float32), 'data': array([[[[ 109.01989746,  106.83239746,  104.01989746, ...,  146.08239746,
           144.33239746,  143.01989746],
         [ 108.58239746,  106.77770996,  104.43005371, ...,  146.10974121,
           144.82458496,  143.89489746],
         [ 108.08239746,  106.74255371,  104.96130371, ...,  146.08239746,
           145.43005371,  145.01989746],
         ..., 
         [ -88.85510254,  -90.52307129,  -92.90588379, ...,   -8.31603718,
             1.14099383,    7.89490032],
         [ -90.66760254,  -92.17150879,  -94.31994629, ...,  -18.15978813,
            -9.99963188,   -3.79259992],
         [ -91.98010254,  -93.29260254,  -95.16760254, ...,  -24.66760063,
           -18.23010063,  -12.98010063]],

        [[  69.05349731,   66.86599731,   64.05349731, ...,  127.11599731,
           126.49099731,  126.05349731],
         [  69.49099731,   67.68630981,   65.33865356, ...,  127.96365356,
           127.36599731,  126.92849731],
         [  70.30349731,   68.93630981,   67.11209106, ...,  128.99880981,
           128.46365356,  128.05349731],
         ..., 
         [-106.94650269, -108.64181519, -111.05978394, ...,   13.15896893,
            20.02224922,   24.61599922],
         [-107.50900269, -109.01290894, -111.16134644, ...,    4.84646893,
            11.29568672,   16.05349922],
         [-107.94650269, -109.25900269, -111.13400269, ...,   -0.75899994,
             4.6784997 ,    9.05350018]],

        [[ -81.771698  ,  -83.959198  ,  -86.771698  , ...,  -56.70920181,
           -58.45920181,  -59.77170181],
         [ -82.209198  ,  -84.0138855 ,  -86.36154175, ...,  -56.27170181,
           -57.77560806,  -58.89670181],
         [ -82.709198  ,  -83.99435425,  -85.71310425, ...,  -55.76779556,
           -56.92404556,  -57.77170181],
         ..., 
         [-104.896698  , -106.50997925, -108.82247925, ...,   70.243927  ,
            81.1579895 ,   88.978302  ],
         [-105.334198  , -106.83810425, -108.98654175, ...,   64.53689575,
            71.81033325,   77.415802  ],
         [-105.771698  , -107.084198  , -108.959198  , ...,   60.790802  ,
            64.728302  ,   68.228302  ]]]], dtype=float32), 'im_info': array([[ 800.        ,  600.        ,    1.60000002]], dtype=float32)}
{'gt_boxes': array([[  65.15151215,  215.15151978,  524.24243164,  569.69696045,   26.        ],
       [ 446.96969604,  357.57574463,  757.57574463,  598.48486328,   26.        ]], dtype=float32), 'data': array([[[[ 103.01989746,   99.58989716,   90.61990356, ...,   -2.3300755 ,
             0.96994275,    2.01990008],
         [  70.67990112,   73.49250031,   70.06439972, ...,    0.33549571,
            -0.24530748,   -0.43009996],
         [  31.01990128,   39.52140045,   42.16740417, ...,    2.10487318,
            -1.52514696,   -2.68009996],
         ..., 
         [  95.76986694,   95.76986694,   96.19736481, ...,  -79.96212769,
           -84.18617249,  -85.53012085],
         [  96.54992676,   96.30001831,   96.413414  , ...,  -81.35386658,
           -85.63728333,  -87.00012207],
         [  98.01989746,   97.52989197,   97.31990051, ...,  -82.76013184,
           -86.72015381,  -87.98010254]],

        [[  88.05349731,   84.62349701,   75.65350342, ...,  -11.29647541,
            -7.9964571 ,   -6.94649982],
         [  54.73349762,   57.5460968 ,   54.11800385, ...,   -8.6309042 ,
            -9.21170807,   -9.39649963],
         [  13.90350056,   22.40500069,   25.0510025 , ...,   -7.01152658,
           -10.6415472 ,  -11.79649925],
         ..., 
         [  86.8034668 ,   86.8034668 ,   87.23097229, ...,  -76.70751953,
           -79.80954742,  -80.79650879],
         [  87.58352661,   87.33361816,   87.44702148, ...,  -77.1728363 ,
           -80.8094635 ,  -81.96652222],
         [  89.05349731,   88.56349182,   88.35350037, ...,  -77.72653198,
           -81.68655396,  -82.94650269]],

        [[  93.228302  ,   89.79829407,   80.82830048, ...,  -13.12167549,
            -9.82165718,   -8.77169991],
         [  59.41830063,   62.23090363,   58.80279922, ...,  -10.45610428,
           -11.0369072 ,  -11.22170067],
         [  17.77830124,   26.27980042,   28.92580032, ...,   -8.38672638,
           -12.01674747,  -13.17169952],
         ..., 
         [  84.97827148,   84.97827148,   85.40576935, ...,  -76.68270874,
           -79.78473663,  -80.771698  ],
         [  85.7583313 ,   85.50842285,   85.62181854, ...,  -76.99803162,
           -80.63465881,  -81.79171753],
         [  87.228302  ,   86.73830414,   86.52830505, ...,  -77.55172729,
           -81.51174927,  -82.771698  ]]]], dtype=float32), 'im_info': array([[ 600.       ,  758.       ,    1.5151515]], dtype=float32)}
{'gt_boxes': array([[   5.98591566,   87.32394409,  633.09857178,  511.97183228,   58.        ]], dtype=float32), 'data': array([[[[-13.91930008,  -8.19929981, -13.30010033, ..., -28.80412483,
          -26.07765961, -28.08595848],
         [-13.20090103,  -4.6729002 ,  -0.88409996, ..., -27.62861633,
          -27.82015991, -27.20064735],
         [ -8.71610069, -17.38809967,  -9.98010063, ..., -26.85939407,
          -24.66007996, -27.10822296],
         ..., 
         [ 24.47587967,  13.56392384,   7.57976389, ..., -69.85986328,
          -72.76400757, -58.2203331 ],
         [ 11.24070168,  12.00706863,   7.49988031, ..., -69.76400757,
          -67.20253754, -65.70031738],
         [ 12.13201427,  15.83773136,  12.22756577, ..., -64.70825958,
          -64.67768097, -67.12080383]],

        [[-11.88569927,  -7.30649996, -13.78650093, ..., -29.77052498,
          -28.66326523, -31.81235886],
         [-12.12890053, -11.34170055, -14.09049988, ..., -34.3550148 ,
          -35.39133453, -37.68704605],
         [-11.88249969, -17.98649979,  -9.54650116, ..., -30.82579803,
          -29.82648087, -32.27462769],
         ..., 
         [  8.50948143,  -0.49042082,  -3.18656278, ..., -57.58682251,
          -53.68251801, -44.98678207],
         [  1.75428128,   4.28065825,   0.77347076, ..., -63.51469421,
          -58.94649887, -58.14669418],
         [ 10.24557018,  13.47286606,  10.861166  , ..., -58.07456207,
          -63.86334229, -61.9271698 ]],

        [[-16.71089935, -14.22929955, -20.69170189, ..., -34.75572586,
          -28.46927834, -25.87755966],
         [-20.71409988, -18.74449921, -20.43569946, ..., -30.66021729,
          -28.73655319, -26.51224709],
         [-21.70769882, -28.2677002 , -19.57170105, ..., -28.65099716,
          -26.85168076, -29.29982376],
         ..., 
         [  3.68428135,  -8.37963963, -14.41178703, ..., -56.33206177,
          -46.57972717, -31.21200943],
         [ -1.31090903,  -3.5445416 , -11.0517292 , ..., -63.64382172,
          -55.13649368, -50.45187759],
         [  7.42037058,   6.6476655 ,  -0.76403391, ..., -59.8997612 ,
          -63.06925964, -61.91228485]]]], dtype=float32), 'im_info': array([[  6.00000000e+02,   8.00000000e+02,   3.52112681e-01]], dtype=float32)}
{'gt_boxes': array([[ 162.5806427 ,   15.48387051,  754.83868408,  584.51611328,  200.        ]], dtype=float32), 'data': array([[[[ 143.01989746,  143.01989746,  143.60324097, ...,  144.21572876,
           144.01989746,  144.01989746],
         [ 143.01989746,  143.01989746,  143.60324097, ...,  144.21572876,
           144.01989746,  144.01989746],
         [ 143.60324097,  143.60324097,  144.12277222, ...,  143.80375671,
           143.43656921,  143.43656921],
         ..., 
         [ 152.01989746,  152.01989746,  152.01989746, ...,  147.87406921,
           147.87406921,  147.87406921],
         [ 152.01989746,  152.01989746,  152.01989746, ...,  148.01989746,
           148.01989746,  148.01989746],
         [ 152.01989746,  152.01989746,  152.01989746, ...,  148.01989746,
           148.01989746,  148.01989746]],

        [[ 130.05349731,  130.05349731,  130.63682556, ...,  136.24932861,
           136.05349731,  136.05349731],
         [ 130.05349731,  130.05349731,  130.63682556, ...,  136.24932861,
           136.05349731,  136.05349731],
         [ 130.63682556,  130.63682556,  131.15635681, ...,  135.83735657,
           135.47016907,  135.47016907],
         ..., 
         [ 139.05349731,  139.05349731,  139.05349731, ...,  139.05349731,
           139.05349731,  139.05349731],
         [ 139.05349731,  139.05349731,  139.05349731, ...,  139.05349731,
           139.05349731,  139.05349731],
         [ 139.05349731,  139.05349731,  139.05349731, ...,  139.05349731,
           139.05349731,  139.05349731]],

        [[ 123.228302  ,  123.228302  ,  123.81163025, ...,  132.228302  ,
           132.228302  ,  132.228302  ],
         [ 123.228302  ,  123.228302  ,  123.81163025, ...,  132.228302  ,
           132.228302  ,  132.228302  ],
         [ 123.81163025,  123.81163025,  124.3311615 , ...,  131.75920105,
           131.64497375,  131.64497375],
         ..., 
         [ 132.228302  ,  132.228302  ,  132.228302  , ...,  132.228302  ,
           132.228302  ,  132.228302  ],
         [ 132.228302  ,  132.228302  ,  132.228302  , ...,  132.228302  ,
           132.228302  ,  132.228302  ],
         [ 132.228302  ,  132.228302  ,  132.228302  , ...,  132.228302  ,
           132.228302  ,  132.228302  ]]]], dtype=float32), 'im_info': array([[ 600.        ,  774.        ,    3.87096763]], dtype=float32)}
{'gt_boxes': array([[  39.62264252,   43.39622498,  903.77355957,  524.52832031,   26.        ]], dtype=float32), 'data': array([[[[-77.98010254, -80.34010315, -84.58010864, ...,  -7.45025396,
            3.42480731,  13.01989937],
         [-80.34010315, -82.2649765 , -85.72323608, ...,  -5.09025431,
            5.63878345,  15.0848999 ],
         [-84.58010864, -85.72322845, -87.77697754, ...,  -0.8502537 ,
            9.61643696,  18.79489899],
         ..., 
         [-88.13037109, -78.21076965, -60.38908768, ...,   9.65501118,
            2.1556592 ,  -6.50506353],
         [-76.47039795, -76.71351624, -77.15029907, ...,  16.62174225,
            5.38339806,  -8.09505939],
         [-69.98010254, -75.88010406, -86.48010254, ...,  20.4996357 ,
            7.18005562,  -8.98009968]],

        [[-70.94650269, -70.06150818, -68.47150421, ..., -10.59167194,
            0.4584074 ,  10.05350018],
         [-72.42150879, -71.36245728, -69.45975494, ...,  -8.23167229,
            2.67238355,  12.11849976],
         [-75.07150269, -73.69975281, -71.23525238, ...,  -3.99167204,
            6.65003538,  15.82849884],
         ..., 
         [-43.44674301, -34.95051575, -19.68610001, ...,  32.68248749,
           26.5257225 ,  19.22848701],
         [-32.84677124, -34.35691452, -37.07006073, ...,  39.63066483,
           30.802845  ,  19.75848579],
         [-26.94650078, -34.0265007 , -46.74649811, ...,  43.4982338 ,
           33.18362427,  20.05349922]],

        [[-59.771698  , -61.54170227, -64.72169495, ...,  12.40810966,
           25.61318588,  37.22829819],
         [-61.83670425, -63.25860214, -65.8132019 , ...,  14.76811028,
           27.8271637 ,  39.29330063],
         [-65.54669952, -66.34320068, -67.77420044, ...,  19.00811005,
           31.80481529,  43.00329971],
         ..., 
         [-61.62191772, -52.48406982, -36.06690979, ...,  13.50731087,
            7.09069204,  -0.47165117],
         [-52.08194351, -52.79411697, -54.07361984, ...,  19.39549255,
            9.52078056,  -2.59164619],
         [-46.77170181, -52.96670151, -64.09670258, ...,  22.67303276,
           10.8734417 ,  -3.77169991]]]], dtype=float32), 'im_info': array([[ 600.        ,  943.        ,    1.88679242]], dtype=float32)}
{'gt_boxes': array([[   0.        ,    0.        ,  100.        ,  992.30767822,  118.        ]], dtype=float32), 'data': array([[[[ 146.01989746,  146.01989746,  146.01989746, ...,  150.01989746,
           150.01989746,  150.01989746],
         [ 146.01989746,  146.01989746,  146.01989746, ...,  150.01989746,
           150.01989746,  150.01989746],
         [ 146.01989746,  146.01989746,  146.01989746, ...,  150.01989746,
           150.01989746,  150.01989746],
         ..., 
         [ 136.01989746,  136.01989746,  136.01989746, ...,  152.01989746,
           152.01989746,  152.01989746],
         [ 136.01989746,  136.01989746,  136.01989746, ...,  152.01989746,
           152.01989746,  152.01989746],
         [ 136.01989746,  136.01989746,  136.01989746, ...,  152.01989746,
           152.01989746,  152.01989746]],

        [[ 138.05349731,  138.05349731,  138.05349731, ...,  137.05349731,
           137.05349731,  137.05349731],
         [ 138.05349731,  138.05349731,  138.05349731, ...,  137.05349731,
           137.05349731,  137.05349731],
         [ 138.05349731,  138.05349731,  138.05349731, ...,  137.05349731,
           137.05349731,  137.05349731],
         ..., 
         [ 122.05349731,  122.05349731,  122.05349731, ...,  139.05349731,
           139.05349731,  139.05349731],
         [ 122.05349731,  122.053

[JohnnyY8]

@leejiajun Hi!
I noticed that I made some mistakes about the content of your blog. As your blog said, both image ration and object bounding box ratio should be within 0.117-15.500, is that right?

[leejiajun]

@JohnnyY8
They should be constrained in different ratio.

[JohnnyY8]

@leejiajun
Do you mean that the bounding box ratio should be within 0.117-15.5, but the image ratio should be other values? What is the range for image ratio ?

[leejiajun]

@JohnnyY8 I updated my blog http://blog.csdn.net/JiaJunLee/article/details/50470897

[JohnnyY8]

@leejiajun Thanks for your help!
I finished my training, the bounding box ratio in 0.117-15.500 is OK. And in pascal_voc 2007 training set, the value of minimum x, maximum x, minimum y, and maximum y are 127, 500, 96, and 500. So I remove all images without in this size.
BTW. There is a little mistakes.

[leejiajun]

@JohnnyY8
Con!
BTW. These "mistakes" are humorous words.

[JohnnyY8]

@leejiajun really thanks for your help and your time. You are so nice!

[soupault]

Removing -1 from both lib/dataset/$CUSTOM_DATASET.py:

x1 = float(get_data_from_tag(obj, 'xmin')) - 1
y1 = float(get_data_from_tag(obj, 'ymin')) - 1
x2 = float(get_data_from_tag(obj, 'xmax')) - 1
y2 = float(get_data_from_tag(obj, 'ymax')) - 1

and lib/dataset/imdb.py:

boxes[:, 0] = widths[i] - oldx2 - 1
boxes[:, 2] = widths[i] - oldx1 - 1

helped in my case.

[JohnnyY8]

@soupault Hi
It helped for me as well.

[vikiboy]

@Jianchao-ICT @GregorySenay how did you outline which file had empty ground truth values ?

[shubh-agrawal]

I am trying to train py-faster-rcnn on my custom dataset. Untill now I was busy with cleaning out dataset for errors like empty gt_boxes and now am stuck with abrupt crashing of iterations at 1st one itself.
Have a look here. It crashes without any error output.
I0927 19:56:33.645869 5137 solver.cpp:245] Train net output #0: rpn_cls_loss = 0.705536 (* 1 = 0.705536 loss) I0927 19:56:33.645876 5137 solver.cpp:245] Train net output #1: rpn_loss_bbox = 0.0306727 (* 1 = 0.0306727 loss) I0927 19:56:33.645882 5137 sgd_solver.cpp:106] Iteration 0, lr = 0.001
Please give me suggestions and possible erroneous targets to resolve the issue. Thanks!

[moyans]

@JohnnyY8 @rbgirshick @GregorySenay @sunshineatnoon @leefionglee @jojotata Where to find ZF_v2.caffemodel