Models: SSD resnet50 fpn model not detecting at all.

[BalajiB3663]

Please go to Stack Overflow for help and support:

http://stackoverflow.com/questions/tagged/tensorflow

Also, please understand that many of the models included in this repository are experimental and research-style code. If you open a GitHub issue, here is our policy:

1. It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).
2. The form below must be filled out.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

---

System information

• What is the top-level directory of the model you are using:
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
• TensorFlow installed from (source or binary):
• TensorFlow version (use command below):
• Bazel version (if compiling from source):
• CUDA/cuDNN version:
• GPU model and memory:
• Exact command to reproduce:

You can collect some of this information using our environment capture script:

https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh

You can obtain the TensorFlow version with

python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"

Describe the problem

Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.

Source code / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.

[BalajiB3663]

I have custom dataset and trying to perform transfer learning for object detection. I used faster rcnn resnet101 config file and achieved 60% accuracy in detection for real time, as this model gives me 1.5 fps not suitable for real time detection. Since last week i am training SSD resnet50 fpn model, I initialtly found that i missed from_detection_checkpoint: true, so it was not detecting any objects, but after this fix still its in same case, not detecting any objects. Anybody can help me in this?

5028

Thanks,

[rootkitchao]

There are many factors that affect the effect of the object detection model.I tried to train retinanet directly on my dataset without finetune.The model works fine.Need more information (such as configuration files, training situations) to solve your problem.

[BalajiB3663]

@rootkitchao
(https://drive.google.com/open?id=1fCAoa9TH1n3uN6X3HNQuZhcHXC7mDUge)
I trained the same dataset for faster rcnn resnet101 model it works fine, but when i try with ssd resnet50 fpn model its not performing, I have shared my config file with above URL.
image size is resized to 640x640 px and padding is done if needed.
And what these below two lines signifies:
load_all_detection_checkpoint_vars: true
fine_tune_checkpoint_type: "detection"
I have not used in my config file.
Thanks

[rootkitchao]

@rootkitchao
(https://drive.google.com/open?id=1fCAoa9TH1n3uN6X3HNQuZhcHXC7mDUge)
I trained the same dataset for faster rcnn resnet101 model it works fine, but when i try with ssd resnet50 fpn model its not performing, I have shared my config file with above URL.
image size is resized to 640x640 px and padding is done if needed.
And what these below two lines signifies:
load_all_detection_checkpoint_vars: true
fine_tune_checkpoint_type: "detection"
I have not used in my config file.
Thanks

Have you trained all 250,000 steps?What is the final loss value?
SSD and Faster R-CNN are different frameworks.Maybe you need to adjust aspect_ratios and fixed_shape_resizer.

[pkulzc]

You should look at your loss to see if it is decreasing. Also 0.04 is used for 256 batch size and I assume it's too large for your model. Most likely this is not bug, you just need to tune your parameters.

[BalajiB3663]

@pkulzc
After training about 100K steps, the loss value I had is 0.29. I agree with you that learning rate 0.04 is too high, can you please suggest how and how much I should change it or any reference link which can teach me how to tune it? I am little new to this so It would be really helpful.

[pkulzc]

My experience is that you could do a search in exponential space.
E.g. try 0.1, 0.001, 0.0001, 0.00001 and find out the rough range that the proper learning rate lives in, then do some fine tuning.

[BalajiB3663]

@pkulzc
Thanks and I will give it a try and come back.
I have a concern in training custom object detection model, the problem is too many false positives i.e. I have used pre-trainned coco models for my custom dataset, it detects cars and human faces as one of my custom object which as more number of training images. To solve this problem I came across concept of adding background images but not sure how to use background images.
Do i need to create a separate new object class as background? Do i need to annotate background images?
Thanks

[pkulzc]

For background images you can simply add images with no groudtruth boxes to your training data. The API will handle the rest for you.

[BalajiB3663]

@pkulzc
Can you please ellaborate on the same that how to use background images with no groundtruth boxes. I tried it once but was not able to generate tf records.
Thanks for all your help :+1:

[pkulzc]

Are you following this ? I'm not 100% sure but I think you should be able to generate tf.example with empty list ?

[BalajiB3663]

Yes, I was successfully able to create tf record files without passing ground truth values for background. But still I have concern it detects cars and human faces as one of my custom object (False positives) which as more number of training images.

[361720691]

@BalajiB3663 Hi, I trained the pretrained model ssd_mobilenet_v2_coco_2018_03_29 with the config file and I got the same TOO MANY FALSE POSITIVES problem. Have you solved it? Did the background images help?

[BalajiB3663]

@361720691 Hello, yes adding background images is mandatory for training any custom models. And it helped a lot in removing False positive detections. If you know in labelmap you have all your classes with index, but they start with 1 and not with 0, the reason behind this is default 0 index is assigned to background.

[nickguimaraes]

Hello everyone, how exactly can I add the background images to my dataset? Should I add a new class to the labelmap with index 0 called 'Background'? Do I need an .xml file with coordinates 0?

[BalajiB3663]

@nickguimaraes
For background images you can simply add images with no groudtruth boxes or with no xml files to your training data and also dont specify about background on your labelmap as well. The procedure is simple just pass all your background images filenames, width, height and classname is background in your CSV files which you will be needing to generate tf records.

[Ekko1992]

@BalajiB3663
Hi, I'm just wondering how did your training look like after taken rootkitchao' and pkulzc's suggestion(using smaller learning rate and tuning aspect ratios and fixed_shape resizer)? Did you get any progress, and if so, could you please share the number you used?

[BalajiB197]

@Ekko1992
Hello, I learnt that different dataset has a different learning rate. So what I suggest you need to check on experimenting which lr helps you to get the loss appropriately.
I suggest you use cyclic learning rate, you no need to stick to single value.

[Ekko1992]

@BalajiB3663 Thanks for your advice, However, it seems that Cyclic learning rate is still an ongoing/recently added function in tensorflow refer to the commit record in [https://github.com/tensorflow/tensorflow/pull/20785].
What I have been done is modifying the sample configuration file and experiment with different learning rate.

[BalajiB197]

@Ekko1992 I used to follow the same doing changes in sample configuration files for CLR. I was not much aware of this recent commits in tensorflow.

[shamik111691]

@BalajiB3663 could you please tell me how to add background images. Do I just need to add background images to my CSV file? My CSV file has the following specifications:
filename,width,height,class,xmin,ymin,xmax,ymax

So for a background image it should be:
background.jpg,200,200,background,some number,some number,some number,some number

I don't need to add anything to the label .pbtxt file?

From that CSV file I can convert to TFR?

@pkulzc please feel free to comment.

[shamik111691]

@BalajiB197 could you please tell me how to add background images. Do I just need to add background images to my CSV file? My CSV file has the following specifications:
filename,width,height,class,xmin,ymin,xmax,ymax

So for a background image it should be:
background.jpg,200,200,background,some number,some number,some number,some number

I don't need to add anything to the label .pbtxt file?

From that CSV file I can convert to TFR?

[BalajiB197]

@shamik111691 You just need to add your background images filename and its width and height(resolution), rest of the things tensorflow will handle.

Format: imagename, width, height, classname(background)
No need to add in labelmap, as default ID: 0 is assigned for the background.
Yes, you need to convert to TFR, just crosscheck your images size and TFR sizes are the same.

[ZAFERSHAMIM]

@shamik111691 You just need to add your background images filename and its width and height(resolution), rest of the things tensorflow will handle.

Format: imagename, width, height, classname(background)
No need to add in labelmap, as default ID: 0 is assigned for the background.
Yes, you need to convert to TFR, just crosscheck your images size and TFR sizes are the same.

what were the results after doing so , i mean were you to able to eliminate false positives or you were able to reduce the number of false positives

[shamik111691]

I tried it. It somewhat reduced the false positives for me. Not a big difference.

[dinesh1218]

@shamik111691 please help me on this

1. What type of images can be our background images, meaning can it be any image without our object in it?
2. How tf records can be generated for background images? as we will not be cropping our background images(not sure on this) to form xml file

[dinesh1218]

@shamik111691 You just need to add your background images filename and its width and height(resolution), rest of the things tensorflow will handle.

Format: imagename, width, height, classname(background)
No need to add in labelmap, as default ID: 0 is assigned for the background.
Yes, you need to convert to TFR, just crosscheck your images size and TFR sizes are the same.

@BalajiB197 Do I need to manually add the height and width of background images to the CSV file and generate TFrecord?

And also What type of images can be our background images, meaning can it be any image without our object in it?

[BalajiB3663]

@dinesh1218 Yes.. if all your images are of same size its simple.
yes without interested objects. Plus you can also confirm with others.

[dinesh1218]

@BalajiB197 Thanks for your reply. I have tried to add the background image height and width in my CSV file(without xmin,ymin,xmax and ymax) while generating TFRecord i'm facing the below error-

Traceback (most recent call last): File "generate_tfrecord.py", line 115, in <module> tf.app.run() File "/root/.local/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run _sys.exit(main(argv)) File "generate_tfrecord.py", line 106, in main tf_example = create_tf_example(group, path) File "generate_tfrecord.py", line 95, in create_tf_example 'image/object/class/label': dataset_util.int64_list_feature(classes), File "/usr/local/lib/python3.6/dist-packages/object_detection-0.1-py3.6.egg/object_detection/utils/dataset_util.py", line 26, in int64_list_feature return tf.train.Feature(int64_list=tf.train.Int64List(value=value)) TypeError: None has type NoneType, but expected one of: int, long

My background image csv format is

I have also tried adding 0 instead of empty values still same error. Any suggestions on this error? please guide me on how to generate tfrecord for background images.

[BalajiB197]

@dinesh1218 can you share all the classnames which your training(labelmap.pbtxt)
And also you have to check those names are matching in tf_record.py file.

[shamik111691]

@dinesh1218 I think I know what is going on with your python script.
def class_text_to_int(row_label):
if row_label == 'pattern':
return 1
else:
return 0

I think instead of returning zero it is returning None. Can you post your script? There might be other issues which I can help you to resolve.

[dinesh1218]

@BalajiB197 Please find my label_map.pbtxt file content

item {
    id: 1
    name: 'duster'
}

item {
    id: 2
    name: 'battery'
}

I have two classes(Duster and Battery). I think there is no need to mention background class as by default it will be set to 0.

As @shamik111691 mentioned my class_text_to_int function from generate_tfrecord.py returns None. To fix that I have added third label 'background'. Please find the code snippet below

# TO-DO replace this with label map
# for multiple labels add more else if statements
def class_text_to_int(row_label):
    #if row_label == "duster":  # 'ship':
        #return 1
    # comment upper if statement and uncomment these statements for multiple labelling
    if row_label == FLAGS.label0:
        return 1
    elif row_label == FLAGS.label1:
        return  2
    elif row_label == FLAGS.label2:
        return  0
    else:
        None

To generate tfrecord i will execute the following command -

python3 generate_tfrecord.py --label0=duster --label1=battery --label2=background --csv_input=/workspace/training_demo/annotations/train_labels.csv --img_path=/workspace/training_demo/images/train --output_path=/workspace/training_demo/annotations/train.record

@shamik111691 I have attached the whole file below

generate_tfrecord.txt

[shamik111691]

It should work.

[dinesh1218]

@shamik111691 Thanks for your reply. Yeah, its working fine but I faced the error attached below-

This is due to empty values present in xmin,ymin,xmax and ymax for background class. While generating TFrecord I fixed it by replacing nan with 0, then I didn't face any issue.

So, what values should we pass in xmin,ymin,xmax and ymax for background class?

[shamik111691]

for index, row in group.object.iterrows():
xmins.append(row['xmin'] / width)
xmaxs.append(row['xmax'] / width)
ymins.append(row['ymin'] / height)
ymaxs.append(row['ymax'] / height)

In your tf record generator script make sure this loop is not excuted for images with class background.

[shamik111691]

Even for classes and classes_text make sure it is not appended for background images

[dinesh1218]

@shamik111691 Thanks for your reply. Yeah, I have skipped the above snippet for background class. I have generated tfrecords and trained the model. But the results were same like the model trained with only two classes(only duster and battery).

Here are the steps I followed -

1. Generate CSV files for train and test set. And add only filename,width,height,class for background class(Note: background class is added only in train set and not in test set)
2. Generate TFRecords for train and test set by skipping the xmin,ymin,xmax and ymax values for background class.
3. Trained the model with ~63k steps and exported the inference graph.
4. Tested the model with live webcam, it detects empty frame(no duster or battery objects in frame) and also a person as battery class.

Am I missing any steps for Object detection, please guide me on this

[shamik111691]

In case of false positives I think the best way is to check your training set if it has anything that contains the aforesaid false positive as background.

[dinesh1218]

@shamik111691 Sure will check it. 90% of training images have only white background. Does it have any effect on detecting false positives?

[shamik111691]

I would say yes. CNNs learn both the object and the background so as to better understand what is the object and what is NOT the object. So it is good to have a variety of backgrounds in your training images to facilitate the training process

[dinesh1218]

@shamik111691 Thank you. Will update my training set accordingly and check the process.

[SLAMgreen]

@pkulzc
Thanks and I will give it a try and come back.
I have a concern in training custom object detection model, the problem is too many false positives i.e. I have used pre-trainned coco models for my custom dataset, it detects cars and human faces as one of my custom object which as more number of training images. To solve this problem I came across concept of adding background images but not sure how to use background images.
Do i need to create a separate new object class as background? Do i need to annotate background images?
Thanks

Are you following this ? I'm not 100% sure but I think you should be able to generate tf.example with empty list ?

Hi,
What the meaning of background image ? What is the rule to pick up the background images?