Models: warnings during retraining the model

Created on 30 Aug 2018 · 15Comments · Source: tensorflow/models

Hi,

I'm getting following warnings when I'm to training my model, I am using SSD. Please someone help me.

Thank you

WARNING:tensorflow:Ignoring ground truth with image id 2132974076 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 2132974076 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 483474429 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 483474429 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 1042541771 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 1042541771 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 374107777 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 374107777 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 851704672 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 851704672 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 1267094741 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 1267094741 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 674017641 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 674017641 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 2011324514 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 2011324514 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 655823972 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 655823972 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 1069886348 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 1069886348 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 432647899 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 432647899 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 192947873 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 192947873 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 1881834530 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 1881834530 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 309074883 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 309074883 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 515532141 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 515532141 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 1804293832 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 1804293832 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 1933803716 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 1933803716 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 144546819 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 144546819 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 1887856555 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 1887856555 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 1334524773 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 1334524773 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 642435942 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 642435942 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 2089332883 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 2089332883 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 963719283 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 963719283 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 788508605 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 788508605 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 1557220932 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 1557220932 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 2013002728 since it was previously added

Source

jillelajitta

👍1

Most helpful comment

Hi,
this is probably due to the fact that the number of images on your test dataset is not equal to the one declared in your .config file.
For example, if your dataset is made up of 480 images for test, then you should set your config file as:

eval_config: {
  num_examples: 480
  eval_interval_secs: 150
  # Note: The below line limits the evaluation process to 10 evaluations.
  # Remove the below line to evaluate indefinitely.
  # max_evals: 10

mawanda-jun on 2 Sep 2018

👍8

All 15 comments

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
What is the top-level directory of the model you are using
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce

tensorflowbutler on 31 Aug 2018

eval_config: {
  num_examples: 480
  eval_interval_secs: 150
  # Note: The below line limits the evaluation process to 10 evaluations.
  # Remove the below line to evaluate indefinitely.
  # max_evals: 10

mawanda-jun on 2 Sep 2018

👍8

@mawanda-jun I gave the correct no of images but still I am getting the same error.

jillelajitta on 28 Sep 2018

hi @jillelajitta are you able to solve your problem?
I still encountered the same issue, when I have batch_size = 1 in pipeline.config and I limit my num_train_steps and num_eval_steps to my number of train/eval records respectively when submitting training job.

The train job ends in about 10 minutes, but only yield one ckpt at step 520. I checked tensorboard there is only one dot at the loss plot.

billybee on 10 Oct 2018

The warning doesn't affect the result of the evaluation step though (although they're pretty annoying). I still obtained the evaluation metrics results on the number of evaluation examples I specified at end of the evaluation run. You can monitor the GPU utilization rate to see that the evaluation is being carried out while the warnings are printing.

I don't think the number of evaluation examples needs to equal to the size of the validation set, since often people just want the metrics computed for a subset/sample of the val set every ~10 minutes instead of the entire val set to save time @mawanda-jun

yangsiyu007 on 23 Oct 2018

The warning doesn't affect the result of the evaluation step though (although they're pretty annoying). I still obtained the evaluation metrics results on the number of evaluation examples I specified at end of the evaluation run. You can monitor the GPU utilization rate to see that the evaluation is being carried out while the warnings are printing.

I don't think the number of evaluation examples needs to equal to the size of the validation set, since often people just want the metrics computed for a subset/sample of the val set every ~10 minutes instead of the entire val set to save time @mawanda-jun

I agree with you . The warning doesn't affect the result. here

Code-Fight on 18 Jan 2019

Hi
If the warnings are still there, then again check your generate_tfrecord.py file. I had changed the tfrecord file for my convenience and there was some bug in it. I would suggest just check whether you tfrecord file is correctly showing the ground truth boxes. And then only proceed for training.

QuickLearner171998 on 6 Jun 2019

@QuickLearner171998 what exactly was the bug? I'm facing the same issue but I can't find the root of the problem.

Thommy257 on 2 Aug 2019

@QuickLearner171998 what exactly was the bug? I'm facing the same issue but I can't find the root of the problem.
I made some error in writing the loop and due to this wrong image matrices were being fed in tf example. After I corrected that the warnings disappeared.
Try setting num_examples to the number of validation images.

QuickLearner171998 on 2 Aug 2019

I made some error in writing the loop and due to this wrong image matrices were being fed in tf example. After I corrected that the warnings disappeared.
Try setting num_examples to the number of validation images.

Thanks, but it seems suspicious that the training is working without any problem. I use the same script for both train.record and val.record file. I'll have a closer look at this.

Thommy257 on 2 Aug 2019

fixed it by reinstalling the whole object detection api. Who knows what I did wring...

Thommy257 on 2 Aug 2019

👀1

Hi There,
We are checking to see if you still need help on this, as this seems to be considerably old issue. Please update this issue with the latest information, code snippet to reproduce your issue and error you are seeing.
If we don't hear from you in the next 7 days, this issue will be closed automatically. If you don't need help on this issue any more, please consider closing this.

tensorflowbutler on 30 Jan 2020

it might be because of sample_1_of_n_examples, in the eval section of config file. If you don't want to multiply the samples of your evaluation dataset just delete this parameter or set to 1 in the configuration file.

OctavDM on 12 Mar 2020

Hi,
this is probably due to the fact that the number of images on your test dataset is not equal to the one declared in your .config file.
For example, if your dataset is made up of 480 images for test, then you should set your config file as:
eval_config: {
  num_examples: 480
  eval_interval_secs: 150
  # Note: The below line limits the evaluation process to 10 evaluations.
  # Remove the below line to evaluate indefinitely.
  # max_evals: 10

This assumes batch size of 1 ? If batch_size is 2, then num_examples should be set to 240?

tispratik on 9 Apr 2020

👀1

I have not seen num_examples have an effect in any of my tests. Regardless of what I list for this parameter, the number of evaluation examples is equal to the samples listed in the eval_input_reader input_path list up to a maximum of 1000 samples. Is this a bug or am I missing something?