Models: ValueError: Entries in boundaries must be strictly increasing.

[mahbubcseju]

I was trying to train my own dataset using tensorflow object detection API. So for this, at first I labeled my dataset using labelImg. From that I got a xml file with the labeling information. Then I convert those data to tfrecord using a script. As I was trying to do transfer learning using faster rasnet inception v2 model, I changes the faster_rcnn_inception_v2_pets.config based on my information. Finally I run the model_main.py to train my data. But getting the following value error.

Traceback (most recent call last):
  File "model_main.py", line 109, in <module>
    tf.app.run()
  File "/home/mahbubcseju/anaconda3/envs/kashem/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/home/mahbubcseju/anaconda3/envs/kashem/lib/python3.7/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/home/mahbubcseju/anaconda3/envs/kashem/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "model_main.py", line 105, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "/home/mahbubcseju/anaconda3/envs/kashem/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate
    return executor.run()
  File "/home/mahbubcseju/anaconda3/envs/kashem/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/training.py", line 613, in run
    return self.run_local()
  File "/home/mahbubcseju/anaconda3/envs/kashem/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/training.py", line 714, in run_local
    saving_listeners=saving_listeners)
  File "/home/mahbubcseju/anaconda3/envs/kashem/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 367, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/home/mahbubcseju/anaconda3/envs/kashem/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1158, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/home/mahbubcseju/anaconda3/envs/kashem/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1188, in _train_model_default
    features, labels, ModeKeys.TRAIN, self.config)
  File "/home/mahbubcseju/anaconda3/envs/kashem/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1146, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/home/mahbubcseju/Desktop/Astro/models/research/object_detection/model_lib.py", line 374, in model_fn
    train_config.optimizer)
  File "/home/mahbubcseju/Desktop/Astro/models/research/object_detection/builders/optimizer_builder.py", line 56, in build
    global_step=global_step)
  File "/home/mahbubcseju/Desktop/Astro/models/research/object_detection/builders/optimizer_builder.py", line 124, in _create_learning_rate
    learning_rate_sequence, config.warmup)
  File "/home/mahbubcseju/Desktop/Astro/models/research/object_detection/utils/learning_schedules.py", line 187, in manual_stepping
    raise ValueError('Entries in boundaries must be strictly increasing.')
ValueError: Entries in boundaries must be strictly increasing.

How to get rid of this error. Badly need some suggestion on this.

[tensorflowbutler]

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
What is the top-level directory of the model you are using
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce

[mahbubcseju]

I got the reason of the error. It is because of the wrong configuration of pipeline.config file.

[1997827]

Same error for me .Please fix this

[Nannigalaxy]

I had same problem when I gave manual learning rate schedule in non ascending order.
I assume you have the similar problem.

[Daniyalasif129]

The solution that work for me is
You almost need to start working from middle of your project to remove this error
You had the labeled images, generate CSV and than tf-records files..
than run the training command

[juanssa]

learning_rate: {
manual_step_learning_rate {
initial_learning_rate: 0.0003
schedule {
step: 20000
learning_rate: .00003
}
schedule {
step: 25000
learning_rate: .000003
}
}

The problem is the number of steps, if you put a greater number in the first line and then a smaller number, you will have that error