Models: new object_detection error from today's commis: 'ManualStepLearningRate' object has no attribute 'warmup'

Created on 22 Mar 2018  路  24Comments  路  Source: tensorflow/models

After pulled today's changes, got an error in previously working training (warmup was not in config file at all)

Traceback (most recent call last):
  File "object_detection/train.py", line 167, in <module>
    tf.app.run()
  File "/home/dsu/tf/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 126, in run
    _sys.exit(main(argv))
  File "object_detection/train.py", line 163, in main
    worker_job_name, is_chief, FLAGS.train_dir)
  File "/home/dsu/ai/tfmodels/research/object_detection/trainer.py", line 255, in train
    train_config.optimizer)
  File "/home/dsu/ai/tfmodels/research/object_detection/builders/optimizer_builder.py", line 50, in build
    learning_rate = _create_learning_rate(config.learning_rate)
  File "/home/dsu/ai/tfmodels/research/object_detection/builders/optimizer_builder.py", line 109, in _create_learning_rate
    learning_rate_sequence, config.warmup)
AttributeError: 'ManualStepLearningRate' object has no attribute 'warmup'
picture jwnsu
👍4

Most helpful comment

I believe it is the same Python3 incompatibility that has crept up before (see #3443 ). The issue is with models/research/object_detection/utils/learning_schedules.py lines 167-169. Currently it is 

rate_index = tf.reduce_max(tf.where(tf.greater_equal(global_step, boundaries),
                                      range(num_boundaries),
                                      [0] * num_boundaries))

Wrap list() around the range() like this:

rate_index = tf.reduce_max(tf.where(tf.greater_equal(global_step, boundaries),
                                     list(range(num_boundaries)),
                                      [0] * num_boundaries))

and you should be good to go. Mine is off and training.

@jwnsu make sure you change your pipeline.config from:

        manual_step_learning_rate {
          initial_learning_rate: 0.00001
          schedule {
            step: 0
            learning_rate: .0003
          }
          schedule {
            step: 900000
            learning_rate: .00003
          }
          schedule {
            step: 1200000
            learning_rate: .000003
          }
        }
      }
      momentum_optimizer_value: 0.9
    }
    use_moving_average: false
  }

to the new slightly different:

        manual_step_learning_rate {
          initial_learning_rate: 0.0003
          schedule {
            step: 900000
            learning_rate: .00003
          }
          schedule {
            step: 1200000
            learning_rate: .000003
          }
        }
      }
      momentum_optimizer_value: 0.9
    }
    use_moving_average: false
  }
picture learnbott on 23 Mar 2018
👍96 ❤38 🎉27 😄4

All 24 comments

I had the same problem, but then realized I need to update my pipeline.config to conform to the new manualsteplearningrate. After I made the adjustments I get this error:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See tf.nn.softmax_cross_entropy_with_logits_v2.

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 510, in _apply_op_helper
    preferred_dtype=default_dtype)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1022, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py", line 233, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py", line 212, in constant
    value, dtype=dtype, shape=shape, verify_shape=verify_shape))
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/tensor_util.py", line 422, in make_tensor_proto
    _GetDenseDimensions(values)))
ValueError: Argument must be a dense tensor: range(0, 3) - got shape [3], but wanted [].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 524, in _apply_op_helper
    values, as_ref=input_arg.is_ref).dtype.name
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1022, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py", line 233, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py", line 212, in constant
    value, dtype=dtype, shape=shape, verify_shape=verify_shape))
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/tensor_util.py", line 422, in make_tensor_proto
    _GetDenseDimensions(values)))
ValueError: Argument must be a dense tensor: range(0, 3) - got shape [3], but wanted [].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/work/repos/models/research/object_detection/train.py", line 167, in <module>
    tf.app.run()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 124, in run
    _sys.exit(main(argv))
  File "/work/repos/models/research/object_detection/train.py", line 163, in main
    worker_job_name, is_chief, FLAGS.train_dir)
  File "/work/repos/models/research/object_detection/trainer.py", line 255, in train
    train_config.optimizer)
  File "/work/repos/models/research/object_detection/builders/optimizer_builder.py", line 58, in build
    learning_rate = _create_learning_rate(config.learning_rate)
  File "/work/repos/models/research/object_detection/builders/optimizer_builder.py", line 109, in _create_learning_rate
    learning_rate_sequence, config.warmup)
  File "/work/repos/models/research/object_detection/utils/learning_schedules.py", line 169, in manual_stepping
    [0] * num_boundaries))
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/array_ops.py", line 2540, in where
    return gen_math_ops._select(condition=condition, x=x, y=y, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_math_ops.py", line 4043, in _select
    "Select", condition=condition, t=x, e=y, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 528, in _apply_op_helper
    (input_name, err))
ValueError: Tried to convert 't' to a tensor and failed. Error: Argument must be a dense tensor: range(0, 3) - got shape [3], but wanted [].
learnbott picture learnbott on 23 Mar 2018

I get this error too

VinceKumar picture VinceKumar on 23 Mar 2018

I believe it is the same Python3 incompatibility that has crept up before (see #3443 ). The issue is with models/research/object_detection/utils/learning_schedules.py lines 167-169. Currently it is 

rate_index = tf.reduce_max(tf.where(tf.greater_equal(global_step, boundaries),
                                      range(num_boundaries),
                                      [0] * num_boundaries))

Wrap list() around the range() like this:

rate_index = tf.reduce_max(tf.where(tf.greater_equal(global_step, boundaries),
                                     list(range(num_boundaries)),
                                      [0] * num_boundaries))

and you should be good to go. Mine is off and training.

@jwnsu make sure you change your pipeline.config from:

        manual_step_learning_rate {
          initial_learning_rate: 0.00001
          schedule {
            step: 0
            learning_rate: .0003
          }
          schedule {
            step: 900000
            learning_rate: .00003
          }
          schedule {
            step: 1200000
            learning_rate: .000003
          }
        }
      }
      momentum_optimizer_value: 0.9
    }
    use_moving_average: false
  }

to the new slightly different:

        manual_step_learning_rate {
          initial_learning_rate: 0.0003
          schedule {
            step: 900000
            learning_rate: .00003
          }
          schedule {
            step: 1200000
            learning_rate: .000003
          }
        }
      }
      momentum_optimizer_value: 0.9
    }
    use_moving_average: false
  }
learnbott picture learnbott on 23 Mar 2018
👍96 ❤38 🎉27 😄4

same problem as here https://github.com/tensorflow/models/issues/3605

jrabary picture jrabary on 23 Mar 2018

manual_step_learning_rate is not used directly, seems a new bug, here is the config:

optimizer {
    momentum_optimizer: {
      learning_rate: {
        manual_step_learning_rate {
          initial_learning_rate: 0.0003
          schedule {
            step: 900000
            learning_rate: .0003
          }
          schedule {
            step: 1200000
            learning_rate: .000003
          }
        }
      }
      momentum_optimizer_value: 0.9
    }
    use_moving_average: false
jwnsu picture jwnsu on 23 Mar 2018

@learnbott I was having the errors you had mentioned before:
ValueError: Argument must be a dense tensor: range(0, 3) - got shape [3], but wanted []
The wraping of list worked for me. Thanks a ton!

allendoss picture allendoss on 23 Mar 2018

remember to recompile protobufs after upgrade: 

# From tensorflow/models/research/
protoc object_detection/protos/*.proto --python_out=.
jlertle picture jlertle on 28 Mar 2018
👍2

I meet this issue too, but thanks to @learnbott , and now I can move on.

huzq85 picture huzq85 on 4 Apr 2018

you rock @learnbott

roy-d picture roy-d on 7 Apr 2018
❤1

when will this fix be merged?

FalcoGer picture FalcoGer on 12 Apr 2018

@learnbott
The wraping of list did not work for me at first.
ValueError: Tried to convert 't' to a tensor and failed. Error: Argument must be a dense tensor: range(0, 3) - got shape [3], but wanted [].
but then I run the following commands in my tensorflow path

\models\research> python setup.py build
\models\research> python setup.py install

Then it works

AliceDinh picture AliceDinh on 10 May 2018
👍8 🎉3

please help me to resolve the issue

(py36_tf_cpu) C:\tensorflow\models\research\object_detection>python train.py --l
ogtostderr --train_dir=training/ --pipeline_config_path=training/faster_rcnn_inc
eption_v2_pets.config
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:Scale of 0 disables regularizer.
Traceback (most recent call last):
File "train.py", line 184, in
tf.app.run()
File "C:\Users\Test\Anaconda3\envs\py36_tf_cpu\lib\site-packages\tensorflow\py
thon\platform\app.py", line 124, in run
_sys.exit(main(argv))
File "train.py", line 180, in main
graph_hook_fn=graph_rewriter_fn)
File "C:\tensorflow\models\research\object_detection\trainer.py", line 288, in
train
train_config.optimizer)
File "C:\tensorflow\models\research\object_detection\builders\optimizer_builde
r.py", line 50, in build
learning_rate = _create_learning_rate(config.learning_rate)
File "C:\tensorflow\models\research\object_detection\builders\optimizer_builde
r.py", line 109, in _create_learning_rate
learning_rate_sequence, config.warmup)
File "C:\tensorflow\models\research\object_detection\utils\learning_schedules.
py", line 156, in manual_stepping
raise ValueError('First step cannot be zero.')
ValueError: First step cannot be zero.

srunitha picture srunitha on 28 May 2018

The wrapping did now work for me. I also tried what AliceDinh Suggested. But it is still giving me the same error.

Deepshikhasinha picture Deepshikhasinha on 1 Jun 2018
👍2

@srunitha same here did u find the solution
Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See @{tf.nn.softmax_cross_entropy_with_logits_v2}.

Traceback (most recent call last):
File "train.py", line 184, in
tf.app.run()
File "C:\Users\JLANKA_PC\Anaconda3\lib\site-packages\tensorflow\python\platform\app.py", line 126, in run
_sys.exit(main(argv))
File "train.py", line 180, in main
graph_hook_fn=graph_rewriter_fn)
File "C:\tensorflow1\models\research\object_detection\trainer.py", line 298, in train
train_config.optimizer)
File "C:\tensorflow1\models\research\object_detection\builders\optimizer_builder.py", line 50, in build
learning_rate = _create_learning_rate(config.learning_rate)
File "C:\tensorflow1\models\research\object_detection\builders\optimizer_builder.py", line 109, in _create_learning_rate
learning_rate_sequence, config.warmup)
File "C:\tensorflow1\models\research\object_detection\utils\learning_schedules.py", line 156, in manual_stepping
raise ValueError('First step cannot be zero.')
ValueError: First step cannot be zero.

NatheesanSelvaratnam picture NatheesanSelvaratnam on 14 Jun 2018

list wrapping and python setup build and install worked for me

alouisos picture alouisos on 18 Jun 2018

@learnbott solution didn't work for me.
I changed range(num_boundaries) to [i for i in range(num_boundaries)] in \models\research\object_detection\utils\learning_schedules.py
Then:
\models\research> python setup.py build
\models\research> python setup.py install
and worked

xro7 picture xro7 on 20 Jul 2018
👍1

how can i solve this error?
raceback (most recent call last):
File "export_inference_graph.py", line 150, in
tf.app.run()
File "C:\Users\DELL LAPTOP\Miniconda3\envs\tfp3.6\lib\site-packages\tensorflow\python\platform\app.py", line 125, in run
_sys.exit(main(argv))
File "export_inference_graph.py", line 146, in main
write_inference_graph=FLAGS.write_inference_graph)
File "C:\Users\DELL LAPTOP\Miniconda3\envs\tfp3.6\lib\site-packages\object_detection-0.1-py3.6.egg\object_detection\exporter.py", line 389, in export_inference_graph
is_training=False)
File "C:\Users\DELL LAPTOP\Miniconda3\envs\tfp3.6\lib\site-packages\object_detection-0.1-py3.6.egg\object_detection\builders\model_builder.py", line 121, in build
raise ValueError('Unknown meta architecture: {}'.format(meta_architecture))
ValueError: Unknown meta architecture: None

Shubhamrock428 picture Shubhamrock428 on 8 Oct 2018

ImportError Traceback (most recent call last)
in
10 from collections import defaultdict
11 from io import StringIO
---> 12 from matplotlib import pyplot as plt
13 from PIL import Image
14

~\AppData\Roaming\Python\Python36\site-packages\matplotlib\pyplot.py in
30 from cycler import cycler
31 import matplotlib
---> 32 import matplotlib.colorbar
33 import matplotlib.image
34 from matplotlib import rcsetup, style

~\AppData\Roaming\Python\Python36\site-packages\matplotlib\colorbar.py in
26
27 import matplotlib as mpl
---> 28 import matplotlib.artist as martist
29 import matplotlib.cbook as cbook
30 import matplotlib.collections as collections

~\AppData\Roaming\Python\Python36\site-packages\matplotlib\artist.py in
9 import matplotlib
10 from . import cbook, docstring, rcParams
---> 11 from .path import Path
12 from .transforms import (Bbox, IdentityTransform, Transform, TransformedBbox,
13 TransformedPatchPath, TransformedPath)

~\AppData\Roaming\Python\Python36\site-packages\matplotlib\path.py in
15 import numpy as np
16
---> 17 from . import _path, rcParams
18 from .cbook import _to_unmasked_float_array, simple_linear_interpolation
19

ImportError: cannot import name '_path'
how to solve this error?

mitarthjain picture mitarthjain on 8 Dec 2018

in the last section of code:-
NameError Traceback (most recent call last)
in
1 for image_path in TEST_IMAGE_PATHS:
----> 2 image = Image.open(image_path)
3 # the array based representation of the image will be used later in order to prepare the
4 # result image with boxes and labels on it.
5 image_np = load_image_into_numpy_array(image)

NameError: name 'Image' is not defined
How to solve this one??

mitarthjain picture mitarthjain on 8 Dec 2018

While executing the train.py file i get the following error:
Traceback (most recent call last):
File "train.py", line 49, in
from object_detection.builders import dataset_builder
ImportError: No module named 'object_detection'
Please help I'm new to this

yadnyeshn picture yadnyeshn on 3 Feb 2019

Where do i need to change some line codes about boxes because i'm planning to set color green if the accuracy >=98 then red if not. i'm using real-time :D

jamesdoom22 picture jamesdoom22 on 1 Mar 2019

Though running successfully the jupyter notebook, the output images is not displaying. anywhere I gone wrng? PLSS HELP!! :)

msdhina picture msdhina on 29 Mar 2019

@yadnyeshn It鈥榮 easy~ Confirm your pythonpath is added again~

InspirationLaurie picture InspirationLaurie on 1 Jun 2019

After pulled today's changes, got an error in previously working training (warmup was not in config file at all)

Traceback (most recent call last):
  File "object_detection/train.py", line 167, in <module>
    tf.app.run()
  File "/home/dsu/tf/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 126, in run
    _sys.exit(main(argv))
  File "object_detection/train.py", line 163, in main
    worker_job_name, is_chief, FLAGS.train_dir)
  File "/home/dsu/ai/tfmodels/research/object_detection/trainer.py", line 255, in train
    train_config.optimizer)
  File "/home/dsu/ai/tfmodels/research/object_detection/builders/optimizer_builder.py", line 50, in build
    learning_rate = _create_learning_rate(config.learning_rate)
  File "/home/dsu/ai/tfmodels/research/object_detection/builders/optimizer_builder.py", line 109, in _create_learning_rate
    learning_rate_sequence, config.warmup)
AttributeError: 'ManualStepLearningRate' object has no attribute 'warmup'

from nets import inception_resnet_v2
ModuleNotFoundError: No module named 'nets'
how to solve this problem

qiu-pinggaizi picture qiu-pinggaizi on 1 Jul 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

trungdn picture trungdn  路  3Comments
sun9700 picture sun9700  路  3Comments
rakashi picture rakashi  路  3Comments
amirjamez picture amirjamez  路  3Comments
chenyuZha picture chenyuZha  路  3Comments