Models: tensorflow slim giving error while training train_image_classifier.py

Created on 14 Aug 2017 · 14Comments · Source: tensorflow/models

Hi, I am trying to train my own model using inception_v3 architecture with using small sample data taken as flowers data

when I am trying to run the following command, its giving the error
python train_image_classifier.py --train_dir=${TRAIN_DIR} --dataset_name=flowers --dataset_split_name=train --dataset_dir=${DATASET_DIR} --model_name=inception_v3

the error is like this

(/home/mcis-lap-49/work/anaconda2) mcis-lap-49@MCIS-LAP-49:~/slim$ python train_image_classifier.py --train_dir=${TRAIN_DIR} --dataset_name=flowers --dataset_split_name=train --dataset_dir=${DATASET_DIR} --model_name=inception_v3
INFO:tensorflow:Summary name /clone_loss is illegal; using clone_loss instead.
2017-08-14 15:14:01.398254: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-14 15:14:01.398303: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-14 15:14:01.398317: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-08-14 15:14:01.398328: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-14 15:14:01.398338: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
INFO:tensorflow:Error reported to Coordinator: , Cannot assign a device for operation 'gradients/InceptionV3/InceptionV3/Conv2d_1a_3x3/BatchNorm/moments/Sub_grad/BroadcastGradientArgs': Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/cpu:0 ]. Make sure the device specification refers to a valid device.
[[Node: gradients/InceptionV3/InceptionV3/Conv2d_1a_3x3/BatchNorm/moments/Sub_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device="/device:GPU:0"](gradients/InceptionV3/InceptionV3/Conv2d_1a_3x3/BatchNorm/moments/Sub_grad/Shape, gradients/InceptionV3/InceptionV3/Conv2d_1a_3x3/BatchNorm/moments/Sub_grad/Shape_1)]]

Caused by op u'gradients/InceptionV3/InceptionV3/Conv2d_1a_3x3/BatchNorm/moments/Sub_grad/BroadcastGradientArgs', defined at:
File "train_image_classifier.py", line 573, in
tf.app.run()
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "train_image_classifier.py", line 533, in main
var_list=variables_to_train)
File "/home/mcis-lap-49/slim/deployment/model_deploy.py", line 297, in optimize_clones
optimizer, clone, num_clones, regularization_losses, *kwargs)
File "/home/mcis-lap-49/slim/deployment/model_deploy.py", line 261, in _optimize_clone
clone_grad = optimizer.compute_gradients(sum_loss, *kwargs)
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/optimizer.py", line 386, in compute_gradients
colocate_gradients_with_ops=colocate_gradients_with_ops)
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py", line 540, in gradients
grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py", line 346, in _MaybeCompile
return grad_fn() # Exit early
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py", line 540, in
grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/math_grad.py", line 650, in _SubGrad
rx, ry = gen_array_ops._broadcast_gradient_args(sx, sy)
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 395, in _broadcast_gradient_args
name=name)
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2506, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1269, in __init__
self._traceback = _extract_stack()

...which was originally created as op u'InceptionV3/InceptionV3/Conv2d_1a_3x3/BatchNorm/moments/Sub', defined at:
File "train_image_classifier.py", line 573, in
tf.app.run()
[elided 0 identical lines from previous traceback]
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "train_image_classifier.py", line 472, in main
clones = model_deploy.create_clones(deploy_config, clone_fn, [batch_queue])
File "/home/mcis-lap-49/slim/deployment/model_deploy.py", line 193, in create_clones
outputs = model_fn(args, *kwargs)
File "train_image_classifier.py", line 455, in clone_fn
logits, end_points = network_fn(images)
File "/home/mcis-lap-49/slim/nets/nets_factory.py", line 114, in network_fn
return func(images, num_classes, is_training=is_training)
File "/home/mcis-lap-49/slim/nets/inception_v3.py", line 483, in inception_v3
depth_multiplier=depth_multiplier)
File "/home/mcis-lap-49/slim/nets/inception_v3.py", line 104, in inception_v3_base
net = slim.conv2d(inputs, depth(32), [3, 3], stride=2, scope=end_point)
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 181, in func_with_args
return func(args, *current_args)
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 956, in convolution
outputs = normalizer_fn(outputs, *normalizer_params)
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 181, in func_with_args
return func(args, *current_args)
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 554, in batch_norm
outputs = layer.apply(inputs, training=is_training)
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/layers/base.py", line 492, in apply
return self.__call__(inputs, *args, *kwargs)
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/layers/base.py", line 441, in __call__
outputs = self.call(inputs, args, *kwargs)

InvalidArgumentError (see above for traceback): Cannot assign a device for operation 'gradients/InceptionV3/InceptionV3/Conv2d_1a_3x3/BatchNorm/moments/Sub_grad/BroadcastGradientArgs': Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/cpu:0 ]. Make sure the device specification refers to a valid device.
[[Node: gradients/InceptionV3/InceptionV3/Conv2d_1a_3x3/BatchNorm/moments/Sub_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device="/device:GPU:0"](gradients/InceptionV3/InceptionV3/Conv2d_1a_3x3/BatchNorm/moments/Sub_grad/Shape, gradients/InceptionV3/InceptionV3/Conv2d_1a_3x3/BatchNorm/moments/Sub_grad/Shape_1)]]

Traceback (most recent call last):
File "train_image_classifier.py", line 573, in
tf.app.run()
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "train_image_classifier.py", line 569, in main
sync_optimizer=optimizer if FLAGS.sync_replicas else None)
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/learning.py", line 732, in train
master, start_standard_services=False, config=session_config) as sess:
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 964, in managed_session
self.stop(close_summary_writer=close_summary_writer)
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 792, in stop
stop_grace_period_secs=self._stop_grace_secs)
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 953, in managed_session
start_standard_services=start_standard_services)
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 708, in prepare_or_wait_for_session
init_feed_dict=self._init_feed_dict, init_fn=self._init_fn)
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/session_manager.py", line 279, in prepare_session
sess.run(init_op, feed_dict=init_feed_dict)
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 789, in run
run_metadata_ptr)
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 997, in _run
feed_dict_string, options, run_metadata)
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1132, in _do_run
target_list, options, run_metadata)
File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1152, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation 'gradients/InceptionV3/InceptionV3/Conv2d_1a_3x3/BatchNorm/moments/Sub_grad/BroadcastGradientArgs': Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/cpu:0 ]. Make sure the device specification refers to a valid device.
[[Node: gradients/InceptionV3/InceptionV3/Conv2d_1a_3x3/BatchNorm/moments/Sub_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device="/device:GPU:0"](gradients/InceptionV3/InceptionV3/Conv2d_1a_3x3/BatchNorm/moments/Sub_grad/Shape, gradients/InceptionV3/InceptionV3/Conv2d_1a_3x3/BatchNorm/moments/Sub_grad/Shape_1)]]

ERROR:tensorflow:==================================
Object was never used (type ):

If you want to mark it as used call its "mark_used()" method.
It was originally created here:

['File "train_image_classifier.py", line 573, in \n tf.app.run()', 'File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run\n _sys.exit(main(_sys.argv[:1] + flags_passthrough))', 'File "train_image_classifier.py", line 569, in main\n sync_optimizer=optimizer if FLAGS.sync_replicas else None)', 'File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/learning.py", line 655, in train\n ready_op = tf_variables.report_uninitialized_variables()', 'File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/util/tf_should_use.py", line 170, in wrapped\n return _add_should_use_warning(fn(args, *kwargs))', 'File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/util/tf_should_use.py", line 139, in _add_should_use_warning\n wrapped = TFShouldUseWarningWrapper(x)', 'File "/home/mcis-lap-49/work/anaconda2/lib/python2.7/site-packages/tensorflow/python/util/tf_should_use.py", line 96, in init\n stack = [s.strip() for s in traceback.format_stack()]']

I didn't understand what is this error.... please help me

awaiting response

Source

rakashi

Most helpful comment

I solved this issue by providing an extra arg '--clone_on_cpu=True' while running ( which is False by default) i.e.

python train_image_classifier.py --clone_on_cpu=True --train_dir=${TRAIN_DIR} --dataset_name=flowers --dataset_split_name=train --dataset_dir=${DATASET_DIR} --model_name=inception_v3

sreeragh-ar on 21 Nov 2017

👍30 🎉7 ❤4

All 14 comments

There's a lot of output but I think this is the important part of the stack traces:

Cannot assign a device for operation 'gradients/InceptionV3/InceptionV3/Conv2d_1a_3x3/BatchNorm/moments/Sub_grad/BroadcastGradientArgs': Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/cpu:0 ]. Make sure the device specification refers to a valid device.

Looks like you're trying to assign to a GPU but you don't have one.

skye on 14 Aug 2017

@skye i have one 1080ti installed on my computer and i still got the error ?? do you know how to fix it ?
when i run normal inception it runs fine.

vaibhav0195 on 7 Sep 2017

Hi @vaibhav0195 , I'm guessing you haven't installed a TF version with GPU configuration, but it's hard to say for certain. Please post on StackOverflow for more help.

skye on 7 Sep 2017

👍1

@skye I have fixed the issue it is due the kernel problem.

vaibhav0195 on 8 Sep 2017

Can you specify how to fix the issue? @vaibhav0195 I got exactly the same problem

diana12333 on 10 Nov 2017

I solved this issue by providing an extra arg '--clone_on_cpu=True' while running ( which is False by default) i.e.

python train_image_classifier.py --clone_on_cpu=True --train_dir=${TRAIN_DIR} --dataset_name=flowers --dataset_split_name=train --dataset_dir=${DATASET_DIR} --model_name=inception_v3

sreeragh-ar on 21 Nov 2017

👍30 🎉7 ❤4

I also encountered the same problem . Could you please talk about how you fix it ?And the kernel problem means? Thanks @vaibhav0195

zapersea on 30 Nov 2017

@SragAR thats is a flag to use cpu not gpu. cloning on cpu will take forever to train especially if you have large dataset. I solved the same issue by reinstalling tenserflow as gpu as per @skye comment. thanks.
pip install tensorflow-gpu==1.2.1 (maybe it is wont work with 1.3.0 version of tensorflow)

AmanuelHirpa on 19 Jan 2018

❤1

I encounter a similar problem, and I try too pass a parameter "session_config = tf.ConfigProto(allow_soft_placement=True)" to slim.learning.train().

robindume on 7 May 2018

🎉4 ❤2 😄2 🚀1

@robindume Do you succeed? I am looking for this too and but have no idea about it.

GraceDSeven on 10 May 2018

@robindume I found the answer:

slim.learning.train(
train_tensor,
session_config=tf.ConfigProto(allow_soft_placement=True),
logdir=FLAGS.train_dir,
master=FLAGS.master,
is_chief=(FLAGS.task == 0),
init_fn=_get_init_fn(),
summary_op=summary_op,
number_of_steps=FLAGS.max_number_of_steps,
log_every_n_steps=FLAGS.log_every_n_steps,
save_summaries_secs=FLAGS.save_summaries_secs,
save_interval_secs=FLAGS.save_interval_secs,
sync_optimizer=optimizer if FLAGS.sync_replicas else None)

GraceDSeven on 12 May 2018

🎉3

@robindume @GraceDSeven This would make your code run but only on CPU. It takes forever if you have a large dataset!

mahyaar on 25 May 2018

Two ways to solve it,
1) Go to the code find the flag clone-on-cpu by default, it is set as false, make it true
2) In the script set the flag clone-on-cpu=True