Models: Im2Txt not working for other languages like hindi

Created on 1 May 2017  路  18Comments  路  Source: tensorflow/models


System information

  • What is the top-level directory of the model you are using models/Im2Txt:
  • OS Platform and Distribution : Linux CentOS 7:
  • TensorFlow installed from Binary:
  • TensorFlow version:('v1.0.0-rc1-102-g1536a84-dirty', '1.0.0-rc2'):

I am trying to use im2txt model with MSCOCO dataset modified to use hindi caption. When I tried to pre-process dataset using bazel-bin/im2txt/download_and_preprocess_mscoco "${MSCOCO_DIR}" command, it will work until generating word_count file. In step of generating TFrecord it throws following error:

Exception in thread Thread-14:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(self.__args, *self.__kwargs)
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 281, in _process_image_files
sequence_example = _to_sequence_example(image, decoder, vocab)
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 234, in _to_sequence_example
"image/caption": _bytes_feature_list(caption),
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 202, in _bytes_feature_list
return tf.train.FeatureList(feature=[_bytes_feature(v) for v in values])
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 192, in _bytes_feature
return tf.train.Feature(bytes_list=tf.train.BytesList(value=str(value)))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)

It seems error in encoding data to tfrecord. I have also tried to modify str(value) using unicode encode function.
Sample caption file is available at: https://drive.google.com/file/d/0B8Rng3ofk0uAbDBJekpPR3BXVzA/view?usp=sharing

bug

Most helpful comment

I've had the same issue because of accented characters (language: Italian) and what worked out for me was resetting the default encoding before tf.app.run():

if __name__ == "__main__":

  reload(sys)
  sys.setdefaultencoding('utf8')  

  tf.app.run()

I don't know if this can apply to your issue, hope it helps.

All 18 comments

@cshallue, could you please take a look? Looks like it is not unicode compatible.

Does the following work?

return tf.train.Feature(bytes_list=tf.train.BytesList(value=value.encode("utf-8")))

After modifying to use encode function it gives following UnicodeDecode error:

Exception in thread Thread-20:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(self.__args, *self.__kwargs)
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 282, in _process_image_files
sequence_example = _to_sequence_example(image, decoder, vocab)
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 228, in _to_sequence_example
"image/data": _bytes_feature(encoded_image),
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 191, in _bytes_feature
return tf.train.Feature(bytes_list=tf.train.BytesList(value=value.encode("utf-8")))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)

Hello, friends: I want to use Chinese caption and occured the same problem.
Have you solved this problem?
Thanks a lot.

Do any of the following work?

return tf.train.Feature(bytes_list=tf.train.BytesList(value=value))
return tf.train.Feature(bytes_list=tf.train.BytesList(value=value.decode("utf-8")))
return tf.train.Feature(bytes_list=tf.train.BytesList(value=value.decode("utf-8").encode("utf-8")))

Tried all three options, Error log of each case is as follows:

return tf.train.Feature(bytes_list=tf.train.BytesList(value=value))
Exception in thread Thread-20:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(self.__args, *self.__kwargs)
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 283, in _process_image_files
sequence_example = _to_sequence_example(image, decoder, vocab)
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 236, in _to_sequence_example
"image/caption": _bytes_feature_list(caption),
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 204, in _bytes_feature_list
return tf.train.FeatureList(feature=[_bytes_feature(v) for v in values])
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 191, in _bytes_feature
return tf.train.Feature(bytes_list=tf.train.BytesList(value=value))
TypeError: u'\u090f' has type unicode, but expected one of: bytes

return tf.train.Feature(bytes_list=tf.train.BytesList(value=value.decode("utf-8")))
Exception in thread Thread-20:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(self.__args, *self.__kwargs)
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 283, in _process_image_files
sequence_example = _to_sequence_example(image, decoder, vocab)
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 229, in _to_sequence_example
"image/data": _bytes_feature(encoded_image),
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 191, in _bytes_feature
return tf.train.Feature(bytes_list=tf.train.BytesList(value=value.decode("utf-8")))
File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xff in position 0: invalid start byte

return tf.train.Feature(bytes_list=tf.train.BytesList(value=value.decode("utf-8").encode("utf-8")))

Exception in thread Thread-20:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(self.__args, *self.__kwargs)
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 283, in _process_image_files
sequence_example = _to_sequence_example(image, decoder, vocab)
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 229, in _to_sequence_example
"image/data": _bytes_feature(encoded_image),
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 191, in _bytes_feature
return tf.train.Feature(bytes_list=tf.train.BytesList(value=value.decode("utf-8").encode("utf-8")))
File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xff in position 0: invalid start byte

Are your images jpg format? The function bytes_feature(value): converts a jpg image to string. If your images contain png, which may be utf-16, it will throw errors.
UnicodeDecodeError: 'utf8' codec can't decode byte 0xff in position 0: invalid start byte
The 0xff is out of range utf-8.

try

def _bytes_feature_list(values):
  """Wrapper for inserting a bytes FeatureList into a SequenceExample proto."""
  return tf.train.FeatureList(feature=[_bytes_feature(v.encode('utf-8')) for v in values])

@DaveyTao Images are same as MSCOCO dataset, I have changed only caption language. For English it is working fine.

Solution from @sboschang has solved error while preprocessing dataset but it introduced new errors in training phase.

[root@tensorpc im2txt]# bazel-bin/im2txt/train \

--input_file_pattern="${MSCOCO_DIR}/train-?????-of-00256" \
--inception_checkpoint_file="${INCEPTION_CHECKPOINT}" \
--train_dir="${MODEL_DIR}/train" \
--train_inception=false \
--number_of_steps=10
INFO:tensorflow:Creating training directory: /root/mshindi/train
INFO:tensorflow:Prefetching values from 7 files matching /root/Downloads/mshindi/done/train-?????-of-00256
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
INFO:tensorflow:Restoring Inception variables from checkpoint file /root/Downloads/inception_v3/inception_v3.ckpt
INFO:tensorflow:Starting Session.
INFO:tensorflow:Starting Queues.
INFO:tensorflow:global_step/sec: 0
W tensorflow/core/framework/op_kernel.cc:993] Invalid argument: Name: , Key image/data, Index: 0. Number of bytes values != expected. Values size: 167814 but output shape: []
W tensorflow/core/framework/op_kernel.cc:993] Invalid argument: Name: , Key image/data, Index: 0. Number of bytes values != expected. Values size: 194223 but output shape: []
W tensorflow/core/framework/op_kernel.cc:993] Invalid argument: Name: , Key image/data, Index: 0. Number of bytes values != expected. Values size: 150862 but output shape: []
W tensorflow/core/framework/op_kernel.cc:993] Invalid argument: Name: , Key image/data, Index: 0. Number of bytes values != expected. Values size: 243976 but output shape: []
W tensorflow/core/framework/op_kernel.cc:993] Invalid argument: Name: , Key image/data, Index: 0. Number of bytes values != expected. Values size: 227163 but output shape: []
INFO:tensorflow:Error reported to Coordinator: , Name: , Key image/data, Index: 0. Number of bytes values != expected. Values size: 167814 but output shape: []
[[Node: ParseSingleSequenceExample/ParseSingleSequenceExample = ParseSingleSequenceExample[Ncontext_dense=1, Ncontext_sparse=0, Nfeature_list_dense=1, Nfeature_list_sparse=0, Tcontext_dense=[DT_STRING], context_dense_shapes=[[]], context_sparse_types=[], feature_list_dense_shapes=[[]], feature_list_dense_types=[DT_INT64], feature_list_sparse_types=[], _device="/job:localhost/replica:0/task:0/cpu:0"](random_input_queue_Dequeue, ParseSingleSequenceExample/ParseSingleSequenceExample/feature_list_dense_missing_assumed_empty, ParseSingleSequenceExample/ParseSingleSequenceExample/context_dense_keys_0, ParseSingleSequenceExample/ParseSingleSequenceExample/feature_list_dense_keys_0, ParseSingleSequenceExample/Const, ParseSingleSequenceExample/ParseSingleSequenceExample/debug_name)]]

Caused by op u'ParseSingleSequenceExample/ParseSingleSequenceExample', defined at:
File "/root/models/im2txt/bazel-bin/im2txt/train.runfiles/im2txt/im2txt/train.py", line 114, in
tf.app.run()
File "/usr/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 44, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "/root/models/im2txt/bazel-bin/im2txt/train.runfiles/im2txt/im2txt/train.py", line 65, in main
model.build()
File "/root/models/im2txt/bazel-bin/im2txt/train.runfiles/im2txt/im2txt/show_and_tell_model.py", line 353, in build
self.build_inputs()
File "/root/models/im2txt/bazel-bin/im2txt/train.runfiles/im2txt/im2txt/show_and_tell_model.py", line 164, in build_inputs
caption_feature=self.config.caption_feature_name)
File "/root/models/im2txt/bazel-bin/im2txt/train.runfiles/im2txt/im2txt/ops/inputs.py", line 46, in parse_sequence_example
caption_feature: tf.FixedLenSequenceFeature([], dtype=tf.int64),
File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/parsing_ops.py", line 780, in parse_single_sequence_example
feature_list_dense_defaults, example_name, name)
File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/parsing_ops.py", line 977, in _parse_single_sequence_example_raw
name=name)
File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/gen_parsing_ops.py", line 287, in _parse_single_sequence_example
name=name)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 763, in apply_op
op_def=op_def)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2395, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1264, in __init__
self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Name: , Key image/data, Index: 0. Number of bytes values != expected. Values size: 167814 but output shape: []
[[Node: ParseSingleSequenceExample/ParseSingleSequenceExample = ParseSingleSequenceExample[Ncontext_dense=1, Ncontext_sparse=0, Nfeature_list_dense=1, Nfeature_list_sparse=0, Tcontext_dense=[DT_STRING], context_dense_shapes=[[]], context_sparse_types=[], feature_list_dense_shapes=[[]], feature_list_dense_types=[DT_INT64], feature_list_sparse_types=[], _device="/job:localhost/replica:0/task:0/cpu:0"](random_input_queue_Dequeue, ParseSingleSequenceExample/ParseSingleSequenceExample/feature_list_dense_missing_assumed_empty, ParseSingleSequenceExample/ParseSingleSequenceExample/context_dense_keys_0, ParseSingleSequenceExample/ParseSingleSequenceExample/feature_list_dense_keys_0, ParseSingleSequenceExample/Const, ParseSingleSequenceExample/ParseSingleSequenceExample/debug_name)]]

Traceback (most recent call last):
File "/root/models/im2txt/bazel-bin/im2txt/train.runfiles/im2txt/im2txt/train.py", line 114, in
tf.app.run()
File "/usr/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 44, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "/root/models/im2txt/bazel-bin/im2txt/train.runfiles/im2txt/im2txt/train.py", line 110, in main
saver=saver)
File "/usr/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/learning.py", line 804, in train
raise
File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 973, in managed_session
self.stop(close_summary_writer=close_summary_writer)
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 801, in stop
stop_grace_period_secs=self._stop_grace_secs)
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py", line 386, in join
six.reraise(*self._exc_info_to_raise)
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/queue_runner_impl.py", line 234, in _run
sess.run(enqueue_op)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 767, in run
run_metadata_ptr)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 965, in _run
feed_dict_string, options, run_metadata)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1015, in _do_run
target_list, options, run_metadata)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1035, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Name: , Key image/data, Index: 0. Number of bytes values != expected. Values size: 167814 but output shape: []
[[Node: ParseSingleSequenceExample/ParseSingleSequenceExample = ParseSingleSequenceExample[Ncontext_dense=1, Ncontext_sparse=0, Nfeature_list_dense=1, Nfeature_list_sparse=0, Tcontext_dense=[DT_STRING], context_dense_shapes=[[]], context_sparse_types=[], feature_list_dense_shapes=[[]], feature_list_dense_types=[DT_INT64], feature_list_sparse_types=[], _device="/job:localhost/replica:0/task:0/cpu:0"](random_input_queue_Dequeue, ParseSingleSequenceExample/ParseSingleSequenceExample/feature_list_dense_missing_assumed_empty, ParseSingleSequenceExample/ParseSingleSequenceExample/context_dense_keys_0, ParseSingleSequenceExample/ParseSingleSequenceExample/feature_list_dense_keys_0, ParseSingleSequenceExample/Const, ParseSingleSequenceExample/ParseSingleSequenceExample/debug_name)]]

Caused by op u'ParseSingleSequenceExample/ParseSingleSequenceExample', defined at:
File "/root/models/im2txt/bazel-bin/im2txt/train.runfiles/im2txt/im2txt/train.py", line 114, in
tf.app.run()
File "/usr/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 44, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "/root/models/im2txt/bazel-bin/im2txt/train.runfiles/im2txt/im2txt/train.py", line 65, in main
model.build()
File "/root/models/im2txt/bazel-bin/im2txt/train.runfiles/im2txt/im2txt/show_and_tell_model.py", line 353, in build
self.build_inputs()
File "/root/models/im2txt/bazel-bin/im2txt/train.runfiles/im2txt/im2txt/show_and_tell_model.py", line 164, in build_inputs
caption_feature=self.config.caption_feature_name)
File "/root/models/im2txt/bazel-bin/im2txt/train.runfiles/im2txt/im2txt/ops/inputs.py", line 46, in parse_sequence_example
caption_feature: tf.FixedLenSequenceFeature([], dtype=tf.int64),
File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/parsing_ops.py", line 780, in parse_single_sequence_example
feature_list_dense_defaults, example_name, name)
File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/parsing_ops.py", line 977, in _parse_single_sequence_example_raw
name=name)
File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/gen_parsing_ops.py", line 287, in _parse_single_sequence_example
name=name)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 763, in apply_op
op_def=op_def)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2395, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1264, in __init__
self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Name: , Key image/data, Index: 0. Number of bytes values != expected. Values size: 167814 but output shape: []
[[Node: ParseSingleSequenceExample/ParseSingleSequenceExample = ParseSingleSequenceExample[Ncontext_dense=1, Ncontext_sparse=0, Nfeature_list_dense=1, Nfeature_list_sparse=0, Tcontext_dense=[DT_STRING], context_dense_shapes=[[]], context_sparse_types=[], feature_list_dense_shapes=[[]], feature_list_dense_types=[DT_INT64], feature_list_sparse_types=[], _device="/job:localhost/replica:0/task:0/cpu:0"](random_input_queue_Dequeue, ParseSingleSequenceExample/ParseSingleSequenceExample/feature_list_dense_missing_assumed_empty, ParseSingleSequenceExample/ParseSingleSequenceExample/context_dense_keys_0, ParseSingleSequenceExample/ParseSingleSequenceExample/feature_list_dense_keys_0, ParseSingleSequenceExample/Const, ParseSingleSequenceExample/ParseSingleSequenceExample/debug_name)]]

@shahparth123 looks like you've converted the image into a sequence of characters, which is why you get that error.

If the issue is the hindi text, a quick fix is to comment out this line, and not write the text in the proto buffers at all. The model only uses the int64 caption ids, so it won't notice.

I've had the same issue because of accented characters (language: Italian) and what worked out for me was resetting the default encoding before tf.app.run():

if __name__ == "__main__":

  reload(sys)
  sys.setdefaultencoding('utf8')  

  tf.app.run()

I don't know if this can apply to your issue, hope it helps.

I am getting same error even after removing line as sugested by @cshallue . I am setting up new project from using latest version of both tensorflow and mode.l Will report back after applying @cshallue and @CaterinaM solution.

@DaveyTao I am trying to generate Chinese capitions as you are.
Do you have a success, or can you teach me?

@shahparth123 Have you solved the coding problem (UTF-8)?
Thank you!

@CaterinaM Your solution worked for me as well, I am working on Hindi caption generation. Thanks a lot.

@j2233ack Yes model is working. I am open sourcing my model soon.

@amankhandelia I'm glad it helped, you're welcome!

Was this page helpful?
0 / 5 - 0 ratings