Models: WARNING:root:Invalid example: with create_pet_tf_record.py ???

Created on 8 Jun 2018  Â·  11Comments  Â·  Source: tensorflow/models

System information

What is the top-level directory of the model you are using: /home/abdu/py3-gpu/models/research/object_detection
Have I written custom code (NO):
OS Platform and Distribution (Linux Ubuntu 16.04):
TensorFlow installed from (binary[using pip3):
TensorFlow version (1.6.0):
Python version: (3.5.2)
Bazel version (N/A):
GCC/Compiler version (N/A):
CUDA/cuDNN version: CUDA 9.0/ CUDDN 7
GPU model and memory: Nvidia GTX 1080/ 8GB
Exact command to reproduce:

(py3-gpu) abdu@grasp-uoit:~/py3-gpu/models/research$ python object_detection/dataset_tools/create_pet_tf_record.py --label_map_path=object_detection/data/pet_label_map.pbtxt --data_dir=pwd--output_dir=/home/abdu/py3-gpu/models/research/ttt/

I am trying to generate TFRecords files using create_pet_tf_record.py based on this instructions HERE I also made 'faces_only', False,

I get 20 TFRecords files 10 for train and 10 for eval

pets_fullbody_with_masks_train.record-00000-of-00010
.
.
pets_fullbody_with_masks_train.record-00009-of-00010

and

pets_fullbody_with_masks_val.record-00000-of-00010
.
.
pets_fullbody_with_masks_val.record-00009-of-00010

How do I suppose to refer TFRecord files in .config during training if I have 20 files, I was expected to have only two (something like pet_train.record and pet_eval.record .???

The output with ignoring many examples:-

(py3-gpu) abdu@grasp-uoit:~/py3-gpu/models/research$ python object_detection/dataset_tools/create_pet_tf_record.py     --label_map_path=object_detection/data/pet_label_map.pbtxt     --data_dir=`pwd`     --output_dir=/home/abdu/py3-gpu/models/research/ttt/
/home/abdu/py3-gpu/models/research/object_detection/utils/dataset_util.py:75: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead.
  if not xml:
WARNING:root:Invalid example: /home/abdu/py3-gpu/models/research/annotations/xmls/miniature_pinscher_14.xml, ignoring.
WARNING:root:Invalid example: /home/abdu/py3-gpu/models/research/annotations/xmls/leonberger_18.xml, ignoring.
WARNING:root:Could not find /home/abdu/py3-gpu/models/research/annotations/xmls/Egyptian_Mau_14.xml, ignoring example.
WARNING:root:Could not find /home/abdu/py3-gpu/models/research/annotations/xmls/saint_bernard_15.xml, ignoring example.
WARNING:root:Could not find /home/abdu/py3-gpu/models/research/annotations/xmls/Ragdoll_199.xml, ignoring example.
WARNING:root:Invalid example: /home/abdu/py3-gpu/models/research/annotations/xmls/Egyptian_Mau_162.xml, ignoring.
WARNING:root:Could not find /home/abdu/py3-gpu/models/research/annotations/xmls/Egyptian_Mau_186.xml, ignoring example.
WARNING:root:Could not find /home/abdu/py3-gpu/models/research/annotations/xmls/Bengal_175.xml, ignoring example.
WARNING:root:Could not find /home/abdu/py3-gpu/models/research/annotations/xmls/samoyed_10.xml, ignoring example.
WARNING:root:Could not find /home/abdu/py3-gpu/models/research/annotations/xmls/Egyptian_Mau_156.xml, ignoring example.
WARNING:root:Could not find /home/abdu/py3-gpu/models/research/annotations/xmls/Abyssinian_104.xml, ignoring example.
WARNING:root:Invalid example: /home/abdu/py3-gpu/models/research/annotations/xmls/saint_bernard_108.xml, ignoring.
WARNING:root:Invalid example: /home/abdu/py3-gpu/models/research/annotations/xmls/Egyptian_Mau_196.xml, ignoring.
WARNING:root:Invalid example: /home/abdu/py3-gpu/models/research/annotations/xmls/Egyptian_Mau_165.xml, ignoring.
WARNING:root:Could not find /home/abdu/py3-gpu/models/research/annotations/xmls/Bengal_111.xml, ignoring example.
awaiting maintainer
Source
picture Abduoit

Most helpful comment

Also having this problem.
Edit, I think I figured out a way around my issue with some edits to create_pet_tf_record.py. In my case, dict_to_tf_example() from create_tf_record() was throwing a ValueError that got caught by the 

except ValueError:
  logging.warning('Invalid example: %s, ignoring.', xml_path)

I found it was this error originally

if image.format != 'JPEG':
  raise ValueError('Image format not JPEG')

I'm using png files, so instead of reformatting I changed this block

  with tf.gfile.GFile(img_path, 'rb') as fid:
    encoded_jpg = fid.read()
  encoded_jpg_io = io.BytesIO(encoded_jpg)
  image = PIL.Image.open(encoded_jpg_io)
  if image.format != 'JPEG':
    raise ValueError('Image format not JPEG')
  key = hashlib.sha256(encoded_jpg).hexdigest()

to this

  with tf.gfile.GFile(img_path, 'rb') as fid:
    encoded_png = fid.read()
  encoded_png_io = io.BytesIO(encoded_png)
  image = PIL.Image.open(encoded_png_io)
  if image.format != 'PNG':
    raise ValueError('Image format not PNG')
  key = hashlib.sha256(encoded_png).hexdigest()

and I had to replace this line

      'image/encoded': dataset_util.bytes_feature(encoded_jpg),

with this

      'image/encoded': dataset_util.bytes_feature(encoded_png),

I also had to change the regular expression in get_class_name_from_filename() from 

match = re.match(r'([A-Za-z_]+)(_[0-9]+\.jpg)', file_name, re.I)

to

match = re.match(r'([A-Za-z_]+)(_[0-9]+\.png)', file_name, re.I)

I also had a problem with label_map_dict not getting created properly so I hardcoded the dictionary in my code.

picture mcsimenc on 3 Jan 2019
👍3 😄2

All 11 comments

Adding an author who can help.

bitfort picture bitfort on 11 Jun 2018

the same issue

aqzwss picture aqzwss on 12 Jun 2018

@aqzwss
have you ever succeed training your dataset with mask model ?

Abduoit picture Abduoit on 12 Jun 2018

you can run python object_detection\dataset_toolscreate_pet_tf_record.py --num_shards=1

luzhengfei picture luzhengfei on 30 Jun 2018

in config file you can do

input_path: "pwd_train.record-?????-of-00010"

but i don't know what is the difference between numshard 1 or more

i don't manage to display mask on eval or detection object what is missing ? i configured load_mask, set PNG, and faces_only = False

i become crazy no one respond... is someone manage to make pet dataset mask segmentation work ?

leccyril picture leccyril on 13 Jul 2018

anyone solved this? it seems like an error in parsing the xml files of annotations, but its all fine and can't seem to find what's wrong with it exactly.

D:\Anaconda\lib\site-packages\h5py__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
data\dataset\annotations
D:\Anaconda\lib\site-packagesobject_detection-0.1-py3.6.eggobject_detection\utils\dataset_util.py:75: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead.
if not xml:
WARNING:root:Invalid example: data\dataset\annotations\xmlsobject33.xml, ignoring.
WARNING:root:Invalid example: data\dataset\annotations\xmlsobject30.xml, ignoring.
WARNING:root:Invalid example: data\dataset\annotations\xmlsobject66.xml, ignoring.
WARNING:root:Invalid example: data\dataset\annotations\xmlsobject70.xml, ignoring.
WARNING:root:Invalid example: data\dataset\annotations\xmlsobject22.xml, ignoring.
WARNING:root:Invalid example: data\dataset\annotations\xmlsobject49.xml, ignoring.
WARNING:root:Invalid example: data\dataset\annotations\xmlsobject79.xml, ignoring.
WARNING:root:Invalid example: data\dataset\annotations\xmlsobject37.xml, ignoring.
WARNING:root:Invalid example: data\dataset\annotations\xmlsobject34.xml, ignoring.
WARNING:root:Invalid example: data\dataset\annotations\xmlsobject40.xml, ignoring.
WARNING:root:Invalid example: data\dataset\annotations\xmlsobject81.xml, ignoring.
WARNING:root:Invalid example: data\dataset\annotations\xmlsobject89.xml, ignoring.
WARNING:root:Invalid example: data\dataset\annotations\xmlsobject8.xml, ignoring.
WARNING:root:Invalid example: data\dataset\annotations\xmlsobject39.xml, ignoring.

GShef picture GShef on 29 Nov 2018

Also having this problem.
Edit, I think I figured out a way around my issue with some edits to create_pet_tf_record.py. In my case, dict_to_tf_example() from create_tf_record() was throwing a ValueError that got caught by the 

except ValueError:
  logging.warning('Invalid example: %s, ignoring.', xml_path)

I found it was this error originally

if image.format != 'JPEG':
  raise ValueError('Image format not JPEG')

I'm using png files, so instead of reformatting I changed this block

  with tf.gfile.GFile(img_path, 'rb') as fid:
    encoded_jpg = fid.read()
  encoded_jpg_io = io.BytesIO(encoded_jpg)
  image = PIL.Image.open(encoded_jpg_io)
  if image.format != 'JPEG':
    raise ValueError('Image format not JPEG')
  key = hashlib.sha256(encoded_jpg).hexdigest()

to this

  with tf.gfile.GFile(img_path, 'rb') as fid:
    encoded_png = fid.read()
  encoded_png_io = io.BytesIO(encoded_png)
  image = PIL.Image.open(encoded_png_io)
  if image.format != 'PNG':
    raise ValueError('Image format not PNG')
  key = hashlib.sha256(encoded_png).hexdigest()

and I had to replace this line

      'image/encoded': dataset_util.bytes_feature(encoded_jpg),

with this

      'image/encoded': dataset_util.bytes_feature(encoded_png),

I also had to change the regular expression in get_class_name_from_filename() from 

match = re.match(r'([A-Za-z_]+)(_[0-9]+\.jpg)', file_name, re.I)

to

match = re.match(r'([A-Za-z_]+)(_[0-9]+\.png)', file_name, re.I)

I also had a problem with label_map_dict not getting created properly so I hardcoded the dictionary in my code.

mcsimenc picture mcsimenc on 3 Jan 2019
👍3 😄2

@mcsimenc Would you please elaborate how you hardcoded the dictionary to redress the issues in label_map_dict?

jisooyu picture jisooyu on 4 Nov 2019

@jisooyu the second line under def main(_): is label_map_dict = label_map_util.get_label_map_dict(FLAGS.label_map_path). So it gets label_map_dict from FLAGS.label_map_path which is shown at the top of the script in the flags.DEFINE_string line as data/pet_label_map.pbtxt. So I think what I did was change the line to the contents of my label_map.pbtxt, which was actually just one label, to label_map_dict = {fish:1} but for the pets_label_map.pbtxt there are a bunch of cat and dog breeds so the dict would be longer. I don't have that code handy but I think that's what I did.

mcsimenc picture mcsimenc on 4 Nov 2019

Thank you for your kind help. I didn’t expect your reply since the issue was discussed a year ago but you kindly responded. Thank you very much. I will try what you advised.

On 5 Nov 2019, at 3:34 AM, mcsimenc <[email protected] notifications@github.com> wrote:

@jisooyu https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjisooyu&data=02%7C01%7C%7Caad51984fa954e28c94c08d76155aaf0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637084892877771609&sdata=2ammahDvyBK1o4cnSEiXq%2BMOIx%2FRs62Z8QhIsgoWhx4%3D&reserved=0 the second line under def main(_): is label_map_dict = label_map_util.get_label_map_dict(FLAGS.label_map_path). So it gets label_map_dict from FLAGS.label_map_path which is shown at the top of the script in the flags.DEFINE_string line as data/pet_label_map.pbtxt. So I think what I did was change the line to the contents of my label_map.pbtxt, which was actually just one label label_map_dict = {fish:1} but for the pets_label_map.pbtxt there are a bunch of cat and dog breeds so the dict would be longer. I don't have that code handy but I think that's what I did.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftensorflow%2Fmodels%2Fissues%2F4490%3Femail_source%3Dnotifications%26email_token%3DAF6IHVMHRQRJKJ4LJTRJFHLQSBTMLA5CNFSM4FECQTH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDAID7A%23issuecomment-549487100&data=02%7C01%7C%7Caad51984fa954e28c94c08d76155aaf0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637084892877781620&sdata=21L0A%2Bhiztvk%2B%2B7rV2gkUyC3GN1fti0z2B%2F04S5QySU%3D&reserved=0, or unsubscribe https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAF6IHVJBZ44N3V7M7YSJQH3QSBTMLANCNFSM4FECQTHQ&data=02%7C01%7C%7Caad51984fa954e28c94c08d76155aaf0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637084892877792682&sdata=2M612JO1FMzmP%2BWikwLHD39lL%2FDLYZjjFVxkt0HAuDU%3D&reserved=0.

jisooyu picture jisooyu on 4 Nov 2019

@mcsimenc According to your advice, here are the changes made.
FROM
flags.DEFINE_string('label_map_path', 'data/pet_label_map.pbtxt',
'Path to label map proto')
TO
flags.DEFINE_string('label_map_path', 'data/landing_label_map.pbtxt',
'Path to label map proto')

The terminal starts show the following. So it seems to work. Many thanks to @mcsimenc !!!
Accumulating evaluation results...
DONE (t=0.00s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000

jisooyu picture jisooyu on 5 Nov 2019
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

hanzy123 picture hanzy123  Â·  3Comments
amirjamez picture amirjamez  Â·  3Comments
noumanriazkhan picture noumanriazkhan  Â·  3Comments
airmak picture airmak  Â·  3Comments
dsindex picture dsindex  Â·  3Comments