Handson-ml: Class names off by one for exercise 8 in CNN notebook.

[WillKoehrsen]

Thanks for the great book Aurelien!

There is a slight issue with exercise 8 in the CNN notebook. When you create the class names for the inception model, you use the textfile imagenet_class_names.txt. This file contains 1000 class names.

def load_class_names():
    with open(os.path.join("datasets", "inception", "imagenet_class_names.txt"), "rb") as f:
        content = f.read().decode("utf-8")
        return CLASS_NAME_REGEX.findall(content)

class_names = load_class_names()
print('There are {} classes.'.format(len(class_names)))

There are 1000 classes.

However, when you run the inception v3 model and pass in the argument num_classes=1001, the predictions returned are for 1001 classes.

X_test = test_image.reshape(-1, height, width, channels)

with tf.Session() as sess:
    saver.restore(sess, INCEPTION_V3_CHECKPOINT_PATH)
    predictions_val = predictions.eval(feed_dict={X: X_test})

print('There are predictions for {} classes.'.format(len(predictions_val[0])))

There are predictions for 1001 classes.

It turns out that the inception function makes a prediction for a background class in addition to the 1000 classes in Imagenet. This can be seen in the program imagenet.py in the Tensorflow slim Git Hub. This program contains a function create_readable_names_for_imagenet_labels() that returns all the Imagenet labels with the background class inserted as the first index.

def create_readable_names_for_imagenet_labels():
  """Create a dict mapping label id to human readable string.
  Returns:
      labels_to_names: dictionary where keys are integers from to 1000
      and values are human-readable names.
  We retrieve a synset file, which contains a list of valid synset labels used
  by ILSVRC competition. There is one synset one per line, eg.
          #   n01440764
          #   n01443537
  We also retrieve a synset_to_human_file, which contains a mapping from synsets
  to human-readable names for every synset in Imagenet. These are stored in a
  tsv format, as follows:
          #   n02119247    black fox
          #   n02119359    silver fox
  We assign each synset (in alphabetical order) an integer, starting from 1
  (since 0 is reserved for the background class).
  Code is based on
  https://github.com/tensorflow/models/blob/master/inception/inception/data/build_imagenet_data.py#L463
  """

  # pylint: disable=g-line-too-long
  base_url = 'https://raw.githubusercontent.com/tensorflow/models/master/inception/inception/data/'
  synset_url = '{}/imagenet_lsvrc_2015_synsets.txt'.format(base_url)
  synset_to_human_url = '{}/imagenet_metadata.txt'.format(base_url)

  filename, _ = urllib.request.urlretrieve(synset_url)
  synset_list = [s.strip() for s in open(filename).readlines()]
  num_synsets_in_ilsvrc = len(synset_list)
  assert num_synsets_in_ilsvrc == 1000

  filename, _ = urllib.request.urlretrieve(synset_to_human_url)
  synset_to_human_list = open(filename).readlines()
  num_synsets_in_all_imagenet = len(synset_to_human_list)
  assert num_synsets_in_all_imagenet == 21842

  synset_to_human = {}
  for s in synset_to_human_list:
    parts = s.strip().split('\t')
    assert len(parts) == 2
    synset = parts[0]
    human = parts[1]
    synset_to_human[synset] = human

  label_index = 1
  labels_to_names = {0: 'background'}
  for synset in synset_list:
    name = synset_to_human[synset]
    labels_to_names[label_index] = name
    label_index += 1

  return labels_to_names

If we use this function for the class names, the class labels are all shifted forward by one:
current code:

class_names[:5]

['tench, Tinca tinca',
 'goldfish, Carassius auratus',
 'great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias',
 'tiger shark, Galeocerdo cuvieri',
 'hammerhead, hammerhead shark']

Imagenet class labels:

from imagenet import create_readable_names_for_imagenet_labels

imagenet_class_names = create_readable_names_for_imagenet_labels()
imagenet_class_names = list(imagenet_class_names.values())

imagenet_class_names[:5]

['background',
 'tench, Tinca tinca',
 'goldfish, Carassius auratus',
 'great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias',
 'tiger shark, Galeocerdo cuvieri']

In the example you show, you get the right answer regardless because the picture you choose (of a hyena) has two adjacent labels in the class names.

imagenet_class_names[276: 278]

['African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus',
 'hyena, hyaena']

This can be fixed by subtracting one from the index of class labels when displaying the predictions.
You can see the difference when you print out the top five predictions.

current code:

top_5 = np.argpartition(predictions_val[0], -5)[-5:]
top_5 = top_5[np.argsort(predictions_val[0][top_5])]
for i in top_5:
    print("{0}: {1:.2f}%".format(class_names[i], 100 * predictions_val[0][i]))

swing: 0.04%
beer bottle: 0.05%
common newt, Triturus vulgaris: 0.05%
red fox, Vulpes vulpes: 2.36%
hyena, hyaena: 93.85%

If you subtract one from the index to account for the background class label:

for i in top_5:
    print("{0}: {1:.2f}%".format(class_names[i - 1], 100 * predictions_val[0][i]))

swimming trunks, bathing trunks: 0.04%
bearskin, busby, shako: 0.05%
European fire salamander, Salamandra salamandra: 0.05%
hyena, hyaena: 2.36%
African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus: 93.85%

The example you choose works because of the multiple labels for hyena! I tried some different images and was confused why the predictions were so strange until I noticed that the length of the predictions returned by the inception function did not match the length of the class names. Subtracting one from the index when displaying the predicted classes works as does adding the 'background' class as the 0th index of class_names

[ageron]

Wow! Great catch, and excellent detailed explanation, thanks so much @WillKoehrsen. I'm in vacation right now but I'll fix this as soon as I get back.
Thanks again!
Aurélien

[WillKoehrsen]

No problem, thanks for being so willing to update the notebooks! I realize now the easiest fix is simply to add 'background' as the first entry in imagenet_class_names.txt

Sorry for such a long explanation to a simple problem!

[ageron]

Hi,
Thanks again for identifying this issue and helping resolve it. I just pushed the fix.
Cheers!

[WillKoehrsen]

No problem, thanks for writing such a great introduction to machine
learning. I am looking forward to the second edition!

Regards,
Will

On Sep 15, 2017 3:47 PM, "Aurélien Geron" [e-mail] wrote:

Hi,
Thanks again for identifying this issue and helping resolve it. I just
pushed the fix.
Cheers!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ageron/handson-ml/issues/61#issuecomment-329880618,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AQrLSPXq0AOWec4K-9ibQrHHZZLB4zMPks5sitMWgaJpZM4ObsCO
.