Hub: [TF2.0] MobileNet v2 feature_vector/3 has worse results than feature_vector/2 in transfer learning

[frogermcs]

In simple example, e.g. flower_photos dataset, and simple network presented below:

tf.keras.Sequential([
    hub.KerasLayer("https://tfhub.dev/google/tf2-preview/mobilenet_v2/feature_vector/3", output_shape=[1280], trainable=False),
  layers.Dense(train_generator.num_classes, activation='softmax')
])

feature_vector/3 has worse results than feature_vector/2 after 5 epochs of transfer learning process (about 10% smaller in val_accuracy). Difference gets bigger in more complex examples.

Env:

Version:  2.0.0-dev20190519
Eager mode:  True
GPU is available

Here are two notebooks to compare:

feature_vector/3:
https://colab.research.google.com/drive/1j4j0f8xWcjcrm0y3HdY3Z9m31XeiB-l-

feature_vector/2:
https://colab.research.google.com/drive/15Pwmy9oUGeEal_ZjRa3rjKBz6-UjhTzv

The only difference between them is:

feature_extractor_url = "https://tfhub.dev/google/tf2-preview/mobilenet_v2/feature_vector/2" #@param {type:"string"}

and

feature_extractor_url = "https://tfhub.dev/google/tf2-preview/mobilenet_v2/feature_vector/3" #@param {type:"string"}

And results are:

Epoch 1/5
WARNING: Logging before flag parsing goes to stderr.
W0519 17:10:06.324720 140516547499904 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
92/92 [==============================] - 20s 213ms/step - loss: 0.9439 - acc: 0.7500 - val_loss: 0.9088 - val_acc: 0.6799
Epoch 2/5
92/92 [==============================] - 15s 160ms/step - loss: 0.5434 - acc: 0.8438 - val_loss: 0.8865 - val_acc: 0.6799
Epoch 3/5
92/92 [==============================] - 16s 171ms/step - loss: 0.4805 - acc: 0.8750 - val_loss: 0.7272 - val_acc: 0.7592
Epoch 4/5
92/92 [==============================] - 15s 160ms/step - loss: 0.4305 - acc: 0.9062 - val_loss: 0.7342 - val_acc: 0.7606
Epoch 5/5
92/92 [==============================] - 14s 157ms/step - loss: 0.3854 - acc: 0.9062 - val_loss: 0.8176 - val_acc: 0.7456

Epoch 1/5
WARNING: Logging before flag parsing goes to stderr.
W0519 17:13:32.105277 140054425941888 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
92/92 [==============================] - 22s 236ms/step - loss: 0.8856 - acc: 0.7812 - val_loss: 0.5994 - val_acc: 0.8044
Epoch 2/5
92/92 [==============================] - 15s 159ms/step - loss: 0.5334 - acc: 0.9062 - val_loss: 0.4915 - val_acc: 0.8564
Epoch 3/5
92/92 [==============================] - 14s 157ms/step - loss: 0.4590 - acc: 0.9375 - val_loss: 0.4827 - val_acc: 0.8591
Epoch 4/5
92/92 [==============================] - 15s 159ms/step - loss: 0.4108 - acc: 0.9062 - val_loss: 0.4699 - val_acc: 0.8755
Epoch 5/5
92/92 [==============================] - 16s 179ms/step - loss: 0.3935 - acc: 0.9375 - val_loss: 0.4630 - val_acc: 0.8673

[arnoegw]

@frogermcs, thank you for your report. I can confirm the quality problem. It appears there is a problem with the update ops of batch normalization.

[arnoegw]

There are two intertwined issues here:

1. https://tfhub.dev/google/tf2-preview/mobilenet_v2/feature_vector/3 does not run the update ops of batch norm (because its export from a tf.compat.v1.wrap_function() was damaged by a bug in commit https://github.com/tensorflow/tensorflow/commit/b14c390fc869b63fd2b1a4e6f4477ce410b9383e). The same issue exists for version /3 of all tf2-preview image models. The TensorFlow team is pursuing a fix for that, and I plan to publish module versions /4 (with that fix or a workaround) to alleviate this.

2. hub.KerasLayer has a bug (now fixed on the master branch by commit https://github.com/tensorflow/hub/commit/9fa99aa7233961661fe65721f7608ffcecce2045) that would make it call the SavedModel from hub with training=True when the surrounding model is training, even if the hub.KerasLayer has trainable=False. That is wrong; a Hub module in a non-trainable layer should be unchanging over time and unchanging between training and inference.

What you saw in your examples with non-trainable hub.KerasLayers and before the fix of (2) was this, respectively:

• For module version /3, the imported Hub module did batch normalization with batch statistics during training (due to problem 2), but did not update the aggregate statistics for use at inference time (due to problem 1). That means the Hub module created very different outputs for the same images during training and inference, damaging performance.

• For module version /2, the imported Hub module did batch normalization with batch statistics during training (again due to problem 2) but also updated aggregate statistics (as batch norm does in training mode, absent problem 1). This provided ok'ish performance, albeit in a very unintended way (that reminds me https://arxiv.org/abs/1603.04779).

I'll provide more updates as the solutions become available for the demo colab. Thank you very much again for this important bug report.

[arnoegw]

I've released new module versions https://tfhub.dev/google/tf2-preview/mobilenet_v2/feature_vector/4 etc., fixing problem (1) described above. (The underlying problem during module creation had been fixed by commit https://github.com/tensorflow/tensorflow/commit/5d75fc280e79636736388cc5a9356b34f60f12d9; thank you @omalleyt12.)

Using the new module version in a hub.KerasLayer with trainable=True should work as expected and provide full quality.

I'll provide further updates on the remaining aspects.

[arnoegw]

I've updated tf2_image_retraining to use the tf-hub-nightly PIP package, which fixes problem (2) from my summary above. A regular tensorflow-hub 0.5.0 release will follow in due course.

With this, tf2_image_retraining should now work as intended both with and without fine-tuning enabled.

[siavash-khodadadeh]

Does this have anything to do with this?

https://stackoverflow.com/questions/56481256/why-mini-batch-output-to-mobilenet-is-different-from-single-output

[arnoegw]

@siavash-khodadadeh: I don't think so. The code in that question uses module version /4 and only ever runs the model in prediction mode, so the issues described above should not kick in. I posted a comment on stackoverflow with my thoughts on the question; let's continue there.