Models: (deeplabv3+) (mobilenetv2) training mobilenet_v2 on deeplabv3+

[yswang1717]

1.System information

the top-level directory of the model **: /home/yswang/models3/research/deeplab
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
TensorFlow installed from (source or binary): pip
TensorFlow version (use command below): tensorflow-gpu 1.6.0
Bazel version (if compiling from source): None.
CUDA/cuDNN version: None.
GPU model and memory: Titan xp / 12GB / 8 GPU
Exact command to reproduce:
for the first 30k iterations :
NUM_ITERATIONS=30000
CUDA_VISIBLE_DEVICES=4,5,6,7 python "${WORK_DIR}"/train.py
--logtostderr
--train_split="trainaug"
--model_variant="mobilenet_v2"
--atrous_rates=6
--atrous_rates=12
--atrous_rates=18
--output_stride=16
--decoder_output_stride=4
--train_crop_size=513
--train_crop_size=513
--num_clones=4
--train_batch_size=16
--dataset="pascal_voc_seg"
--training_number_of_steps="${NUM_ITERATIONS}"
--fine_tune_batch_norm=True
--tf_initial_checkpoint="${INIT_FOLDER}/mobilenet_v2/mobilenet_v2_1.0_224.ckpt"
--train_logdir="${TRAIN_LOGDIR}/gpu_test/20"
--dataset_dir="${PASCAL_DATASET}"

Describe the problem
I trained the deeplabv3+ models with the mobilenet_v2_1.0_224 versions.
I want to achieve the mIOU=75.7% in the table 7 of mobilenetv2 paper, but I got just 70.89% for 30k iterations.

I followed the same training protocol as in deeplabv3 paper. firstly, train with following parameters(poly, initial lr=0.007, crop size = 513 x 513 , OS=16, fine-tuning batch=True, augmentation during training. ) with 4 GPU. and it performs 73.49% mIOU.

The table7 in the mobilenetv2 paper describes that 75% mIOU without decoder. However, I achieved just 70.89% with decoder ( without MS COCO pretrained models. ) Does anyone tell me how to reproduce mIOU=75.7%?

[tensorflowbutler]

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
What is the top-level directory of the model you are using
Have I written custom code

[aquariusjay]

Hi yswang0522,

You need to train the model with train_aug set, using the extra annotations provided by Bharath Hariharan et al. See Q3 in the FAQ for details.

[wangyooseung]

oh that is typo. I trained on trainaug set. do you think other parameters and settings are ok?

[aquariusjay]

As mentioned in the model_zoo, the MobileNet-v2 model variant that attains 75% mIOU has been pretrained on MSCOCO dataset. You need to pretrain on MSCOCO.

[advaza]

Hi,

1.System information

the top-level directory of the model **: models
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
TensorFlow installed from (source or binary): source.
TensorFlow version (use command below): tensorflow-gpu 1.8.
Bazel version (if compiling from source): N/A.
CUDA/cuDNN version: 9/7.
GPU model and memory: Tesla V100-SXM2-16GB
Exact command to reproduce:
python3 research/deeplab/train.py \
--train_logdir output/mobile_net \
--model_variant=mobilenet_v2 \
--tf_initial_checkpoint=checkpoints/mobilenet_v2_1.0_224/mobilenet_v2_1.0_224.ckpt \
--train_split=train_aug \
--dataset_dir=/data/aug_pascal_tfrecord/tfrecord

All the other parameters are set to default values from the code.

Describe the problem
I also trained the deeplabv3+ model on the train_aug dataset, using the initial checkpoint mobilenet_v2_1.0_224.
But I got only 27% mIOU after 30K iterations.
I built the dataset using this source for the mask images + the build_voc2012_data.py script to convert the dataset to tfrecords.

Output from the training:

/opt/ds3/lib/python3.5/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. 
In future, it will be treated as `np.float64 == np.dtype(float).type`. from ._conv import register_converters as _register_converters  
INFO:tensorflow:Training on train_aug set  

INFO:tensorflow:Initializing model from path: checkpoints/mobilenet_v2_1.0_224/mobilenet_v2_1.0_224.ckpt  
WARNING:tensorflow:Variable MobilenetV2/expanded_conv_14/depthwise/depthwise_weights/Momentum missing in checkpoint checkpoints/mobilenet_v2_1.0_224/mobilenet_v2_1.0_224.ckpt  
WARNING:tensorflow:Variable MobilenetV2/expanded_conv_8/expand/weights/Momentum missing in checkpoint checkpoints/mobilenet_v2_1.0_224/mobilenet_v2_1.0_224.ckpt  
WARNING:tensorflow:Variable MobilenetV2/expanded_conv_7/depthwise/BatchNorm/gamma/Momentum missing in checkpoint checkpoints/mobilenet_v2_1.0_224/mobilenet_v2_1.0_224.ckpt  
WARNING:tensorflow:Variable MobilenetV2/expanded_conv_9/project/BatchNorm/beta/Momentum missing in checkpoint checkpoints/mobilenet_v2_1.0_224/mobilenet_v2_1.0_224.ckpt  
...

WARNING:tensorflow:From /opt/ds3/lib/python3.5/site-packages/tensorflow/contrib/slim/python/slim/learning.py:736: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession 2018-11-06 19:39:40.247068: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2018-11-06 19:39:40.247537: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties: name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53 pciBusID: 0000:00:1e.0 totalMemory: 15.78GiB freeMemory: 15.37GiB 2018-11-06 19:39:40.247568: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0 2018-11-06 19:40:26.442477: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix: 2018-11-06 19:40:26.442536: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929] 0 2018-11-06 19:40:26.442548: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0: N 2018-11-06 19:40:26.443309: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14884 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)  

INFO:tensorflow:Restoring parameters from checkpoints/mobilenet_v2_1.0_224/mobilenet_v2_1.0_224.ckpt  
INFO:tensorflow:Running local_init_op.  
INFO:tensorflow:Done running local_init_op.  
INFO:tensorflow:Starting Session.  
INFO:tensorflow:Saving checkpoint to path output/mobile_net/model.ckpt  
INFO:tensorflow:Starting Queues.  
INFO:tensorflow:global_step/sec: 0  
INFO:tensorflow:Recording summary at step 0.  
INFO:tensorflow:global step 10: loss = 3.1285 (0.138 sec/step)  
INFO:tensorflow:global step 20: loss = 2.9970 (0.133 sec/step)  
INFO:tensorflow:global step 30: loss = 2.8800 (0.135 sec/step)  
INFO:tensorflow:global step 40: loss = 2.5745 (0.134 sec/step)  
INFO:tensorflow:global step 50: loss = 2.4919 (0.137 sec/step)  
INFO:tensorflow:global step 60: loss = 2.4559 (0.137 sec/step)  
...

I also tried setting the atrous rates and used the same parameters as @yswang0522 but this was even worse with 9.8% mIOU.

Any idea why I got such bad results?

Thanks!
Adva

[advaza]

Answering my own question: the default base learning rate is set to fine-tuning mode (0.0001), the learning rate that should be set when retraining the network is 0.007.
Now, mIOU is 70.67%! (although in the benchmark they got 75.32%, maybe more fine-tuning is needed for that).

[KleinXin]

@advaza According to your experience, I did three experiments using pascal voc trainval dataset.

I. Donot use any pretrained model, set NUM_ITERATIONS=3k, the mIoU is 5.64%
II. Donot use any pretrained model, set NUM_ITERATIONS=30k, the mIoU is 44.63%
III. Use mobilenet v2 1.0 224 as the pretrained model, set NUM_ITERATIONS=30k, the mIoU is 83.72%

In experiment III, it seems the backbone parameters are not initialised by mobilenet v2 1.0 224 imagenet pretrained model as the outputs showed in console

W0726 15:53:19.950658 139725007759104 variables.py:676] Checkpoint is missing variable [image_pooling/weights] W0726 15:53:19.950911 139725007759104 variables.py:676] Checkpoint is missing variable [image_pooling/BatchNorm/gamma] W0726 15:53:19.951029 139725007759104 variables.py:676] Checkpoint is missing variable [image_pooling/BatchNorm/beta] W0726 15:53:19.951102 139725007759104 variables.py:676] Checkpoint is missing variable [image_pooling/BatchNorm/moving_mean] W0726 15:53:19.951172 139725007759104 variables.py:676] Checkpoint is missing variable [image_pooling/BatchNorm/moving_variance] W0726 15:53:19.951312 139725007759104 variables.py:676] Checkpoint is missing variable [aspp0/weights] W0726 15:53:19.951390 139725007759104 variables.py:676] Checkpoint is missing variable [aspp0/BatchNorm/gamma] W0726 15:53:19.951459 139725007759104 variables.py:676] Checkpoint is missing variable [aspp0/BatchNorm/beta] W0726 15:53:19.951543 139725007759104 variables.py:676] Checkpoint is missing variable [aspp0/BatchNorm/moving_mean] W0726 15:53:19.951616 139725007759104 variables.py:676] Checkpoint is missing variable [aspp0/BatchNorm/moving_variance] W0726 15:53:19.951682 139725007759104 variables.py:676] Checkpoint is missing variable [concat_projection/weights] W0726 15:53:19.951747 139725007759104 variables.py:676] Checkpoint is missing variable [concat_projection/BatchNorm/gamma] W0726 15:53:19.951811 139725007759104 variables.py:676] Checkpoint is missing variable [concat_projection/BatchNorm/beta] W0726 15:53:19.951874 139725007759104 variables.py:676] Checkpoint is missing variable [concat_projection/BatchNorm/moving_mean] W0726 15:53:19.951937 139725007759104 variables.py:676] Checkpoint is missing variable [concat_projection/BatchNorm/moving_variance] W0726 15:53:19.952002 139725007759104 variables.py:676] Checkpoint is missing variable [logits/semantic/weights] W0726 15:53:19.952067 139725007759104 variables.py:676] Checkpoint is missing variable [logits/semantic/biases] W0726 15:53:19.952163 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/Conv/weights/Momentum] W0726 15:53:19.952236 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/Conv/BatchNorm/gamma/Momentum] W0726 15:53:19.952301 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/Conv/BatchNorm/beta/Momentum] W0726 15:53:19.952365 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv/depthwise/depthwise_weights/Momentum] W0726 15:53:19.952431 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv/depthwise/BatchNorm/gamma/Momentum] W0726 15:53:19.952496 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv/depthwise/BatchNorm/beta/Momentum] W0726 15:53:19.952561 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv/project/weights/Momentum] W0726 15:53:19.952626 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv/project/BatchNorm/gamma/Momentum] W0726 15:53:19.952690 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv/project/BatchNorm/beta/Momentum] W0726 15:53:19.952756 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_1/expand/weights/Momentum] W0726 15:53:19.952821 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_1/expand/BatchNorm/gamma/Momentum] W0726 15:53:19.952886 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_1/expand/BatchNorm/beta/Momentum] W0726 15:53:19.952950 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_1/depthwise/depthwise_weights/Momentum] W0726 15:53:19.953016 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_1/depthwise/BatchNorm/gamma/Momentum] W0726 15:53:19.953080 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_1/depthwise/BatchNorm/beta/Momentum] W0726 15:53:19.953145 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_1/project/weights/Momentum] W0726 15:53:19.953210 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_1/project/BatchNorm/gamma/Momentum] W0726 15:53:19.953276 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_1/project/BatchNorm/beta/Momentum] W0726 15:53:19.953341 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_2/expand/weights/Momentum] W0726 15:53:19.953407 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_2/expand/BatchNorm/gamma/Momentum] W0726 15:53:19.953474 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_2/expand/BatchNorm/beta/Momentum] W0726 15:53:19.953551 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_2/depthwise/depthwise_weights/Momentum] W0726 15:53:19.953627 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_2/depthwise/BatchNorm/gamma/Momentum] W0726 15:53:19.953701 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_2/depthwise/BatchNorm/beta/Momentum] W0726 15:53:19.953827 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_2/project/weights/Momentum] W0726 15:53:19.953901 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_2/project/BatchNorm/gamma/Momentum] W0726 15:53:19.953973 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_2/project/BatchNorm/beta/Momentum] W0726 15:53:19.954045 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_3/expand/weights/Momentum] W0726 15:53:19.954134 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_3/expand/BatchNorm/gamma/Momentum] W0726 15:53:19.954205 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_3/expand/BatchNorm/beta/Momentum] W0726 15:53:19.954274 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_3/depthwise/depthwise_weights/Momentum] W0726 15:53:19.954343 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_3/depthwise/BatchNorm/gamma/Momentum] W0726 15:53:19.954416 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_3/depthwise/BatchNorm/beta/Momentum] W0726 15:53:19.954487 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_3/project/weights/Momentum] W0726 15:53:19.954559 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_3/project/BatchNorm/gamma/Momentum] W0726 15:53:19.954674 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_3/project/BatchNorm/beta/Momentum] W0726 15:53:19.954766 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_4/expand/weights/Momentum] W0726 15:53:19.954837 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_4/expand/BatchNorm/gamma/Momentum] W0726 15:53:19.954921 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_4/expand/BatchNorm/beta/Momentum] W0726 15:53:19.954990 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_4/depthwise/depthwise_weights/Momentum] W0726 15:53:19.955059 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_4/depthwise/BatchNorm/gamma/Momentum] W0726 15:53:19.955127 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_4/depthwise/BatchNorm/beta/Momentum] W0726 15:53:19.955193 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_4/project/weights/Momentum] W0726 15:53:19.955260 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_4/project/BatchNorm/gamma/Momentum] W0726 15:53:19.955327 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_4/project/BatchNorm/beta/Momentum] W0726 15:53:19.955394 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_5/expand/weights/Momentum] W0726 15:53:19.955462 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_5/expand/BatchNorm/gamma/Momentum] W0726 15:53:19.955528 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_5/expand/BatchNorm/beta/Momentum] W0726 15:53:19.955596 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_5/depthwise/depthwise_weights/Momentum] W0726 15:53:19.955664 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_5/depthwise/BatchNorm/gamma/Momentum] W0726 15:53:19.955747 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_5/depthwise/BatchNorm/beta/Momentum] W0726 15:53:19.955813 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_5/project/weights/Momentum] W0726 15:53:19.955877 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_5/project/BatchNorm/gamma/Momentum] W0726 15:53:19.955942 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_5/project/BatchNorm/beta/Momentum] W0726 15:53:19.956006 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_6/expand/weights/Momentum] W0726 15:53:19.956071 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_6/expand/BatchNorm/gamma/Momentum] W0726 15:53:19.956136 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_6/expand/BatchNorm/beta/Momentum] W0726 15:53:19.956216 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_6/depthwise/depthwise_weights/Momentum] W0726 15:53:19.956281 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_6/depthwise/BatchNorm/gamma/Momentum] W0726 15:53:19.956345 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_6/depthwise/BatchNorm/beta/Momentum] W0726 15:53:19.956408 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_6/project/weights/Momentum] W0726 15:53:19.956473 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_6/project/BatchNorm/gamma/Momentum] W0726 15:53:19.956539 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_6/project/BatchNorm/beta/Momentum] W0726 15:53:19.956609 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_7/expand/weights/Momentum] W0726 15:53:19.956679 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_7/expand/BatchNorm/gamma/Momentum] W0726 15:53:19.956749 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_7/expand/BatchNorm/beta/Momentum] W0726 15:53:19.956819 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_7/depthwise/depthwise_weights/Momentum] W0726 15:53:19.956888 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_7/depthwise/BatchNorm/gamma/Momentum] W0726 15:53:19.956957 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_7/depthwise/BatchNorm/beta/Momentum] W0726 15:53:19.957026 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_7/project/weights/Momentum] W0726 15:53:19.957095 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_7/project/BatchNorm/gamma/Momentum] W0726 15:53:19.957163 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_7/project/BatchNorm/beta/Momentum] W0726 15:53:19.957232 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_8/expand/weights/Momentum] W0726 15:53:19.957299 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_8/expand/BatchNorm/gamma/Momentum] W0726 15:53:19.957367 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_8/expand/BatchNorm/beta/Momentum] W0726 15:53:19.957436 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_8/depthwise/depthwise_weights/Momentum] W0726 15:53:19.957504 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_8/depthwise/BatchNorm/gamma/Momentum] W0726 15:53:19.957570 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_8/depthwise/BatchNorm/beta/Momentum] W0726 15:53:19.957637 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_8/project/weights/Momentum] W0726 15:53:19.957706 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_8/project/BatchNorm/gamma/Momentum] W0726 15:53:19.957776 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_8/project/BatchNorm/beta/Momentum] W0726 15:53:19.957844 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_9/expand/weights/Momentum] W0726 15:53:19.957913 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_9/expand/BatchNorm/gamma/Momentum] W0726 15:53:19.957982 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_9/expand/BatchNorm/beta/Momentum] W0726 15:53:19.958050 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_9/depthwise/depthwise_weights/Momentum] W0726 15:53:19.958118 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_9/depthwise/BatchNorm/gamma/Momentum] W0726 15:53:19.958187 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_9/depthwise/BatchNorm/beta/Momentum] W0726 15:53:19.958256 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_9/project/weights/Momentum] W0726 15:53:19.958326 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_9/project/BatchNorm/gamma/Momentum] W0726 15:53:19.958437 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_9/project/BatchNorm/beta/Momentum] W0726 15:53:19.958510 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_10/expand/weights/Momentum] W0726 15:53:19.958578 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_10/expand/BatchNorm/gamma/Momentum] W0726 15:53:19.958699 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_10/expand/BatchNorm/beta/Momentum] W0726 15:53:19.958785 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_10/depthwise/depthwise_weights/Momentum] W0726 15:53:19.958851 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_10/depthwise/BatchNorm/gamma/Momentum] W0726 15:53:19.958932 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_10/depthwise/BatchNorm/beta/Momentum] W0726 15:53:19.958995 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_10/project/weights/Momentum] W0726 15:53:19.959059 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_10/project/BatchNorm/gamma/Momentum] W0726 15:53:19.959122 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_10/project/BatchNorm/beta/Momentum] W0726 15:53:19.959187 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_11/expand/weights/Momentum] W0726 15:53:19.959250 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_11/expand/BatchNorm/gamma/Momentum] W0726 15:53:19.959313 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_11/expand/BatchNorm/beta/Momentum] W0726 15:53:19.959376 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_11/depthwise/depthwise_weights/Momentum] W0726 15:53:19.959439 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_11/depthwise/BatchNorm/gamma/Momentum] W0726 15:53:19.959503 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_11/depthwise/BatchNorm/beta/Momentum] W0726 15:53:19.959569 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_11/project/weights/Momentum] W0726 15:53:19.959638 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_11/project/BatchNorm/gamma/Momentum] W0726 15:53:19.959707 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_11/project/BatchNorm/beta/Momentum] W0726 15:53:19.959775 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_12/expand/weights/Momentum] W0726 15:53:19.959844 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_12/expand/BatchNorm/gamma/Momentum] W0726 15:53:19.959912 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_12/expand/BatchNorm/beta/Momentum] W0726 15:53:19.959981 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_12/depthwise/depthwise_weights/Momentum] W0726 15:53:19.960050 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_12/depthwise/BatchNorm/gamma/Momentum] W0726 15:53:19.960118 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_12/depthwise/BatchNorm/beta/Momentum] W0726 15:53:19.960186 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_12/project/weights/Momentum] W0726 15:53:19.960253 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_12/project/BatchNorm/gamma/Momentum] W0726 15:53:19.960321 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_12/project/BatchNorm/beta/Momentum] W0726 15:53:19.960389 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_13/expand/weights/Momentum] W0726 15:53:19.960457 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_13/expand/BatchNorm/gamma/Momentum] W0726 15:53:19.960525 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_13/expand/BatchNorm/beta/Momentum] W0726 15:53:19.960593 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_13/depthwise/depthwise_weights/Momentum] W0726 15:53:19.960663 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_13/depthwise/BatchNorm/gamma/Momentum] W0726 15:53:19.960730 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_13/depthwise/BatchNorm/beta/Momentum] W0726 15:53:19.960798 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_13/project/weights/Momentum] W0726 15:53:19.960867 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_13/project/BatchNorm/gamma/Momentum] W0726 15:53:19.960933 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_13/project/BatchNorm/beta/Momentum] W0726 15:53:19.961002 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_14/expand/weights/Momentum] W0726 15:53:19.961070 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_14/expand/BatchNorm/gamma/Momentum] W0726 15:53:19.961138 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_14/expand/BatchNorm/beta/Momentum] W0726 15:53:19.961207 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_14/depthwise/depthwise_weights/Momentum] W0726 15:53:19.961274 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_14/depthwise/BatchNorm/gamma/Momentum] W0726 15:53:19.961342 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_14/depthwise/BatchNorm/beta/Momentum] W0726 15:53:19.961410 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_14/project/weights/Momentum] W0726 15:53:19.961479 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_14/project/BatchNorm/gamma/Momentum] W0726 15:53:19.961547 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_14/project/BatchNorm/beta/Momentum] W0726 15:53:19.961615 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_15/expand/weights/Momentum] W0726 15:53:19.961682 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_15/expand/BatchNorm/gamma/Momentum] W0726 15:53:19.961750 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_15/expand/BatchNorm/beta/Momentum] W0726 15:53:19.961819 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_15/depthwise/depthwise_weights/Momentum] W0726 15:53:19.961887 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_15/depthwise/BatchNorm/gamma/Momentum] W0726 15:53:19.961982 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_15/depthwise/BatchNorm/beta/Momentum] W0726 15:53:19.962054 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_15/project/weights/Momentum] W0726 15:53:19.962124 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_15/project/BatchNorm/gamma/Momentum] W0726 15:53:19.962192 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_15/project/BatchNorm/beta/Momentum] W0726 15:53:19.962259 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_16/expand/weights/Momentum] W0726 15:53:19.962327 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_16/expand/BatchNorm/gamma/Momentum] W0726 15:53:19.962393 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_16/expand/BatchNorm/beta/Momentum] W0726 15:53:19.962460 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_16/depthwise/depthwise_weights/Momentum] W0726 15:53:19.962529 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_16/depthwise/BatchNorm/gamma/Momentum] W0726 15:53:19.962628 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_16/depthwise/BatchNorm/beta/Momentum] W0726 15:53:19.962721 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_16/project/weights/Momentum] W0726 15:53:19.962790 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_16/project/BatchNorm/gamma/Momentum] W0726 15:53:19.962856 139725007759104 variables.py:676] Checkpoint is missing variable [MobilenetV2/expanded_conv_16/project/BatchNorm/beta/Momentum] W0726 15:53:19.962936 139725007759104 variables.py:676] Checkpoint is missing variable [image_pooling/weights/Momentum] W0726 15:53:19.963001 139725007759104 variables.py:676] Checkpoint is missing variable [image_pooling/BatchNorm/gamma/Momentum] W0726 15:53:19.963063 139725007759104 variables.py:676] Checkpoint is missing variable [image_pooling/BatchNorm/beta/Momentum] W0726 15:53:19.963159 139725007759104 variables.py:676] Checkpoint is missing variable [aspp0/weights/Momentum] W0726 15:53:19.963230 139725007759104 variables.py:676] Checkpoint is missing variable [aspp0/BatchNorm/gamma/Momentum] W0726 15:53:19.963294 139725007759104 variables.py:676] Checkpoint is missing variable [aspp0/BatchNorm/beta/Momentum] W0726 15:53:19.963357 139725007759104 variables.py:676] Checkpoint is missing variable [concat_projection/weights/Momentum] W0726 15:53:19.963420 139725007759104 variables.py:676] Checkpoint is missing variable [concat_projection/BatchNorm/gamma/Momentum] W0726 15:53:19.963482 139725007759104 variables.py:676] Checkpoint is missing variable [concat_projection/BatchNorm/beta/Momentum] W0726 15:53:19.963546 139725007759104 variables.py:676] Checkpoint is missing variable [logits/semantic/weights/Momentum] W0726 15:53:19.963608 139725007759104 variables.py:676] Checkpoint is missing variable [logits/semantic/biases/Momentum]

I am trying to use the mobilenet v2 0.5 224 to train the model. The configuration is the same as Experimants III. Let's see what will happen.

Thank you for your suggestion to set base_learning_rate to 0.007！

[bhavitvyamalik]

@advaza did you try using the optimizer flag while training the script? Loss will go down drastically within a few epochs. My model started overfitting in 2000 epochs using adam optimizer.

[tensorflow-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[tensorflow-butler[bot]]

Closing as stale. Please reopen if you'd like to work on this further.