Models: ssd_resnet_50_v1_fpn training problems: not available in checkpoint
System information
What is the top-level directory of the model you are using:
Tensorflow Object Detection API
Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
No, except comment 'from object_detection import eval_util' because i can't install pycocotools using vs 2017
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
windows 10 17134.191
TensorFlow installed from (source or binary):
Binary
TensorFlow version (use command below):
1.8.0
CUDA/cuDNN version:
CUDA 9.0
cuDNN 7.1
GPU model and memory:
Quodra M4000, 8G memory
Exact command to reproduce:
python .\model_main.py
--pipeline_config_path .\ssd_resnet50_v1_fpn_shared_box_predictor_640x640_coco14_sync.config
--model_dir models --num_train_steps 3000 --alsologtostderr
Bazel version: N/A
Describe the problem
WARNING:tensorflow:Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x00000250139181E0>) includes params argument, but params are not passed to Estimator.
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv3/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv1/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv1/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv2/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv3/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv1/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv2/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv3/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/shortcut/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/shortcut/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/shortcut/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv1/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv2/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv3/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv1/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv2/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv3/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv1/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv2/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv3/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv1/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv3/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/shortcut/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/shortcut/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/shortcut/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv1/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv2/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv3/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv1/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv2/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv3/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv1/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv2/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv3/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv1/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv3/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv1/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv2/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv3/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv1/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv2/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv3/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/shortcut/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv1/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv2/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv3/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv1/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv2/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/conv1/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/conv1/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/conv1/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/conv1/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/conv1/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block5/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block5/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block5/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block5/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block5/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block6/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block6/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block6/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block6/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block6/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/projection_1/biases] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/projection_1/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/projection_2/biases] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/projection_2/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/projection_3/biases] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/projection_3/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/smoothing_1/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/smoothing_1/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/smoothing_1/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/smoothing_1/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/smoothing_1/weights] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/smoothing_2/BatchNorm/beta] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/smoothing_2/BatchNorm/gamma] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/smoothing_2/BatchNorm/moving_mean] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/smoothing_2/BatchNorm/moving_variance] is not available in checkpoint
WARNING:root:Variable [resnet_v1_50/fpn/smoothing_2/weights] is not available in checkpoint
Traceback (most recent call last):
File ".\model_main.py", line 101, in <module>
tf.app.run()
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\platform\app.py", line 126, in run
_sys.exit(main(argv))
File ".\model_main.py", line 97, in main
tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 439, in train_and_evaluate
executor.run()
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 518, in run
self.run_local()
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 650, in run_local
hooks=train_hooks)
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 363, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 843, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 856, in _train_model_default
features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 831, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "E:\ChinaRS\code\tensorflow\models\research\object_detection\model_lib.py", line 282, in model_fn
losses = [loss_tensor for loss_tensor in losses_dict.itervalues()]
AttributeError: 'dict' object has no attribute 'itervalues'
when using codes below to test:
import tensorflow as tf
prefix = "model.ckpt"
saver = tf.train.import_meta_graph("{}.meta".format(prefix))
sess = tf.Session()
saver.restore(sess, prefix)
graph=tf.get_default_graph()
for op in graph.get_operations():
print(op.name)
it print like this: (too long to paste here)
FeatureExtractor/resnet_v1_50/conv1/weights/Initializer/truncated_normal/shape
FeatureExtractor/resnet_v1_50/conv1/weights/Initializer/truncated_normal/TruncatedNormal
FeatureExtractor/resnet_v1_50/conv1/weights/Initializer/truncated_normal/mean
FeatureExtractor/resnet_v1_50/conv1/weights/Initializer/truncated_normal/stddev
FeatureExtractor/resnet_v1_50/conv1/weights/Initializer/truncated_normal/mul
FeatureExtractor/resnet_v1_50/conv1/weights/Initializer/truncated_normal
FeatureExtractor/resnet_v1_50/conv1/weights
FeatureExtractor/resnet_v1_50/conv1/weights/Assign
FeatureExtractor/resnet_v1_50/conv1/weights/read
FeatureExtractor/resnet_v1_50/resnet_v1_50/conv1/Conv2D
FeatureExtractor/resnet_v1_50/resnet_v1_50/conv1/kernel/Regularizer/l2_regularizer/L2Loss
FeatureExtractor/resnet_v1_50/resnet_v1_50/conv1/kernel/Regularizer/l2_regularizer/scale
FeatureExtractor/resnet_v1_50/resnet_v1_50/conv1/kernel/Regularizer/l2_regularizer
FeatureExtractor/resnet_v1_50/resnet_v1_50/conv1/dilation_rate
FeatureExtractor/resnet_v1_50/conv1/BatchNorm/gamma/Initializer/ones
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/gamma/read
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/beta/Initializer/zeros
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/beta
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/beta/Assign
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/beta/read
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean/Initializer/zeros
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean/Assign
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean/read
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance/Initializer/ones
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance/Assign
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance/read
FeatureExtractor/resnet_v1_50/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/FusedBatchNorm
FeatureExtractor/resnet_v1_50/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/Const
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/moving_mean/read
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance/Initializer/ones
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance/Assign
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance/read
FeatureExtractor/resnet_v1_50/resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/FusedBatchNorm
FeatureExtractor/resnet_v1_50/resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/Relu
FeatureExtractor/resnet_v1_50/resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/Const
FeatureExtractor/resnet_v1_50/resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/Relu
FeatureExtractor/resnet_v1_50/resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/BatchNorm/Const
FeatureExtractor/resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/weights/Initializer/truncated_normal/shape
FeatureExtractor/resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/weights/Initializer/truncated_normal/TruncatedNormal
FeatureExtractor/resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/weights/Initializer/truncated_normal/mean
FeatureExtractor/resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/weights/Initializer/truncated_normal/stddev
FeatureExtractor/resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/weights/Initializer/truncated_normal/mul
it seems that all have a prefix -- 'FeatureExtractor'
i think it has something to do with it.
yuezhilanyi
All 27 comments
Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Bazel version
tensorflowbutler
on 30 Jul 2018
@tensorflowbutler updated. thanks
yuezhilanyi
on 31 Jul 2018
updated:
- i am using python 3.5, while i found there are some python2 code in ssd models
- when i ran object_detection/meta_architectures/ssd_meta_arch_test.py, it failed with 1 error:
ERROR: test_restore_map_for_classification_ckpt (__main__.SsdMetaArchTest)
Traceback (most recent call last):
File ".\ssd_meta_arch_test.py", line 492, in test_restore_map_for_classification_ckpt
self.assertNotIn('FeatureExtractor', var)
File "D:\Program Files\Anaconda3\lib\unittest\case.py", line 1083, in assertNotIn
if member in container:
TypeError: a bytes-like object is required, not 'str'
Ran 14 tests in 7.938s
FAILED (errors=1)
yuezhilanyi
on 31 Jul 2018
updated2:
modify object_detection/meta_architectures/ssd_meta_arch.py, change line 870 from
re.split('^' + self._extract_features_scope + '/',
to
re.split('^' + '/',
can train sucessfully, but it has duplicated steps like
INFO:tensorflow:global step 1954: loss = 0.2881 (1.205 sec/step)
INFO:tensorflow:global step 1954: loss = 0.2881 (1.205 sec/step))
INFO:tensorflow:global step 1955: loss = 0.2873 (1.335 sec/step)
INFO:tensorflow:global step 1955: loss = 0.2873 (1.335 sec/step)
Exact command to reproduce:
python object_detection/legacy/train.py --logtostderr --pipeline_config_path config_file --train_dir models
yuezhilanyi
on 31 Jul 2018
As I mentioned in the PR, removing _extract_features_scope in
re.split('^' + self._extract_features_scope + '/', is not the right move.
Could you please share your config file? What fine tune checkpoint are you using?
pkulzc
on 1 Aug 2018
config file (txt file for upload)
ssd_resnet50_v1_fpn_shared_box_predictor_640x640_coco14_sync.txt
meta arch (txt file for upload)
ssd_meta_arch.txt
fine tune checkpoint:
ssd_resnet50_v1_fpn_shared_box_predictor_640x640_coco14_sync_2018_07_03
exact command to reproduce:
python train.py --logtostderr --pipeline_config_path .\ssd_resnet50_v1
_fpn_shared_box_predictor_640x640_coco14_sync.config --train_dir models
new command like below still fails with this modification
python .\model_main.py
--pipeline_config_path .\ssd_resnet50_v1_fpn_shared_box_predictor_640x640_coco14_sync.config
--model_dir models --num_train_steps 3000 --alsologtostderr
@pkulzc
yuezhilanyi
on 2 Aug 2018
As I said, when using a detection checkpoint you should set from_detection_checkpoint to true like this.
pkulzc
on 2 Aug 2018
@pkulzc
after setting from_detection_checkpoint to true:
- nothing changes while using 'python train.py --logtostderr --pipeline_config_path .\ssd_resnet50_v1_fpn_shared_box_predictor_640x640_coco14_sync.config --train_dir models'
- using model_main.py to train failed with below
python .\model_main.py --pipeline_config_path .\ssd_resnet50_v1_fpn_shared_box_predictor_640x640_coco14_sync.config --model_dir models --num_train_steps 3000 --alsologtostderr
WARNING:tensorflow:Estimator's model_fn (.model_fn at 0x000002345B7E51E0>) includes params argument, but params are not passed to Estimator.
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
2018-08-02 13:51:38.110591: I T:src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2018-08-02 13:51:38.301673: I T:src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1356] Found device 0 with properties:
name: Quadro M4000 major: 5 minor: 2 memoryClockRate(GHz): 0.7725
pciBusID: 0000:03:00.0
totalMemory: 8.00GiB freeMemory: 6.70GiB
2018-08-02 13:51:38.307075: I T:src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1435] Adding visible gpu devices: 0
2018-08-02 13:51:38.583619: I T:src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-08-02 13:51:38.586696: I T:src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:929] 0
2018-08-02 13:51:38.588632: I T:src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:942] 0: N
2018-08-02 13:51:38.590678: I T:src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6476 MB memory) -> physical GPU (device: 0, name: Quadro M4000, pci bus id: 0000:03:00.0, compute capability: 5.2)
ERROR:tensorflow:Model diverged with loss = NaN.
Traceback (most recent call last):
File ".\model_main.py", line 101, in
tf.app.run()
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\platform\app.py", line 126, in run
_sys.exit(main(argv))
File ".\model_main.py", line 97, in main
tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 439, in train_and_evaluate
executor.run()
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 518, in run
self.run_local()
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 650, in run_local
hooks=train_hooks)
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 363, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 843, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 859, in _train_model_default
saving_listeners)
File "D:\Program Files\Anaconda3\lib\contextlib.py", line 77, in __exit__
self.gen.throw(type, value, traceback)
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 5265, in get_controller
yield g
File "D:\Program Files\Anaconda3\lib\contextlib.py", line 77, in __exit__
self.gen.throw(type, value, traceback)
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 5060, in get_controller
yield default
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 5265, in get_controller
yield g
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 859, in _train_model_default
saving_listeners)
File "D:\Program Files\Anaconda3\lib\contextlib.py", line 77, in __exit__
self.gen.throw(type, value, traceback)
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 4338, in device
yield
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 859, in _train_model_default
saving_listeners)
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1059, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 567, in run
run_metadata=run_metadata)
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1043, in run
run_metadata=run_metadata)
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1134, in run
raise six.reraise(original_exc_info)
File "D:\Program Files\Anaconda3\lib\site-packages\six.py", line 686, in reraise
raise value
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1119, in run
return self._sess.run(args, **kwargs)
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1199, in run
run_metadata=run_metadata))
File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 623, in after_run
raise NanLossDuringTrainingError
tensorflow.python.training.basic_session_run_hooks.NanLossDuringTrainingError: NaN loss during training.
yuezhilanyi
on 2 Aug 2018
Looks like you may need a lower learning rate.
pkulzc
on 2 Aug 2018
i have tried from .04 to .00000004 (warmup_learning_rate from .013333 to .000000013333), it made no difference; meanwhile, using legacy training (python object_detection/legacy/train.py --logtostderr --pipeline_config_path config_file --train_dir models) can run sucessfully;
does it means that we need to change learning rate when using new train commands?
yuezhilanyi
on 6 Aug 2018
@yuezhilanyi same problem with me on windows.the loss is Nan when I train ssd_inception_v3_pets.config with Pascal datasets.
kingstarcraft
on 8 Aug 2018
@kingstarcraft do you have tried legacy/train.py? it may help
yuezhilanyi
on 8 Aug 2018
@yuezhilanyi It's work, thank you.
kingstarcraft
on 8 Aug 2018
@tensorflowbutler updated. thanks
What field? I'm having the same problem
Elites2017
on 14 Sep 2018
System information
What is the top-level directory of the model you are using: Tensorflow Object Detection API Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No, except comment 'from object_detection import eval_util' because i can't install pycocotools using vs 2017 OS Platform and Distribution (e.g., Linux Ubuntu 16.04): windows 10 17134.191 TensorFlow installed from (source or binary): Binary TensorFlow version (use command below): 1.8.0 CUDA/cuDNN version: CUDA 9.0 cuDNN 7.1 GPU model and memory: Quodra M4000, 8G memory Exact command to reproduce: python .\model_main.py --pipeline_config_path .\ssd_resnet50_v1_fpn_shared_box_predictor_640x640_coco14_sync.config --model_dir models --num_train_steps 3000 --alsologtostderr Bazel version: N/ADescribe the problem
WARNING:tensorflow:Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x00000250139181E0>) includes params argument, but params are not passed to Estimator. WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards. WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block1/unit_3/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/shortcut/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/shortcut/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_1/bottleneck_v1/shortcut/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_2/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_3/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block2/unit_4/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/shortcut/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/shortcut/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_1/bottleneck_v1/shortcut/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_2/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_3/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_4/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_5/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block3/unit_6/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_1/bottleneck_v1/shortcut/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_2/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/conv1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/conv1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/conv1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/conv1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/conv1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block5/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block5/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block5/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block5/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block5/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block6/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block6/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block6/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block6/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/bottom_up_block6/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/projection_1/biases] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/projection_1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/projection_2/biases] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/projection_2/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/projection_3/biases] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/projection_3/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/smoothing_1/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/smoothing_1/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/smoothing_1/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/smoothing_1/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/smoothing_1/weights] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/smoothing_2/BatchNorm/beta] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/smoothing_2/BatchNorm/gamma] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/smoothing_2/BatchNorm/moving_mean] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/smoothing_2/BatchNorm/moving_variance] is not available in checkpoint WARNING:root:Variable [resnet_v1_50/fpn/smoothing_2/weights] is not available in checkpoint Traceback (most recent call last): File ".\model_main.py", line 101, in <module> tf.app.run() File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\platform\app.py", line 126, in run _sys.exit(main(argv)) File ".\model_main.py", line 97, in main tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0]) File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 439, in train_and_evaluate executor.run() File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 518, in run self.run_local() File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\estimator\training.py", line 650, in run_local hooks=train_hooks) File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 363, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 843, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 856, in _train_model_default features, labels, model_fn_lib.ModeKeys.TRAIN, self.config) File "D:\Program Files\Anaconda3\lib\site-packages\tensorflow\python\estimator\estimator.py", line 831, in _call_model_fn model_fn_results = self._model_fn(features=features, **kwargs) File "E:\ChinaRS\code\tensorflow\models\research\object_detection\model_lib.py", line 282, in model_fn losses = [loss_tensor for loss_tensor in losses_dict.itervalues()] AttributeError: 'dict' object has no attribute 'itervalues'when using codes below to test:
import tensorflow as tf prefix = "model.ckpt" saver = tf.train.import_meta_graph("{}.meta".format(prefix)) sess = tf.Session() saver.restore(sess, prefix) graph=tf.get_default_graph() for op in graph.get_operations(): print(op.name)
it print like this: (too long to paste here)FeatureExtractor/resnet_v1_50/conv1/weights/Initializer/truncated_normal/shape
FeatureExtractor/resnet_v1_50/conv1/weights/Initializer/truncated_normal/TruncatedNormal
FeatureExtractor/resnet_v1_50/conv1/weights/Initializer/truncated_normal/mean
FeatureExtractor/resnet_v1_50/conv1/weights/Initializer/truncated_normal/stddev
FeatureExtractor/resnet_v1_50/conv1/weights/Initializer/truncated_normal/mul
FeatureExtractor/resnet_v1_50/conv1/weights/Initializer/truncated_normal
FeatureExtractor/resnet_v1_50/conv1/weights
FeatureExtractor/resnet_v1_50/conv1/weights/Assign
FeatureExtractor/resnet_v1_50/conv1/weights/read
FeatureExtractor/resnet_v1_50/resnet_v1_50/conv1/Conv2D
FeatureExtractor/resnet_v1_50/resnet_v1_50/conv1/kernel/Regularizer/l2_regularizer/L2Loss
FeatureExtractor/resnet_v1_50/resnet_v1_50/conv1/kernel/Regularizer/l2_regularizer/scale
FeatureExtractor/resnet_v1_50/resnet_v1_50/conv1/kernel/Regularizer/l2_regularizer
FeatureExtractor/resnet_v1_50/resnet_v1_50/conv1/dilation_rate
FeatureExtractor/resnet_v1_50/conv1/BatchNorm/gamma/Initializer/ones
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/gamma/read
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/beta/Initializer/zeros
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/beta
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/beta/Assign
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/beta/read
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean/Initializer/zeros
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean/Assign
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean/read
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance/Initializer/ones
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance/Assign
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance/read
FeatureExtractor/resnet_v1_50/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/FusedBatchNorm
FeatureExtractor/resnet_v1_50/resnet_v1_50/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/Const
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/moving_mean/read
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance/Initializer/ones
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance/Assign
FeatureExtractor/resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance/read
FeatureExtractor/resnet_v1_50/resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/FusedBatchNorm
FeatureExtractor/resnet_v1_50/resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/Relu
FeatureExtractor/resnet_v1_50/resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/BatchNorm/Const
FeatureExtractor/resnet_v1_50/resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/Relu
FeatureExtractor/resnet_v1_50/resnet_v1_50/block1/unit_2/bottleneck_v1/conv2/BatchNorm/Const
FeatureExtractor/resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/weights/Initializer/truncated_normal/shape
FeatureExtractor/resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/weights/Initializer/truncated_normal/TruncatedNormal
FeatureExtractor/resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/weights/Initializer/truncated_normal/mean
FeatureExtractor/resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/weights/Initializer/truncated_normal/stddev
FeatureExtractor/resnet_v1_50/block1/unit_2/bottleneck_v1/conv3/weights/Initializer/truncated_normal/mulit seems that all have a prefix -- 'FeatureExtractor'
i think it has something to do with it.
I have the same problem with the model " mobilenet_v1_0.5_128 " / " mobilenet_v1_1.0_224 " using the config file: ssd_mobilenet_v1_pets.config downloaded from: https://github.com/pkulzc/models/blob/master/research/object_detection/samples/configs/ssd_mobilenet_v1_pets.config
What file and field to edit Guys?
Elites2017
on 14 Sep 2018
@Elites2017 When using a classification checkpoint, you should set from_detection_checkpoint: false.
pkulzc
on 14 Sep 2018
@pkulzc I was trying to solve my problem by updating my trainer.py file with the updated one on the tensorflow official github repository (the trainer.py file has less line of code now (copy and paste from the repository)). The training started well and stop / finished automatically at step 0.
I'm using a checkpoint from the mobilenet_v1_1.0_224 model to train my custom model. I don't know if i still have to set the from_detection_checkpoint to false.
Elites2017
on 14 Sep 2018
The code block here handles these prefixs. Basically,
- if you load from a classification checkpoint it removes the FeatureExtractor prefix for you.
- if you load from a detection checkpoint and set load_all_detection_checkpoint_vars=true, it loads everything
- if you load from a detection checkpoint and set load_all_detection_checkpoint_vars=false, it loads feature extractor nodes(aka classification nodes) from a detection checkpoint.
You should only modify the code if your situation doesn't match any of the aboves.
pkulzc
on 14 Sep 2018
It's training now...
@pkulzc Now I'm using the trainer.py (updated on the tensorflow official github repository, 370 lines, before it had 410)
I'm using a checkpoint from the mobilenet_v1_1.0_224 model and I set from_detection_checkpoint: false as you recommended in a comment above.
It's training well now.
Thanks Genius
Elites2017
on 14 Sep 2018
@pkulzc I ended the training because my goal is to have a frozen_graph.pb file that can be converted. That was why I used the mobilenet_v1_1.0_224 model, because it comes with the parameters such as: Input: input
Output: MobilenetV1/Predictions/Reshape_1
But I when I use the tflite tool to convert the fozen graph (using these mentioned parameters), it gives me an error: ValueError: Invalid tensors 'input' were found.
Any help please,
Elites2017
on 14 Sep 2018
@pkulzc
How or where can I see my input and output parameters in this graph?
Or what the command to directly import model (.pb) files to TensorBoard because the images attached to this comment come from the training model .ckpt
Elites2017
on 15 Sep 2018
@pkulzc
How or where can I see my input and output parameters in this graph?
Or what the command to directly import model (.pb) files to TensorBoard because the images attached to this comment come from the training model .ckpt
@yuezhilanyi @kingstarcraft can someone help me to identify the input and output names in this graph guys? I need them so that I can convert my .pb file to .tflite file for mobile devices
Thanks in advance
Elites2017
on 15 Sep 2018
sorry to tell you, i have never convert .pb to .tflite @Elites2017
yuezhilanyi
on 21 Sep 2018
sorry to tell you, i have never convert .pb to .tflite @Elites2017
It;s okay
Elites2017
on 22 Sep 2018
Some examples of warning
WARNING:root:Variable [FeatureExtractor/InceptionV2/Mixed_5c_2_Conv2d_5_3x3_s2_128/BatchNorm/gamma/RMSProp] is not available in checkpoint
WARNING:root:Variable [FeatureExtractor/InceptionV2/Mixed_5c_2_Conv2d_5_3x3_s2_128/BatchNorm/gamma/RMSProp_1] is not available in checkpoint
WARNING:root:Variable [FeatureExtractor/InceptionV2/Mixed_5c_2_Conv2d_5_3x3_s2_128/weights/ExponentialMovingAverage] is not available in checkpoint
WARNING:root:Variable [FeatureExtractor/InceptionV2/Mixed_5c_2_Conv2d_5_3x3_s2_128/weights/RMSProp] is not available in checkpoint
WARNING:root:Variable [FeatureExtractor/InceptionV2/Mixed_5c_2_Conv2d_5_3x3_s2_128/weights/RMSProp_1] is not available in checkpoint
**Any solutions? I'm using ssd_inception_v2_coco.
### pipline.config looks like that**
odel {
ssd {
num_classes: 10
box_coder {
faster_rcnn_box_coder {
y_scale: 10.0
x_scale: 10.0
height_scale: 5.0
width_scale: 5.0
}
}
matcher {
argmax_matcher {
matched_threshold: 0.5
unmatched_threshold: 0.5
ignore_thresholds: false
negatives_lower_than_unmatched: true
force_match_for_each_row: true
}
}
similarity_calculator {
iou_similarity {
}
}
anchor_generator {
ssd_anchor_generator {
num_layers: 6
min_scale: 0.2
max_scale: 0.95
aspect_ratios: 1.0
aspect_ratios: 2.0
aspect_ratios: 0.5
aspect_ratios: 3.0
aspect_ratios: 0.3333
reduce_boxes_in_lowest_layer: true
}
}
image_resizer {
fixed_shape_resizer {
height: 300
width: 300
}
}
box_predictor {
convolutional_box_predictor {
min_depth: 0
max_depth: 0
num_layers_before_predictor: 0
use_dropout: false
dropout_keep_probability: 0.8
kernel_size: 3
box_code_size: 4
apply_sigmoid_to_scores: false
conv_hyperparams {
activation: RELU_6,
regularizer {
l2_regularizer {
weight: 0.00004
}
}
initializer {
truncated_normal_initializer {
stddev: 0.03
mean: 0.0
}
}
}
}
}
feature_extractor {
type: 'ssd_inception_v2'
min_depth: 16
depth_multiplier: 1.0
conv_hyperparams {
activation: RELU_6,
regularizer {
l2_regularizer {
weight: 0.00004
}
}
initializer {
truncated_normal_initializer {
stddev: 0.03
mean: 0.0
}
}
batch_norm {
train: true,
scale: true,
center: true,
decay: 0.9997,
epsilon: 0.001,
}
}
override_base_feature_extractor_hyperparams: true
}
loss {
classification_loss {
weighted_sigmoid {
}
}
localization_loss {
weighted_smooth_l1 {
}
}
hard_example_miner {
num_hard_examples: 3000
iou_threshold: 0.99
loss_type: CLASSIFICATION
max_negatives_per_positive: 3
min_negatives_per_image: 0
}
classification_weight: 1.0
localization_weight: 1.0
}
normalize_loss_by_num_matches: true
post_processing {
batch_non_max_suppression {
score_threshold: 1e-8
iou_threshold: 0.6
max_detections_per_class: 100
max_total_detections: 100
}
score_converter: SIGMOID
}
}
}
train_config: {
batch_size: 10
optimizer {
rms_prop_optimizer: {
learning_rate: {
exponential_decay_learning_rate {
initial_learning_rate: 0.004
decay_steps: 800720
decay_factor: 0.95
}
}
momentum_optimizer_value: 0.9
decay: 0.9
epsilon: 1.0
}
}
fine_tune_checkpoint: "C:\Users\AB\Tensorflow\workplace\training_demo\pre-trained-model\model.ckpt"
from_detection_checkpoint: true
# Note: The below line limits the training process to 200K steps, which we
# empirically found to be sufficient enough to train the pets dataset. This
# effectively bypasses the learning rate schedule (the learning rate will
# never decay). Remove the below line to train indefinitely.
num_steps: 200000
fine_tune_checkpoint:"C:\Users\AB\Tensorflow\workplace\training_demo\pre-trained-model\model.ckpt"
fine_tune_checkpoint_type: "detection"
data_augmentation_options {
random_horizontal_flip {
}
}
data_augmentation_options {
ssd_random_crop {
}
}
}
train_input_reader: {
tf_record_input_reader {
input_path: "C:\Users\AB\Tensorflow\workplace\training_demo\annotations\train.record"
}
label_map_path: "C:\Users\AB\Tensorflow\workplace\training_demo\annotations\label_map.pbtxt"
}
eval_config: {
num_examples: 8000
# Note: The below line limits the evaluation process to 10 evaluations.
# Remove the below line to evaluate indefinitely.
max_evals: 10
}
eval_input_reader: {
tf_record_input_reader {
input_path: "C:\Users\AB\Tensorflow\workplace\training_demo\annotations\test.record"
}
label_map_path: "C:\Users\AB\Tensorflow\workplace\training_demo\annotations\label_map.pbtxt"
shuffle: false
num_readers: 1
}
shABanty
on 6 Feb 2020
@kingstarcraft do you have tried legacy/train.py? it may help
Hello, when i ran legacy/train.py, i caught error ModuleNotFoundError: No module named 'nets'
But if i ran model_main.py, not caught
AnhPC03
on 9 Jun 2020
@AnhPC03 I fixed the issue by adding $path_to_models/research and $path_to_models/research/slim to PYTHONPATH environment variable
parshad1710
on 8 Oct 2020
Related issues
ddurgaprasad
路
48Comments
walkerlala
路
98Comments
edwardHujber
路
82Comments
10183308
路
50Comments
pjeambrun
路
51Comments
Most helpful comment
As I said, when using a detection checkpoint you should set from_detection_checkpoint to true like this.