Models: [Deeplab DPC]Running eval/vis.py failed,Key aspp0/BatchNorm/beta not found in checkpoint

[huiyiygy]

System information

What is the top-level directory of the model you are using:/home/ygy/models_2/research/deeplab
OS Platform and Distribution : Ubuntu 16.04
TensorFlow installed from: pip
TensorFlow version: 1.10.1
CUDA/cuDNN version: 9.0
GPU model and memory: GTX1080Ti 11GB

Describe the problem

I tested train.py and got the checkpoint successfully, but as I tested the eval/vis.py, I got the same error:
Key aspp0/BatchNorm/beta not found in checkpoint
[[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]
[[Node: save/RestoreV2/_2531 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_916_save/RestoreV2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

Neither xception71_dpc_cityscapes_trainfine nor xception71_dpc_cityscapes_trainval could run successfully.
What did I do wrong?

Source code / logs

Training code:
python deeplab/train.py
--logtostderr
--training_number_of_steps=1000
--train_split="train"
--model_variant="xception_71"
--atrous_rates=6
--atrous_rates=12
--atrous_rates=18
--output_stride=16
--decoder_output_stride=4
--train_crop_size=769
--train_crop_size=769
--train_batch_size=1
--fine_tune_batch_norm=False
--dataset="cityscapes"
--dense_prediction_cell_json="deeplab/core/dense_prediction_cell_branch5_top1_cityscapes.json"
--tf_initial_checkpoint='deeplab/backbone/train_fine/model.ckpt'
--train_logdir='deeplab/datasets/cityscapes/exp/train_on_train_set/train_fine'
--dataset_dir='deeplab/datasets/cityscapes/tfrecord'

eval conde:
python deeplab/eval.py
--logtostderr
--eval_split="val"
--model_variant="xception_71"
--atrous_rates=6
--atrous_rates=12
--atrous_rates=18
--output_stride=16
--decoder_output_stride=4
--eval_crop_size=1025
--eval_crop_size=2049
--dataset="cityscapes"
--checkpoint_dir='deeplab/datasets/cityscapes/exp/train_on_train_set/train_fine'
--eval_logdir='deeplab/datasets/cityscapes/exp/train_on_train_set/eval_fine'
--dataset_dir='deeplab/datasets/cityscapes/tfrecord'

vis code:
python deeplab/vis.py
--logtostderr
--vis_split="val"
--model_variant="xception_71"
--atrous_rates=6
--atrous_rates=12
--atrous_rates=18
--output_stride=16
--decoder_output_stride=4
--vis_crop_size=1025
--vis_crop_size=2049
--dataset="cityscapes"
--colormap_type="cityscapes"
--checkpoint_dir='deeplab/datasets/cityscapes/exp/train_on_train_set/train_fine'
--vis_logdir='deeplab/datasets/cityscapes/exp/train_on_train_set/vis_fine'
--dataset_dir='deeplab/datasets/cityscapes/tfrecord'
--also_save_raw_predictions=True

[tensorflowbutler]

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Have I written custom code
Bazel version
Exact command to reproduce

[huiyiygy]

Have I written custom code: NO
Bazel version: N/A
Exact command to reproduce: I have displayed my used codes above

[aquariusjay]

Hi huiyiygy,

Please also include the flag
--dense_prediction_cell_json="deeplab/core/dense_prediction_cell_branch5_top1_cityscapes.json"
in eval.py and vis.py.

Cheers,

[huiyiygy]

Thank you very much!