Models: Double tensorflow INFO: when using a slim pretrained model

[Lizzard93]

System information

• What is the top-level directory of the model you are using: object-detection
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
• TensorFlow installed from (source or binary): source and binary
• TensorFlow version (use command below): 1.4.1
• Bazel version (if compiling from source):
• CUDA/cuDNN version: 8.0
• GPU model and memory: gtx 980
• Exact command to reproduce:

Describe the problem

When i restore a slim pretrained model ( like inception-v2) all the tensorflow mwssage info are duplicated.
Es:
INFO:tensorflow:Restoring parameters from /home/simone/Documents/inception_v2.ckpt
INFO:tensorflow:Restoring parameters from /home/simone/Documents/inception_v2.ckpt
INFO:tensorflow:Processed 0 images...
INFO:tensorflow:Processed 0 images...

The bug seems to be in the function get_variables_available_in_checkpoint in https://github.com/tensorflow/models/blob/master/research/object_detection/utils/variables_helper.py

Source code / logs

i solved this issue commenting this part of the function :

       # else:
        #     logging.warning('Variable [%s] not available in checkpoint',
        #                     variable_name)

def get_variables_available_in_checkpoint(variables, checkpoint_path):
    """Returns the subset of variables available in the checkpoint.

    Inspects given checkpoint and returns the subset of variables that are
    available in it.

    TODO: force input and output to be a dictionary.

    Args:
      variables: a list or dictionary of variables to find in checkpoint.
      checkpoint_path: path to the checkpoint to restore variables from.

    Returns:
      A list or dictionary of variables.
    Raises:
      ValueError: if `variables` is not a list or dict.
    """
    if isinstance(variables, list):
        variable_names_map = {variable.op.name: variable for variable in variables}
    elif isinstance(variables, dict):
        variable_names_map = variables
    else:
        raise ValueError('`variables` is expected to be a list or dict.')
    ckpt_reader = tf.train.NewCheckpointReader(checkpoint_path)
    ckpt_vars = ckpt_reader.get_variable_to_shape_map().keys()
    vars_in_ckpt = {}
    for variable_name, variable in sorted(variable_names_map.items()):
        if variable_name in ckpt_vars:
            vars_in_ckpt[variable_name] = variable
        # else:
        #     logging.warning('Variable [%s] not available in checkpoint',
        #                     variable_name)
    if isinstance(variables, list):
        return vars_in_ckpt.values()
    return vars_in_ckpt

[shivaniag]

Could you share the exact code you are trying to run? I am not able to reproduce with the details you have provided. Looping @tombstone for object-detection.

[Lizzard93]

python train.py --logtostderr --pipeline_config_path=./configs/faster_rcnn_inception_v2.config --train_dir=./checkpoints/checkpoint_inception_v2_imagenet

where faster_rcnn_inception_v2.config is:

model {
  faster_rcnn {
    num_classes: 546
    image_resizer {
      keep_aspect_ratio_resizer {
        min_dimension: 600
        max_dimension: 1024
      }
    }
    feature_extractor {
      type: "faster_rcnn_inception_v2"
      first_stage_features_stride: 16
    }
    first_stage_anchor_generator {
      grid_anchor_generator {
        height_stride: 16
        width_stride: 16
        scales: 0.25
        scales: 0.5
        scales: 1.0
        scales: 2.0
        aspect_ratios: 0.5
        aspect_ratios: 1.0
        aspect_ratios: 2.0
      }
    }
    first_stage_box_predictor_conv_hyperparams {
      op: CONV
      regularizer {
        l2_regularizer {
          weight: 0.0
        }
      }
      initializer {
        truncated_normal_initializer {
          stddev: 0.00999999977648
        }
      }
    }
    first_stage_nms_score_threshold: 0.0
    first_stage_nms_iou_threshold: 0.699999988079
    first_stage_max_proposals: 300
    first_stage_localization_loss_weight: 2.0
    first_stage_objectness_loss_weight: 1.0
    initial_crop_size: 14
    maxpool_kernel_size: 2
    maxpool_stride: 2
    second_stage_box_predictor {
      mask_rcnn_box_predictor {
        fc_hyperparams {
          op: FC
          regularizer {
            l2_regularizer {
              weight: 0.0
            }
          }
          initializer {
            variance_scaling_initializer {
              factor: 1.0
              uniform: true
              mode: FAN_AVG
            }
          }
        }
        use_dropout: false
        dropout_keep_probability: 1.0
      }
    }
    second_stage_post_processing {
      batch_non_max_suppression {
        score_threshold: 0.0
        iou_threshold: 0.600000023842
        max_detections_per_class: 100
        max_total_detections: 300
      }
      score_converter: SOFTMAX
    }
    second_stage_localization_loss_weight: 2.0
    second_stage_classification_loss_weight: 1.0
  }
}
train_config {
  batch_size: 1
  data_augmentation_options {
    random_horizontal_flip {
    }
  }
  optimizer {
    momentum_optimizer {
      learning_rate {
        manual_step_learning_rate {
          initial_learning_rate: 0.000199999994948
          schedule {
            step: 0
            learning_rate: 0.000199999994948
          }
          schedule {
            step: 900000
            learning_rate: 1.99999994948e-05
          }
          schedule {
            step: 1200000
            learning_rate: 1.99999999495e-06
          }
        }
      }
      momentum_optimizer_value: 0.899999976158
    }
    use_moving_average: false
  }
  gradient_clipping_by_norm: 10.0
  fine_tune_checkpoint: "./pretrained_models/inception_v2_imagenet_2016_08_28/inception_v2.ckpt"
  from_detection_checkpoint: false
  num_steps: 8000000
}
train_input_reader {
  label_map_path: "./data/oid_bbox_trainable_label_map.pbtxt"
  tf_record_input_reader {
    input_path: "./tf_records/train.tfrecord"
  }
}
eval_config: {
  metrics_set: "open_images_metrics"
  num_examples: 1000
}
eval_input_reader {
  label_map_path: "./data/oid_bbox_trainable_label_map.pbtxt"
  shuffle: false
  num_readers: 1
  tf_record_input_reader {
    input_path: "./tf_records/test.tfrecord"
  }
}

where:
fine_tune_checkpoint: "./pretrained_models/inception_v2_imagenet_2016_08_28/inception_v2.ckpt"
is the pretrained slim model from http://download.tensorflow.org/models/inception_v2_2016_08_28.tar.gz

and from_detection_checkpoint: false because is not a detection model.

./tf_records/train.tfrecord are open images tfrecord created with object-detection script.

[shivaniag]

@jch1 and @tombstone could you take a look here.

[dgriffiths3]

I also get this when running the RCNN models in the object detection api.

[DavoFranco]

I am having the same issue. I commented the same lines in variables_helper.py and still getting double INFO

[fernandorovai]

Same here, the entire logging is duplicated
INFO:tensorflow:Recording summary at step 2814.
INFO:tensorflow:Recording summary at step 2814.
INFO:tensorflow:global step 2815: loss = 0.3653 (4.359 sec/step)
INFO:tensorflow:global step 2815: loss = 0.3653 (4.359 sec/step)

[hitlk]

This issue could be solved by referring to logger duplication.
In variables_helper.py, it uses logging instead of tf.logging.

[apriandito]

sorry, but i dont get it. csn you explain more?
@hitlk

[slackhare]

@dhitology Open variables_helper.py in models/research/object_detection/utils/variables_helper.py and replace all occurrences of logging with tf.logging

[wt-huang]

Closing as this is resolved