Models: Object detection: PR-Curves in Tensorboard

[blaskowitz100]

Hello,
is it planned to plot PR-Curves as separate summaries in Tensorboard? When not, this would be a nice feature :)

[bignamehyp]

We currently don't have anybody working on this. It would be great if you could help us by working on this and submitting a PR. Let us know if you need further clarification. Thanks!

[foreverYoungGitHub]

Hi, @bignamehyp. Currently, I'm trying to figure out this problem.

I found that in ./utils/object_detection_evaluation.py has already calculated the PR value by the compute_precision_recall function in ./utils/metrics.py.

The detail of how to evaluate the detector is contained in ClassObjectDetectionEvaluation and Funcevaluate(). It returns the pr curve to the interface class ObjectDetectionEvaluator Func evaluate(). So if you do not need the TP,FP values, Func evaluate() can be change like this:

./utils/object_detection_evaluation.py class ObjectDetectionEvaluator Func evaluate()

  def evaluate(self):
    """Compute evaluation result.

    Returns:
      A dictionary of metrics with the following fields -

      1. summary_metrics:
        'Precision/mAP@<matching_iou_threshold>IOU': mean average precision at
        the specified IOU threshold.

      2. per_category_ap: category specific results with keys of the form
        'PerformanceByCategory/mAP@<matching_iou_threshold>IOU/category'.
    """
    (per_class_ap, mean_ap, precisions_per_class, recalls_per_class, per_class_corloc, mean_corloc) = (
        self._evaluation.evaluate())
    pascal_metrics = {
        self._metric_prefix +
        'Precision/mAP@{}IOU'.format(self._matching_iou_threshold):
            mean_ap
    }
    pr_value = {}
    if self._evaluate_corlocs:
      pascal_metrics[self._metric_prefix + 'Precision/meanCorLoc@{}IOU'.format(
          self._matching_iou_threshold)] = mean_corloc
    category_index = label_map_util.create_category_index(self._categories)
    for idx in range(per_class_ap.size):
      if idx + self._label_id_offset in category_index:
        display_name = (
            self._metric_prefix + 'PerformanceByCategory/AP@{}IOU/{}'.format(
                self._matching_iou_threshold,
                category_index[idx + self._label_id_offset]['name']))
        pascal_metrics[display_name] = per_class_ap[idx]


        # PR curve
        display_name = (
            'PR_curve@{}'.format(category_index[idx + self._label_id_offset]['name']))
        pr_value[display_name] = {'precisions': precisions_per_class[idx], 'recalls': recalls_per_class[idx]}

        # Optionally add CorLoc metrics.classes
        if self._evaluate_corlocs:
          display_name = (
              self._metric_prefix + 'PerformanceByCategory/CorLoc@{}IOU/{}'
              .format(self._matching_iou_threshold,
                      category_index[idx + self._label_id_offset]['name']))
          pascal_metrics[display_name] = per_class_corloc[idx]

    return pascal_metrics, pr_value

Notice that the precisions_per_class and recalls_per_class returns the pr value per class. And we return all the PR_value with in a new object pr_value.

Then we change the interface of func _run_checkpoint_once and func repeated_checkpoint_run in ./eval_util.py as following:

./eval_util.py func _run_checkpoint_once

for evaluator in evaluators:
        #metrics = evaluator.evaluate()
        metrics, pr_value = evaluator.evaluate()

....

#return (global_step, all_evaluator_metrics)
return (global_step, all_evaluator_metrics, pr_value)

#### ./eval_util.py func repeated_checkpoint_run

        # global_step, metrics = _run_checkpoint_once_change(tensor_dict, evaluators,
        #                                             batch_processor,
        #                                             last_evaluated_model_path,
        #                                             variables_to_restore,
        #                                             None, num_batches,
        #                                             master, save_graph,
        #                                             save_graph_dir)
        global_step, metrics, pr_value = _run_checkpoint_once_change(tensor_dict, evaluators,
                                                    batch_processor,
                                                    last_evaluated_model_path,
                                                    variables_to_restore,
                                                    None, num_batches,
                                                    master, save_graph,
                                                    save_graph_dir)

Then, change the function write_metrics() in file ./eval_util.py to write the pr_curve.

./eval_util.py func write_pr_curve()

def write_metrics(metrics, global_step, summary_dir, pr_value=None):
  """Write metrics to a summary directory.

  Args:
    metrics: A dictionary containing metric names and values.
    global_step: Global step at which the metrics are computed.
    summary_dir: Directory to write tensorflow summaries to.
    pr_value: A dictionary containing pr_value names and values.
  """
  logging.info('Writing metrics to tf summary.')
  summary_writer = tf.summary.FileWriter(summary_dir)
  for key in sorted(metrics):
    summary = tf.Summary(value=[
        tf.Summary.Value(tag=key, simple_value=metrics[key]),
    ])
    summary_writer.add_summary(summary, global_step)
    logging.info('%s: %f', key, metrics[key])
  if pr_value is not None:
    for key in sorted(pr_value):
      # this part is used to control the num of the points in the PR curve, 
      # since it always generates much more than 10,000 points in the pr curve, which is not necessary to plot all these point into the PR curve.
      num_thresholds = min(500, len(pr_value[key]['precisions']))
      if num_thresholds != len(pr_value[key]['precisions']):
        gap = len(pr_value[key]['precisions']) / num_thresholds
        pr_value[key]['precisions'] = np.append(pr_value[key]['precisions'][::gap], pr_value[key]['precisions'][-1])
        pr_value[key]['recalls'] = np.append(pr_value[key]['recalls'][::gap], pr_value[key]['recalls'][-1])
        num_thresholds = len(pr_value[key]['precisions'])
      # the pr_curve_raw_data_pb() needs the a ascending precisions array and a descending recalls array
      pr_value[key]['precisions'].sort()
      pr_value[key]['recalls'][::-1].sort()
      #write pr curve
      summary = summary_lib.pr_curve_raw_data_pb(
        name=key,
        true_positive_counts=-np.ones(num_thresholds),
        false_positive_counts=-np.ones(num_thresholds),
        true_negative_counts=-np.ones(num_thresholds),
        false_negative_counts=-np.ones(num_thresholds),
        precision=pr_value[key]['precisions'],
        recall=pr_value[key]['recalls'],
        num_thresholds=num_thresholds
        )
      summary_writer.add_summary(summary, global_step)
      logging.info('%s: %f', key, pr_value[key])
  summary_writer.close()
  logging.info('Metrics written to tf summary.')

and use this function in func repeated_checkpoint_run
#### ./eval_util.py func repeated_checkpoint_run

#write_metrics(metrics, global_step, summary_dir)
write_metrics(metrics=metrics, pr_value=pr_value, global_step=global_step, summary_dir=summary_dir)

The re-run the eval.py, you can get some result like this:

Cheers,

Let me know if there are some bugs.

[forvd]

Hi,@foreverYoungGitHub. I got an error: NameError: name 'summary_lib' is not defined in ./eval_util.py func write_pr_curve(). Seems like you forgot some of your codes.

[foreverYoungGitHub]

@forvd, Yes, you are right, I forget to mention that, it needs to import tensorboard to make the pr curve works fine, which is mentioned in PR curve.

So, in this case, you need to add from tensorboard import summary as summary_lib in the beginning of ./eval_util.py.

Besides, the tensorboard should be the newest one to make the pr_curve function work.

If it is necessary, I could make a pull request for this function.

[PythonImageDeveloper]

Hi ,
i faced with this error when i run eval.py:

INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
Traceback (most recent call last):
File "eval.py", line 142, in
tf.app.run()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 124, in run
_sys.exit(main(argv))
File "eval.py", line 138, in main
FLAGS.checkpoint_dir, FLAGS.eval_dir)
File "/home/mm/models/research/object_detection/evaluator.py", line 240, in evaluate
save_graph_dir=(eval_dir if eval_config.save_graph else ''))
File "/home/mm/models/research/object_detection/eval_util.py", line 448, in repeated_checkpoint_run
return metrics
UnboundLocalError: local variable 'metrics' referenced before assignment

[foreverYoungGitHub]

@zeynali, This is caused by the metrics is not define first.

If in repeated_checkpoint_run it should be defined as following:

global_step, metrics, pr_value = _run_checkpoint_once_change(tensor_dict, evaluators,
                                                    batch_processor,
                                                    last_evaluated_model_path,
                                                    variables_to_restore,
                                                    None, num_batches,
                                                    master, save_graph,
                                                    save_graph_dir)

[PythonImageDeveloper]

again got error , i once saw that must be initial metrics : in the beginning of repeated_checkpoint_run i i added this :
metrics={}
pr_value={}
global_step={}
i this case , that error don't face but after for a while processing don't give me any value :
mm@mm:~/models/research/object_detection$ python3 eval.py \

    --logtostderr \
    --checkpoint_dir=training_ssd_mobile/model.ckpt-200000 \
    --eval_dir=eval_log\
    --pipeline_config_path=ssd_mobilenet_v1_coco_2017_11_17/ssd_mobilenet_v1_focal_loss_coco.config

INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0

#

Q2: in the evaluator.py why is EVAL_DEFAULT_METRIC = 'pascal_voc_detection_metrics'.
in the folder metrics just only have coco files but don't have any file about pascal.
Should not be change EVAL_DEFAULT_METRIC to 'coco_detection_metrics' ?

###########################################

Q3: when i run eval.py , What will it show? only mAP or open tensorboard window .
you show mAP in the tensorboard window , is this for training or for evaluation ?

#

Q4: my dataset both train and test are 5.6G , in the eval.py this uses test dataset (5.6G) ? in the config file i set inut_of_eval = test.record .
my question this is : are this uses all of test set for get mAP ?

[tanndx17]

@foreverYoungGitHub I got error in object_detection_evalution.py:
File "/data/detection/research/object_detection/utils/object_detection_evaluation.py", line 288, in evaluate
pr_value[display_name] = {'precisions':precisions_per_class[idx],'recalls':recalls_per_class[idx]}
IndexError: list index out of range

I followed all your changes but I got this error. Did I miss anything?

[ywpkwon]

@foreverYoungGitHub Thanks for the useful codes.

I have a question about your codes in write_metrics()

# the pr_curve_raw_data_pb() needs an ascending precisions array and a descending recalls array
pr_value[key]['precisions'].sort() 
pr_value[key]['recalls'][::-1].sort()

Why sorting is here? I think the correct PR curve shouldn't have the artificial sorting. As in the comments, is the sorting just because pr_curve_raw_data_pb() requires such constraint? Why the function has such artificial constraint?

[austinmw]

@foreverYoungGitHub Hi, after following your instructions I got the error:
File "/home/neuro/.local/lib/python3.5/site-packages/tensorflow/models/research/object_detection/eval_util.py", line 65, in write_metrics pr_value[key]['precisions'] = np.append(pr_value[key]['precisions'][::gap], pr_value[key]['precisions'][-1]) TypeError: slice indices must be integers or None or have an __index__ method

it looks like gap can be a floating point which causes type error for slice indices. I wrapped int() around gap and can successfully see PR curves in tensorboard. Thanks!

Any idea how I can additionally get curves for true positive and false negative rates?

[bsaendig]

Thank you for the instructions but I am getting an error.
Any suggestions?

WARNING:root:The following classes have no ground truth examples: [12 26 29 30 45 60 66 68 69 71 80 83 89]
/tensorflow-models/research/object_detection/utils/metrics.py:144: RuntimeWarning: invalid value encountered in true_divide
num_images_correctly_detected_per_class / num_gt_imgs_per_class)
Traceback (most recent call last):
File "/tensorflow-models/research/object_detection/eval.py", line 136, in
tf.app.run()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "/tensorflow-models/research/object_detection/eval.py", line 132, in main
FLAGS.checkpoint_dir, FLAGS.eval_dir)
File "/tensorflow-models/research/object_detection/evaluator.py", line 212, in evaluate
save_graph_dir=(eval_dir if eval_config.save_graph else ''))
File "/tensorflow-models/research/object_detection/eval_util.py", line 417, in repeated_checkpoint_run
save_graph_dir)
File "/tensorflow-models/research/object_detection/eval_util.py", line 309, in _run_checkpoint_once
metrics, pr_value = evaluator.evaluate()
File "/tensorflow-models/research/object_detection/utils/object_detection_evaluation.py", line 251, in evaluate
pr_value[display_name] = {'precisions': precisions_per_class[idx], 'recalls': recalls_per_class[idx]}
IndexError: list index out of range

[YNVoon]

@bsaendig I faced the same error as well, I have downgraded my numpy to version 1.11.0, and it works fine now.

[frankdeepl]

Hi @foreverYoungGitHub, do you know how could we get the mapped value by tensorboard or coding?

[acg93-pixel]

hi, I'm using Tensorflow object detection new API, I've implemented your code and I don't get any errors. But when I run Tensorboard, I don't get any PR curve data recognized. Is this maybe incompatible with the new API?
thanks

[BernardinD]

For everyone have the index out of range error, the 'evaluate' function at the end of 'object_detection_evaluation.py' appends to the precision and recall list based on number of items in your labelmap and your items are probably not id'ed to correspond with that.

My advise would be to add a new mapping that accounts for this appending and use that to index 'precisions_per_class' and 'recall_per_class'

[acg93-pixel]

any suggestions how to do that? I've been using mscoco label map

[alokranjan007]

@foreverYoungGitHub Hello,
Thank you for sharing your solution. I tried your steps and made necessary changes in the scripts. In fact, I used the pulled request (git hub) scripts for the evaluation. But the tensorboard is not showing any PR curve tab like the scalar and others. When tried to select from the drop down list then it is saying that "No precision Recall data was found" and do not show the tab what you have shown for the PR curve.

Could you please help me to understand the problem.

Thank you

[austinmw]

@alokranjan007 I have not tried this in quite a while, but it's possible the posted workaround is incompatible with your version of tensorflow

[rkolaf]

when following the above instructions I got the TP = -1 TN= -1,FP=-1 , AND FN = -1.
whether this is correct?? I don't know much about these as I'm just a beginner.

[drisyack]

@foreverYoungGitHub Thank you for sharing your solution. I made necessary changes and executed without error after that tensorboard is appeared. But the tensorboard is not showing any PR curve tab like the scalar and others. When tried to select from the drop down list then it is saying that "No precision Recall data was found" and do not show the tab what you have shown for the PR curve.what is the reason for this. will i have to do more correction in the code?.

Could you please please help me to understand the problem.

Thank you

[aysark]

Yeah, the PR that @foreverYoungGitHub was not merged and now doesn't work with latest code it seems.

[austinmw]

Yup I tried it too and couldn't get it to work even after making some obvious fixes.

[Rakiba291]

@foreverYoungGitHub Hi, after following your instructions I got the error:
File "/home/neuro/.local/lib/python3.5/site-packages/tensorflow/models/research/object_detection/eval_util.py", line 65, in write_metrics pr_value[key]['precisions'] = np.append(pr_value[key]['precisions'][::gap], pr_value[key]['precisions'][-1]) TypeError: slice indices must be integers or None or have an __index__ method

it looks like gap can be a floating point which causes type error for slice indices. I wrapped int() around gap and can successfully see PR curves in tensorboard. Thanks!

Any idea how I can additionally get curves for true positive and false negative rates?

@austinmw Hi! can you just share how you got the PR curves in tensorboard? how did you solve the problem of the one you mentioned? is it pr_value[key]['precisions'] = np.append(pr_value[key]['precisions'][::int(gap)], pr_value[key]['precisions'][-1]) ?
I tried to plot by matplotlib in object_detection_evaluation.py after compute.metrics(precision, recall) but I saw no graph plots. I have tried looking for all possible solutions but I could not plot the graphs. So, if you could help to know , would be grateful to you. The tensorflow version I am using in v1.12 with 3.6 python interpreter. Thanks!

[baronpalacios]

@foreverYoungGitHub hi brot, thank you for given that solution, it worked for me.
In other hand do you know how to get Precision-Recall curves at various IoU thresholds?

[hanifjunos]

@baronpalacios how did you able to run the eval. I get an issue of:
metrics, pr_value = evaluator.evaluate()
ValueError: too many values to unpack (expected 2)

[WANGSHUAISWU]

@hanifjunos Hi, I meet the same problem with you. Have you solved it? Could you please give me some advice?

[hanifjunos]

@hanifjunos Hi, I meet the same problem with you. Have you solved it? Could you please give me some advice?

I still couldn't solved it.

[hanifjunos]

@WANGSHUAISWU have u solved this issue?

@baronpalacios how did you able to run the eval. I get an issue of:
metrics, pr_value = evaluator.evaluate()
ValueError: too many values to unpack (expected 2)

[jhonjam]

@baronpalacios, ¿cómo pudo ejecutar la evaluación? Tengo un problema de:
metrics, pr_value = evaluator.evaluate ()
ValueError: demasiados valores para desempaquetar (esperado 2)

hello i have the same problem, you could solve it. Help MEEE

[BeautyMess]

@hanifjunos @WANGSHUAISWU @jhonjam
Please check whether you are using COCO metrics in the config file like metrics_set: 'coco_detection_metrics'.
I think the function provided here only supports the pascal metrics.

[tw0226]

I want to run this function in model_main.py.
anybody can tell me the process?? then I will be able to customize that.
how can I try like @foreverYoungGitHub ??
otherwise, somebody can tell me which python file runs that process??
help me please

[aguspray001]

when following the above instructions I got the TP = -1 TN= -1,FP=-1 , AND FN = -1.
whether this is correct?? I don't know much about these as I'm just a beginner.

do you found the solving?

[tensorflow-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[weenkiejinky]

C:\Users\Admin\Anaconda3\envs\tf_gpu\notebook\models\research\object_detection\utils\object_detection_evaluation.py:1195: UserWarning: Matplotlib is currently using agg, which is a non-GUI backend, so cannot show the figure.
plt.show()
Traceback (most recent call last):
File "research/object_detection/eval.py", line 142, in
tf.app.run()
File "C:\Users\Admin\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\platform\app.py", line 125, in run
_sys.exit(main(argv))
File "C:\Users\Admin\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\util\deprecation.py", line 306, in new_func
return func(args, *kwargs)
File "research/object_detection/eval.py", line 138, in main
graph_hook_fn=graph_rewriter_fn)
File "C:\Users\Admin\Anaconda3\envs\tf_gpu\notebook\models\research\object_detection\legacy\evaluator.py", line 296, in evaluate
eval_export_path=eval_config.export_path)
File "C:\Users\Admin\Anaconda3\envs\tf_gpu\notebook\models\research\object_detection\eval_util.py", line 536, in repeated_checkpoint_run
write_metrics(metrics=metrics, pr_value=pr_value, global_step=global_step, summary_dir=summary_dir)
File "C:\Users\Admin\Anaconda3\envs\tf_gpu\notebook\models\research\object_detection\eval_util.py", line 80, in write_metrics
logging.info('Writing metrics to tf summary.')
NameError: name 'logging' is not defined

...I followed also the steps but this is the error

[tensorflow-butler[bot]]

Closing as stale. Please reopen if you'd like to work on this further.

[diego0718]

Hi all,

metrics, pr_value = evaluator.evaluate()

ValueError: too many values to unpack (expected 2)

i try to get rid of of this problem.

def evaluate(self):
"""Compute evaluation result.

Returns:
  A dictionary of metrics with the following fields -

  1. summary_metrics:
    'Precision/mAP@<matching_iou_threshold>IOU': mean average precision at
    the specified IOU threshold.

  2. per_category_ap: category specific results with keys of the form
    'PerformanceByCategory/mAP@<matching_iou_threshold>IOU/category'.
"""
print("Entre a evaluate")
(per_class_ap, mean_ap, precisions_per_class, recalls_per_class, per_class_corloc, mean_corloc) = (
    self._evaluation.evaluate())
pascal_metrics = {
    self._metric_prefix +
    'Precision/mAP@{}IOU'.format(self._matching_iou_threshold):
        mean_ap
}
pr_value = {}
if self._evaluate_corlocs:
  pascal_metrics[self._metric_prefix + 'Precision/meanCorLoc@{}IOU'.format(
      self._matching_iou_threshold)] = mean_corloc
category_index = label_map_util.create_category_index(self._categories)
for idx in range(per_class_ap.size):
  if idx + self._label_id_offset in category_index:
    display_name = (
        self._metric_prefix + 'PerformanceByCategory/AP@{}IOU/{}'.format(
            self._matching_iou_threshold,
            category_index[idx + self._label_id_offset]['name']))
    pascal_metrics[display_name] = per_class_ap[idx]


    # PR curve
    display_name = (
        'PR_curve@{}'.format(category_index[idx + self._label_id_offset]['name']))
    pr_value[display_name] = {'precisions': precisions_per_class[idx], 'recalls': recalls_per_class[idx]}

    # Optionally add CorLoc metrics.classes
    if self._evaluate_corlocs:
      display_name = (
          self._metric_prefix + 'PerformanceByCategory/CorLoc@{}IOU/{}'
          .format(self._matching_iou_threshold,
                  category_index[idx + self._label_id_offset]['name']))
      pascal_metrics[display_name] = per_class_corloc[idx]

return pascal_metrics, pr_value

what i am doing wrong?