Models: Object detection | Keypoints to object detection model

[ALEXKIRNAS]

Hi,
@jch1 , @tombstone , @derekjchow , I would appreciate your help.

ADDING FUNCTIONALITY
I recently start to use SSD with MobileNet feature extractor for solving face detection problem . In term of this task I also need to to detect landmarks (keypoints). I saw that repo contain some files that connected with keypoints ( link ), but I don`t see how to add them to model. So can you describe how to do that?

With the best regards,
Alexander

[Sheepybloke2-0]

Any update on this? It seems that the visualizations there and the standard field is there, but I can't seem to get the keypoints from the graph (I'm using ssd_mobilenet_v1_coco). Currently failing when I'm calling keypoints = graph.get_tensor_by_name('detection_keypoints:0')

[wcastil]

@pkulzc I saw you had done most of the work on adding keypoints to the detection framework. Any chance you can give an update on if/when that functionality will be available?

[dougsouza]

As far as I've got in the code I saw that only MaskRCNNBoxPredictor supports keypoint prediction. How hard it would be to add keypoint prediction to a ConvolutionalBoxPredictor as well? Any thoughts?

[dustindorroh]

I had the same issue, but got reading and writing tfrecords to work. I have links to the source that helped me understand but, here is the tl,dr

Short Answer

1. When creating tfrecords write the keypoints to 'image/object/keypoint/x' and 'image/object/keypoint/y'
2. When reading tfrecords use 'groundtruth_keypoints' key
3. Trying to use higher level inputs.create_train_input_fn will not load keypoints because a few levels down when tf_example_decoder.TfExampleDecoder is called it isn't called with num_keypoints which defaults to 0 and thus never runs decoding.
4. Example of writing keypoints tfrecords . A little messy, but eh not sure if this topic is still alive.

More Detail:

1. We can see this happening in the decoding block
   https://github.com/tensorflow/models/blob/b9ef963d1e84da0bb9c0a6039457c39a699ea149/research/object_detection/data_decoders/tf_example_decoder.py#L285-L293

2. The mapping specifically happens at:
   https://github.com/tensorflow/models/blob/b9ef963d1e84da0bb9c0a6039457c39a699ea149/research/object_detection/data_decoders/tf_example_decoder.py#L290-L293

   Where fields.InputDataFields.groundtruth_keypoints is defined here:
   https://github.com/tensorflow/models/blob/b9ef963d1e84da0bb9c0a6039457c39a699ea149/research/object_detection/core/standard_fields.py#L94

3. Trying to use the higher level dataset_builder.build doesn't pass a num_keypoints and this defaults to 0.
   https://github.com/tensorflow/models/blob/b9ef963d1e84da0bb9c0a6039457c39a699ea149/research/object_detection/builders/dataset_builder.py#L124-L131

   https://github.com/tensorflow/models/blob/b9ef963d1e84da0bb9c0a6039457c39a699ea149/research/object_detection/data_decoders/tf_example_decoder.py#L136-L148

[pharrellyhy]

Hi, @dustindorroh. Very useful information and I'm curious that have you successfully trained keypoints regression with object detection API? Thanks.

[pharrellyhy]

@dustindorroh I can create tfrecord with keypoints as labels and set num_keypoints=1. During training, the keypoint labels are loaded but I don't think the computation graph contains the keypoint output nodes. I can't find where these nodes are added to the graph automatically, so maybe we need to add them manually?

[soldierofhell]

Hi, is anyone working on keypoint functionality now? IMO there's still ~50% of work to do, basicly adding keypoint head to faster rcnn meta architecture. I can do this, but don't want to waste time if the work is in progress or sheduled?

[bhack]

Keypoints API is useful also for new anchor free models:
/cc @see--
https://github.com/see--/keras-centernet
https://github.com/princeton-vl/CornerNet-Lite

[bhack]

A new anchor free model https://github.com/Star-Clouds/CenterFace
We need to support bbox + keypoints API for anchorfree models.

[bhack]

See also https://github.com/tensorflow/hub/issues/424

[bhack]

I added a Keypoints/Heatmaps issue in Tensorflow addons at https://github.com/tensorflow/addons/issues/1366

[kensuke-ueda]

It seems that you should specify "KeypointBoxCoder" in pipeline.config to enable keypoint detection feature.

--------------------------------------- default pipeline.config
box_coder {
faster_rcnn_box_coder {
y_scale: 10.0
x_scale: 10.0
height_scale: 5.0
width_scale: 5.0
}
}
--------------------------------------- you should change like this
box_coder {
keypoint_box_coder {
y_scale: 10.0
x_scale: 10.0
height_scale: 5.0
width_scale: 5.0
}
}

[vesor]

I have been extending the box_coder to add keypoints detection for some time. I have tested on both Faster RCNN and SSD with customized training data which has keypoints labeling. The problem is it is a bit confusing with the keypoints concept in Mask RCNN related code. In Mask RCNN, keypoints are separated from box coding.

[pharrellyhy]

Hi @vesor , nice work! I think what your are saying is totally reasonable. If you don't mind, can you give some sample code on how to achieve that? Thanks!