Openpose: Standalone Hand Keypoint Detection - hand rectangle format

[pjain03]

Issue Summary

I am trying to build a standalone hand keypoint detector. As stated in the docs here and explained by @gineshidalgo99 in other issues, I used only the HandExtractor package from the C++ API with the output of my hand detector which places squared bounding boxes around hands.
I read in the code that op::Rectangle is supposed to be like cv::Rect. Does this mean we are supposed to pass in coordinates for the top left part of the image along with the width and height? (Which is what I did)

Executed Command (if any)

I am more than happy to post my code here in case further clarification is required, but in order to reduce clutter as of right now, I will keep it concise.
I used the C++ API and looked through the code on #343. The following are the packages I used:

OpenPose Input (if any)

I passed the following squared bounding box coordinate information [xmin, ymin, width, height]:
[23.167587280273438, 96.12846, 830.98724, 830.98724]

OpenPose Output (if any)

The rendering worked just fine but we weren't able to locate the hand keypoints.

Type of Issue

• Execution error
• Help wanted
• Question

Your System Configuration

• OpenPose version: 1.3.0

[ortegatron]

Hi, I made it work. I modifed the Python wrapper and added a new method for the hand detection.
You can see the C++ method here: https://gist.github.com/ortegaton/d2b2fa620d00823866343ceaab5824b8

I used that method from Python, for the image you gave works good with the hand box [23.167587280273438, 96.12846, 830.98724, 830.98724].
If you have any specific doubt I can answer.
Regards,
Marcelo

[pjain03]

Hey @ortegaton! Thank you for your response!
I do have a couple of questions about your code.

1. Where does it go? I put it with the other OP_EXPORTs.
2. What is handsOutput? It seems to be undefined...?
3. What does forward_hands refer to?

[ortegatron]

Hi, I have finished it now, I will upload to a repository in the days coming and will let you know here.

[pjain03]

Awesome! Thank you!
I was able to create my own version of your code as well and it worked beautifully! If needed, I will create a standalone version of the hand keypoint extractor as well and post it here!

[ortegatron]

Sweet! I uploaded it here: https://github.com/ortegaton/hand_standalone
Have a look pls and check if you notice something wrong.
Unfortunately it's not working 100% good yet, doesn't give good outputs for a lot of cases.

[pjain03]

@ortegaton this looks really good! I'll try to use it/emulate it and if I find something I'll let you know!
Just to get me started, in what cases does your project not give good outputs?
Thank you so much once again!

[pjain03]

This issue is resolved, thanks to @ortegaton. We can talk more here but I'm closing it up so that it's better organized.

[yavana]

@ortegaton this looks really good! I'll try to use it/emulate it and if I find something I'll let you know!
Just to get me started, in what cases does your project not give good outputs?
Thank you so much once again!

[ 3%] Built target openpose_caffe
[ 48%] Built target openpose
[ 53%] Built target openpose_core
[ 53%] Built target openpose_thread
[ 58%] Built target openpose_filestream
[ 62%] Built target openpose_producer
[ 65%] Built target openpose_utilities
[ 70%] Built target openpose_pose
[ 73%] Built target openpose_hand
[ 75%] Built target openpose_tracking
[ 78%] Built target openpose_gui
[ 82%] Built target openpose_face
[ 85%] Built target openpose_wrapper
[ 89%] Built target openpose_net
[ 91%] Built target openpose_3d
[ 92%] Built target openpose_gpu
[ 93%] Built target openpose_calibration
[ 94%] Built target calibration.bin
[ 94%] Built target openpose.bin
[ 94%] Built target 1_custom_post_processing.bin
[ 94%] Built target 1_extract_from_image.bin
[ 95%] Built target 2_extract_pose_or_heatmat_from_image.bin
[ 96%] Built target 1_openpose_read_and_display.bin
[ 96%] Built target 2_user_processing_function.bin
[ 97%] Built target 3_user_input_processing_and_output.bin
[ 98%] Built target 4_user_input_processing_output_and_datum.bin
[ 98%] Built target 1_user_synchronous_postprocessing.bin
[ 98%] Built target 2_user_synchronous_input.bin
[ 98%] Built target 3_user_synchronous_output.bin
[ 98%] Built target 4_user_synchronous_all.bin
[ 98%] Built target 5_user_asynchronous.bin
[ 99%] Built target 6_user_asynchronous_output.bin
[ 99%] Built target handFromJsonTest.bin
[100%] Built target resizeTest.bin
Scanning dependencies of target _openpose
[100%] Building CXX object python/openpose/CMakeFiles/_openpose.dir/_openpose.cpp.o
/home/openpose/python/openpose/_openpose.cpp: In member function ‘void OpenPose::poseFromHeatmap(const cv::Mat&, std::vector > >&, op::Array&, cv::Mat&, std::vector >&)’:
/home/openpose/python/openpose/_openpose.cpp:232:35: error: ‘caffeHmPtr’ was not declared in this scope
caffeNetOutputBlob.emplace_back(caffeHmPtr);
^
/home/openpose/python/openpose/_openpose.cpp:289:3: error: expected ‘)’ before ‘bodyPartConnectorCaffe’
bodyPartConnectorCaffe->setInterThreshold((float)poseExtractorCaffe->get(op::PoseProperty::ConnectInterThreshold));
^
make[2]: * [python/openpose/CMakeFiles/_openpose.dir/_openpose.cpp.o] Error 1
make[1]: [python/openpose/CMakeFiles/_openpose.dir/all] Error 2
make: ** [all] Error 2

[LeeKyungwook]

@yavana
I used this code. Lots of cases are fine, but we have to consider how we set detection box.
"Hand Keypoint Detection in Single Images using Multiview Bootstrapping"
This paper gives you a lot of information how to set hand detection box. If you set correctly, then you can
get good results.

[MLsmaller]

Hi,I am also planning to write the code of a hand detector myself, and include the hand-keypoint-detector of openpose. However, I am a little confused. Could you please give me some contact information?Thank you very much.

[MLsmaller]

Hi, I made it work. I modifed the Python wrapper and added a new method for the hand detection.
You can see the C++ method here: https://gist.github.com/ortegaton/d2b2fa620d00823866343ceaab5824b8

I used that method from Python, for the image you gave works good with the hand box [23.167587280273438, 96.12846, 830.98724, 830.98724].
If you have any specific doubt I can answer.
Regards,
Marce

Hi, I made it work. I modifed the Python wrapper and added a new method for the hand detection.
You can see the C++ method here: https://gist.github.com/ortegaton/d2b2fa620d00823866343ceaab5824b8

I used that method from Python, for the image you gave works good with the hand box [23.167587280273438, 96.12846, 830.98724, 830.98724].
If you have any specific doubt I can answer.
Regards,
Marcelo
Hi,I am also planning to write the code of a hand detector myself, and include the hand-keypoint-detector of openpose. However, I am a little confused. Could you please give me some contact information?Thank you very much.

[Hzzone]

openpose detects hand by the result of body pose estimation, please refer to the code of handDetector.cpp.
In the paper, it states as:

This is an important detail: to use the keypoint detector in any practical situation, 
we need a way to generate this bounding box. 
We directly use the body pose estimation models from [29] and [4], 
and use the wrist and elbow position to approximate the hand location, 
assuming the hand extends 0.15 times the length of the forearm in the same direction.

If anybody wants a pure python wrapper, please refer to my pytorch implementation of openpose, maybe it helps you to implement a standalone hand keypoint detector.

[ortegatron]

I updated the code at the repository to use the last OpenPose version and integrate with a hand bounding box detector.
https://github.com/ortegaton/hand_standalone