Openpose: 3D demo (Windows) TDatums size problem in wQueueAssembler.hpp

[Linshiyuan]

Issue Summary

I am trying to run 3D reconstruction using two webcam, each one can work fine on 2D mode.

I modify the WUserInput in 9_synchronous_custom_all.cpp, then I got a Error message says there should be only 1 element per TDatums.

I am confused about it, TDatums is a vector of Datum but the 3D mode requires only 1 per TDatums ? Where should I put the second webcam's Datum ?

Any help with this would be much appreciated!

My custom WUserInput

    std::shared_ptr<std::vector<UserDatum>> workProducer()
    {
        try
        {
        cv::Mat fram1, fram2;
        cap1.read(fram1);
        cap2.read(fram2);

        auto datumsPtr = std::make_shared<std::vector<UserDatum>>();

        datumsPtr->emplace_back();
        auto& datum = datumsPtr->at(0);
        datum.cvInputData = fram1;

        datumsPtr->emplace_back();
        auto& datum2 = datumsPtr->at(1);
        datum2.cvInputData = fram2;

        return datumsPtr;
        }

Error message from wQueueAssembler.hpp

 // Sanity check
 if (tDatums->size() > 1)
        error("This function assumes that WQueueSplitter (inside WDatumProducer)"
        " was applied in the first place, i.e., that there is only 1 element"
        " per TDatums (size = " + std::to_string(tDatums->size()) + ").",
        __LINE__, __FUNCTION__, __FILE__);
auto tDatum = (*tDatums)[0];

Executed Command

--3d --number_people_max 1

Errors

Error: This function assumes that WQueueSplitter (inside WDatumProducer) was applied in the first place, i.e., that there is only 1 element per TDatums (size = 2).

Type of Issue

• Help wanted
• Question

System Configuration

Operating system : Windows 10
Installation mode: VS 2017 Enterprise
CUDA version : CUDA 10.0
cuDNN version: v 7.3
CMake version : 3.13.0
Release or Debug mode? : Release
3-D Reconstruction module added? : Yes
GPU model (nvidia-smi in Ubuntu): Nvidia GTX 1060
Caffe version: Default from OpenPose

[Linshiyuan]

I just delete the wQueueAssembler thread and the data can pass through now, but get a new warning now,
Unusual high re-projection error (averaged over #keypoints) of value 4.392997 pixels, while the average for a good OpenPose detection from 4 cameras is about 2-3 pixels. It might be simply a wrong OpenPose detection. If this message appears very frequently, your calibration parameters might be wrong.
And the 3-D display window is always black.

I am using the dataset 150821_dance2 01_01 , 06_15 , 14_03

The 3-D pose output is 0 all the time.

[MaidouPP]

Similar problem here. Have you figured out a work-around?

The thing is that I don't want to modify the source code of OpenPose (I'm worried that getting rid of wQueueAssembler in thread manager will cause some problems).

Currently I'm using OpenPose service on server side, so there is no way I can plug in the camera directly to the machine running it. Say, I have synchronized my stereo images collected from client side and streaming this pair to the server side, then how should I construct the TDatums, so that I can feed it to op wrapper? I am getting the same error (Error: This function assumes that WQueueSplitter (inside WDatumProducer) was applied in the first place, i.e., that there is only 1 element per TDatums (size = 2).) as @Linshiyuan did.

Currently I am borrowing example from 4_asynchronous_loop_custom_input_and_output, where I push the two synchronized stereo images (with there camera matrix) into TDatums. When I tried to emplaceAndPop the TDatums I created, it will crash and give mentioned error. I am just wondering, if we cannot put two images into TDatums ourselves, then how are we supposed to use our own synchronized images? (As you mentioned in doc/modules/3d_reconstruction_module.md, we should be able to achieve this.) I guess I had some misunderstanding about how to use it. Can anyone please help me?

Thanks in advance!

[djns99]

@MaidouPP @Linshiyuan I had a similar issue. Looking at the wDatumProducer.hpp implementation it appears you have to set subId and subIdMax to and return one at a time. I presume this allows openpose to tell that there are more than one frames to process.

[MaidouPP]

@MaidouPP @Linshiyuan I had a similar issue. Looking at the wDatumProducer.hpp implementation it appears you have to set subId and subIdMax to and return one at a time. I presume this allows openpose to tell that there are more than one frames to process.

Hi! @djns99 Thank you for pointing out the workaround! I actually also managed to solve it that way. What I did is to create two datums, each containing one image and corresponding camera matrix, with subid and subidmax correctly set. But now my problem is that it always throws error like:
"Unusual high re-projection error (averaged over #keypoints) of value -nan pixels, while the
average for a good OpenPose detection from 4 cameras is about 2-3 pixels. It might be simp
ly a wrong OpenPose detection. If this message appears very frequently, your calibration pa
rameters might be wrong."

I tested the OpenPose for 2d, and the detection is correct. So it must be the calibration parameter issue. But the thing is that I am using some online datasets, like KITTI stereo, to do the test, so the calibration should be fine. In my understanding, the cameraMatrix is intrinsics * extrinsics, which is also known as projection matrix (3x4), and that's what I feed into cameraMatrix field. But still, the error is always there no matter what image and data I am using. Do you have similar experience? Or can you shed some light on how to feed into calibration parameters? Thank you!

[djns99]

@MaidouPP I did have the same issue when running with multiple cameras, however I presumed it was due to an error in my setup, since I was using two cameras with different resolutions (what I happened to have lying around). I don't really know much about the calibration process so unfortunately I can't help much there. I might mention that I also had a reprojection error of ~200 for some cases, but usually it was -nan as with you.
In the new year I was going to attempt it again with identical cameras so I will let you know if I have any luck with that.

[MaidouPP]

@MaidouPP I did have the same issue when running with multiple cameras, however I presumed it was due to an error in my setup, since I was using two cameras with different resolutions (what I happened to have lying around). I don't really know much about the calibration process so unfortunately I can't help much there. I might mention that I also had a reprojection error of ~200 for some cases, but usually it was -nan as with you.
In the new year I was going to attempt it again with identical cameras so I will let you know if I have any luck with that.

@djns99 I think using two different cameras with different resolutions is totally ok, as long as we feed the correct camera matrix with datums. I checked the triangulation code in openpose, I think this process is simply handled by constructing a SVD problem, and camera matrices are used for different image points (if on different images). Anyway, thank you in advance! I will also update on this thread if I get any progress (next year)!

[MaidouPP]

@MaidouPP I did have the same issue when running with multiple cameras, however I presumed it was due to an error in my setup, since I was using two cameras with different resolutions (what I happened to have lying around). I don't really know much about the calibration process so unfortunately I can't help much there. I might mention that I also had a reprojection error of ~200 for some cases, but usually it was -nan as with you.
In the new year I was going to attempt it again with identical cameras so I will let you know if I have any luck with that.

I managed to solve the issue by pulling newest update. It's fixed in #998

[pritorius]

@MaidouPP @Linshiyuan I had a similar issue. Looking at the wDatumProducer.hpp implementation it appears you have to set subId and subIdMax to and return one at a time. I presume this allows openpose to tell that there are more than one frames to process.

Hi! @djns99 Thank you for pointing out the workaround! I actually also managed to solve it that way. What I did is to create two datums, each containing one image and corresponding camera matrix, with subid and subidmax correctly set. But now my problem is that it always throws error like:
"Unusual high re-projection error (averaged over #keypoints) of value -nan pixels, while the
average for a good OpenPose detection from 4 cameras is about 2-3 pixels. It might be simp
ly a wrong OpenPose detection. If this message appears very frequently, your calibration pa
rameters might be wrong."

I tested the OpenPose for 2d, and the detection is correct. So it must be the calibration parameter issue. But the thing is that I am using some online datasets, like KITTI stereo, to do the test, so the calibration should be fine. In my understanding, the cameraMatrix is intrinsics * extrinsics, which is also known as projection matrix (3x4), and that's what I feed into cameraMatrix field. But still, the error is always there no matter what image and data I am using. Do you have similar experience? Or can you shed some light on how to feed into calibration parameters? Thank you!

hi @MaidouPP, is it possible to illustrate what you actually did?
Thanks.

[MaidouPP]

@MaidouPP @Linshiyuan I had a similar issue. Looking at the wDatumProducer.hpp implementation it appears you have to set subId and subIdMax to and return one at a time. I presume this allows openpose to tell that there are more than one frames to process.

Hi! @djns99 Thank you for pointing out the workaround! I actually also managed to solve it that way. What I did is to create two datums, each containing one image and corresponding camera matrix, with subid and subidmax correctly set. But now my problem is that it always throws error like:
"Unusual high re-projection error (averaged over #keypoints) of value -nan pixels, while the
average for a good OpenPose detection from 4 cameras is about 2-3 pixels. It might be simp
ly a wrong OpenPose detection. If this message appears very frequently, your calibration pa
rameters might be wrong."
I tested the OpenPose for 2d, and the detection is correct. So it must be the calibration parameter issue. But the thing is that I am using some online datasets, like KITTI stereo, to do the test, so the calibration should be fine. In my understanding, the cameraMatrix is intrinsics * extrinsics, which is also known as projection matrix (3x4), and that's what I feed into cameraMatrix field. But still, the error is always there no matter what image and data I am using. Do you have similar experience? Or can you shed some light on how to feed into calibration parameters? Thank you!

hi @MaidouPP, is it possible to illustrate what you actually did?
Thanks.

Hey @pritorius, sorry this is my bad. I though #998 fixed the problem, but it didn't... I did a wrong experiment yesterday. The problem of unusual high re-projection error is still there.

[pritorius]

Hey @pritorius, sorry this is my bad. I though #998 fixed the problem, but it didn't... I did a wrong experiment yesterday. The problem of unusual high re-projection error is still there.

@MaidouPP hi, Thanks for your reply. I am sorry my question was not clear. The place where I am still stuck is the error message: "This function assumes that WQueueSplitter (inside WDatumProducer)
was applied in the first place, i.e., that there is only 1 element"

You mentioned above that: ### "What I did is to create two datums, each containing one image and corresponding camera matrix, with subid and subidmax correctly set."

My question is can you elaborate how you did this? Was this in the workProducer Code of your Custom User Input worker? I have a vector of custom op::Datum as follows that is returned from the workProducer. The basic structure is as follows:

std::shared_ptr> workProducer() {
const auto cvMats = acquireImages(cameraList, matList);
for (auto i = 0u; i < cvMats.size(); i++) {
datums3d->at(i).cvInputData = cvMats.at(i);
// SHOULD I SET THE SUBID and SUBIDMAX of each datums3d[i]?
}
return datums3d;
}
So, in the above code, I have 2 cameras, so there are 2 datums. Should I set the subid and subidmax of each datum?

Thank you.

[MaidouPP]

Hey @pritorius, sorry this is my bad. I though #998 fixed the problem, but it didn't... I did a wrong experiment yesterday. The problem of unusual high re-projection error is still there.

@MaidouPP hi, Thanks for your reply. I am sorry my question was not clear. The place where I am still stuck is the error message: "This function assumes that WQueueSplitter (inside WDatumProducer) was applied in the first place, i.e., that there is only 1 element"

You mentioned above that: ### "What I did is to create two datums, each containing one image and corresponding camera matrix, with subid and subidmax correctly set."

My question is can you elaborate how you did this? Was this in the workProducer Code of your Custom User Input worker? I have a vector of custom op::Datum as follows that is returned from the workProducer. The basic structure is as follows:

std::shared_ptr workProducer() {
const auto cvMats = acquireImages(cameraList, matList);
for (auto i = 0u; i < cvMats.size(); i++) {
datums3d->at(i).cvInputData = cvMats.at(i);
// SHOULD I SET THE SUBID and SUBIDMAX of each datums3d[i]?
}
return datums3d;
}
So, in the above code, I have 2 cameras, so there are 2 datums. Should I set the subid and subidmax of each datum?

Thank you.

Yes, you should set the subbid and subidmax for each datum. Given your circumstance, subid would be 0 and 1, and subidmax would be 1 for each datum. You can go read code in datum.hpp/cpp and see what's the meaning of those data fields. Basically queue splitter is trying to split the data in datums so as to better take advantage of multithread computation, so you are not supposed to put multiple images into one datum manually.

[pritorius]

Hey @pritorius, sorry this is my bad. I though #998 fixed the problem, but it didn't... I did a wrong experiment yesterday. The problem of unusual high re-projection error is still there.

@MaidouPP hi, Thanks for your reply. I am sorry my question was not clear. The place where I am still stuck is the error message: "This function assumes that WQueueSplitter (inside WDatumProducer) was applied in the first place, i.e., that there is only 1 element"
You mentioned above that: ### "What I did is to create two datums, each containing one image and corresponding camera matrix, with subid and subidmax correctly set."
My question is can you elaborate how you did this? Was this in the workProducer Code of your Custom User Input worker? I have a vector of custom op::Datum as follows that is returned from the workProducer. The basic structure is as follows:
std::shared_ptrstd::vector workProducer() {
const auto cvMats = acquireImages(cameraList, matList);
for (auto i = 0u; i < cvMats.size(); i++) {
datums3d->at(i).cvInputData = cvMats.at(i);
// SHOULD I SET THE SUBID and SUBIDMAX of each datums3d[i]?
}
return datums3d;
}
So, in the above code, I have 2 cameras, so there are 2 datums. Should I set the subid and subidmax of each datum?
Thank you.

Yes, you should set the subbid and subidmax for each datum. Given your circumstance, subid would be 0 and 1, and subidmax would be 1 for each datum. You can go read code in datum.hpp/cpp and see what's the meaning of those data fields. Basically queue splitter is trying to split the data in datums so as to better take advantage of multithread computation, so you are not supposed to put multiple images into one datum manually.

@MaidouPP Many thanks for the reply. Actually, I tried that but it did not help. It still failed in the same location. I will look more into it.
Thanks for answering my question.

[djns99]

@pritorius Make sure you return datums3d[0] and datums3d [1] separately.
i.e. Put the frame in a queue and return that on the next call to the function if the queue is not empty

[pritorius]

d return that on the next call to the function if th

@djns99 hi. Cool, I will try it. thanks for the reply.

[djns99]

@MaidouPP @pritorius I tried again with identical cameras and I successfully got the 3d detection working. I'm not sure if this was due to having identical cameras or a more accurate calibration, since I increased the number of images I used for the calibration and took much more care in synchronising the images. It was possibly a combination of both.

I'm still having a few issues with the detection not rendering some key points. But I suspect that has something to do with the fact that the cameras might not be able to see some keypoints.

Below is the class I used to generate the synchronised images. I extended 7_synchronous_custom_input example as suggested.

class WUserInput : public op::WorkerProducer<std::shared_ptr<std::vector<std::shared_ptr<op::Datum>>>>
{
public:
    WUserInput(std::vector<uint32_t> cam_ids)
    {
        for(const auto& cam_id : cam_ids) {
            mCams.emplace_back(std::make_shared<op::WebcamReader>( cam_id ));
        }
    }

    void initializationOnThread() {}

    std::shared_ptr<std::vector<std::shared_ptr<op::Datum>>> workProducer()
    {
        try
        {
            std::lock_guard<std::mutex> g(lock);
            if(mBlocked.empty()) {
                for (size_t i = 0; i < mCams.size(); i++) {
                    auto datumsPtr = std::make_shared<std::vector<std::shared_ptr<op::Datum>>>();
                    // Create new datum
                    datumsPtr->emplace_back();
                    auto& datum = datumsPtr->back();
                    datum = std::make_shared<op::Datum>();

                    // Fill datum
                    datum->cvInputData = mCams[i]->getFrame();
                    datum->cvOutputData = datum->cvInputData;

                    // If empty frame -> return nullptr
                    if (datum->cvInputData.empty())
                    {
                        this->stop();
                        return nullptr;
                    }

                    mBlocked.push(datumsPtr);
                }
            }

            auto ret = mBlocked.front();
            mBlocked.pop();
            return ret;
        }
        catch (const std::exception& e)
        {
            this->stop();
            op::error(e.what(), __LINE__, __FUNCTION__, __FILE__);
            return nullptr;
        }
    }

private:
    std::vector<std::shared_ptr<op::WebcamReader>> mCams;
    std::queue<std::shared_ptr<std::vector<std::shared_ptr<op::Datum>>>> mBlocked;
    std::mutex lock;
};

with this python script to rename so that they were processed in the right order

import os

for file in sorted(os.listdir('.')):
        parts = file.split('_')
        id = int(parts[0])
        os.system("mv {} {:012d}_{}".format(file, id, '_'.join(parts[1:])))

And here is my class for rendering the images.

class WUserInput : public op::WorkerProducer<std::shared_ptr<std::vector<std::shared_ptr<op::Datum>>>>
{
public:
    WUserInput(std::vector<uint32_t> cam_ids, bool use3d)
        : mUse3d(use3d),
          mParamReader(std::make_shared<op::CameraParameterReader>())
    {
        for(const auto& cam_id : cam_ids) {
            mCams.emplace_back(std::make_shared<op::WebcamReader>( cam_id ));
        }
        if (use3d) {
            mParamReader->readParameters(FLAGS_calibration_dir);
            mIntrinsics = mParamReader->getCameraIntrinsics();
            mExtrinsics = mParamReader->getCameraExtrinsics();
            mMatrices = mParamReader->getCameraMatrices();
        }
    }

    void initializationOnThread() {}

    std::shared_ptr<std::vector<std::shared_ptr<op::Datum>>> workProducer()
    {
        try
        {
            std::lock_guard<std::mutex> g(lock);
            if(mBlocked.empty()) {

                for (size_t i = 0; i < mCams.size(); i++) {
                    auto datumsPtr = std::make_shared<std::vector<std::shared_ptr<op::Datum>>>();
                    // Create new datum
                    datumsPtr->emplace_back();
                    auto& datum = datumsPtr->back();
                    datum = std::make_shared<op::Datum>();

                    // Fill datum
                    datum->cvInputData = mCams[i]->getFrame();
                    datum->cvOutputData = datum->cvInputData;
                    datum->subId = i;
                    datum->subIdMax = mCams.size() - 1;
                    if(mUse3d) {
                        datum->cameraIntrinsics = mIntrinsics[i];
                        datum->cameraExtrinsics = mExtrinsics[i];
                        datum->cameraMatrix = mMatrices[i];
                    }

                    // If empty frame -> return nullptr
                    if (datum->cvInputData.empty())
                    {
                        this->stop();
                        return nullptr;
                    }

                    mBlocked.push(datumsPtr);
                }
            }

            auto ret = mBlocked.front();
            mBlocked.pop();
            return ret;
        }
        catch (const std::exception& e)
        {
            this->stop();
            op::error(e.what(), __LINE__, __FUNCTION__, __FILE__);
            return nullptr;
        }
    }

private:
    bool mUse3d;
    std::shared_ptr<op::CameraParameterReader> mParamReader;
    std::vector<cv::Mat> mIntrinsics;
    std::vector<cv::Mat> mExtrinsics;
    std::vector<cv::Mat> mMatrices;
    std::vector<std::shared_ptr<op::WebcamReader>> mCams;
    std::queue<std::shared_ptr<std::vector<std::shared_ptr<op::Datum>>>> mBlocked;
    std::mutex lock;
};

[MaidouPP]

@MaidouPP @pritorius I tried again with identical cameras and I successfully got the 3d detection working. I'm not sure if this was due to having identical cameras or a more accurate calibration, since I increased the number of images I used for the calibration and took much more care in synchronising the images. It was possibly a combination of both.

I'm still having a few issues with the detection not rendering some key points. But I suspect that has something to do with the fact that the cameras might not be able to see some keypoints.

Below is the class I used to generate the synchronised images. I extended 7_synchronous_custom_input example as suggested.

class WUserInput : public op::WorkerProducer<std::shared_ptr<std::vector<std::shared_ptr<op::Datum>>>>
{
public:
    WUserInput(std::vector<uint32_t> cam_ids)
    {
      for(const auto& cam_id : cam_ids) {
          mCams.emplace_back(std::make_shared<op::WebcamReader>( cam_id ));
      }
    }

    void initializationOnThread() {}

    std::shared_ptr<std::vector<std::shared_ptr<op::Datum>>> workProducer()
    {
        try
        {
          std::lock_guard<std::mutex> g(lock);
          if(mBlocked.empty()) {
              for (size_t i = 0; i < mCams.size(); i++) {
                  auto datumsPtr = std::make_shared<std::vector<std::shared_ptr<op::Datum>>>();
                  // Create new datum
                  datumsPtr->emplace_back();
                  auto& datum = datumsPtr->back();
                  datum = std::make_shared<op::Datum>();

                  // Fill datum
                  datum->cvInputData = mCams[i]->getFrame();
                  datum->cvOutputData = datum->cvInputData;

                  // If empty frame -> return nullptr
                  if (datum->cvInputData.empty())
                  {
                      this->stop();
                      return nullptr;
                  }

                  mBlocked.push(datumsPtr);
              }
          }

          auto ret = mBlocked.front();
          mBlocked.pop();
          return ret;
        }
        catch (const std::exception& e)
        {
            this->stop();
            op::error(e.what(), __LINE__, __FUNCTION__, __FILE__);
            return nullptr;
        }
    }

private:
    std::vector<std::shared_ptr<op::WebcamReader>> mCams;
    std::queue<std::shared_ptr<std::vector<std::shared_ptr<op::Datum>>>> mBlocked;
    std::mutex lock;
};

with this python script to rename so that they were processed in the right order

import os

for file in sorted(os.listdir('.')):
        parts = file.split('_')
        id = int(parts[0])
        os.system("mv {} {:012d}_{}".format(file, id, '_'.join(parts[1:])))

And here is my class for rendering the images.

class WUserInput : public op::WorkerProducer<std::shared_ptr<std::vector<std::shared_ptr<op::Datum>>>>
{
public:
    WUserInput(std::vector<uint32_t> cam_ids, bool use3d)
      : mUse3d(use3d),
        mParamReader(std::make_shared<op::CameraParameterReader>())
    {
      for(const auto& cam_id : cam_ids) {
          mCams.emplace_back(std::make_shared<op::WebcamReader>( cam_id ));
      }
      if (use3d) {
          mParamReader->readParameters(FLAGS_calibration_dir);
          mIntrinsics = mParamReader->getCameraIntrinsics();
          mExtrinsics = mParamReader->getCameraExtrinsics();
          mMatrices = mParamReader->getCameraMatrices();
      }
    }

    void initializationOnThread() {}

    std::shared_ptr<std::vector<std::shared_ptr<op::Datum>>> workProducer()
    {
        try
        {
          std::lock_guard<std::mutex> g(lock);
          if(mBlocked.empty()) {

              for (size_t i = 0; i < mCams.size(); i++) {
                  auto datumsPtr = std::make_shared<std::vector<std::shared_ptr<op::Datum>>>();
                  // Create new datum
                  datumsPtr->emplace_back();
                  auto& datum = datumsPtr->back();
                  datum = std::make_shared<op::Datum>();

                  // Fill datum
                  datum->cvInputData = mCams[i]->getFrame();
                  datum->cvOutputData = datum->cvInputData;
                  datum->subId = i;
                  datum->subIdMax = mCams.size() - 1;
                  if(mUse3d) {
                      datum->cameraIntrinsics = mIntrinsics[i];
                      datum->cameraExtrinsics = mExtrinsics[i];
                      datum->cameraMatrix = mMatrices[i];
                  }

                  // If empty frame -> return nullptr
                  if (datum->cvInputData.empty())
                  {
                      this->stop();
                      return nullptr;
                  }

                  mBlocked.push(datumsPtr);
              }
          }

          auto ret = mBlocked.front();
          mBlocked.pop();
          return ret;
        }
        catch (const std::exception& e)
        {
            this->stop();
            op::error(e.what(), __LINE__, __FUNCTION__, __FILE__);
            return nullptr;
        }
    }

private:
    bool mUse3d;
    std::shared_ptr<op::CameraParameterReader> mParamReader;
    std::vector<cv::Mat> mIntrinsics;
    std::vector<cv::Mat> mExtrinsics;
    std::vector<cv::Mat> mMatrices;
    std::vector<std::shared_ptr<op::WebcamReader>> mCams;
    std::queue<std::shared_ptr<std::vector<std::shared_ptr<op::Datum>>>> mBlocked;
    std::mutex lock;
};

@djns99 Hey thanks for sharing this! I can see that in your use case, you actually feed the cameraMatrix by using .xml as stated in docs because you are using real cameras. I think our scenarios are somehow different, in the sense that I want to feed the cameraMatrix in the datum I constructed, and getting 3d reconstruction only given the datums containing images and matrices.

Anyway, I guess in my use case, I need to do more hack than I thought. But again, thank you for reply! I will also try doing 3d reconstruction in this way : )

[MaidouPP]

@djns99 Hi! I am curious that, in your 3d-rendering worker class, you mentioned that is for display. What is the relation between this worker, and the previous worker with pure camera reading? In my understanding, you are using this 3d-render worker class to do both image feeding, camera matrices feeding all in datums, without having to use cameraParameterReader, right?

[djns99]

@djns99 Hi! I am curious that, in your 3d-rendering worker class, you mentioned that is for display. What is the relation between this worker, and the previous worker with pure camera reading? In my understanding, you are using this 3d-render worker class to do both image feeding, camera matrices feeding all in datums, without having to use cameraParameterReader, right?

@MaidouPP The first class generates synchronised images I used for doing extrinsic calibration. The second one is used for the actual 3d reconstruction implementation.

edit: The 3d reconstruction class reads from the webcam and sets the cameraMatrix in the same operation if thats what you mean. I don't strictly have to use CameraParameterReader. It was just for convenience sake.

[djns99]

@djns99 Hey thanks for sharing this! I can see that in your use case, you actually feed the cameraMatrix by using .xml as stated in docs because you are using real cameras. I think our scenarios are somehow different, in the sense that I want to feed the cameraMatrix in the datum I constructed, and getting 3d reconstruction only given the datums containing images and matrices.

Anyway, I guess in my use case, I need to do more hack than I thought. But again, thank you for reply! I will also try doing 3d reconstruction in this way : )

@MaidouPP Im not sure I quite understand what you mean here? You can just pass in your cameraMatrix from some other source if you do not have an .xml file.

[MaidouPP]

@djns99 Hi! I am curious that, in your 3d-rendering worker class, you mentioned that is for display. What is the relation between this worker, and the previous worker with pure camera reading? In my understanding, you are using this 3d-render worker class to do both image feeding, camera matrices feeding all in datums, without having to use cameraParameterReader, right?

@MaidouPP The first class generates synchronised images I used for doing extrinsic calibration. The second one is used for the actual 3d reconstruction implementation.

edit: The 3d reconstruction class reads from the webcam and sets the cameraMatrix in the same operation if thats what you mean. I don't strictly have to use CameraParameterReader. It was just for convenience sake.

Thanks for pointing out. Your code really reminds me of something I didn't do right... I should've customized my input wrapper, instead of leaving it there and only pushing datums onto thread manager.

[pritorius]

@MaidouPP @pritorius I tried again with identical cameras and I successfully got the 3d detection working. I'm not sure if this was due to having identical cameras or a more accurate calibration, since I increased the number of images I used for the calibration and took much more care in synchronising the images. It was possibly a combination of both.

I'm still having a few issues with the detection not rendering some key points. But I suspect that has something to do with the fact that the cameras might not be able to see some keypoints.

Below is the class I used to generate the synchronised images. I extended 7_synchronous_custom_input example as suggested.

class WUserInput : public op::WorkerProducer<std::shared_ptr<std::vector<std::shared_ptr<op::Datum>>>>
{
public:
    WUserInput(std::vector<uint32_t> cam_ids)
    {
      for(const auto& cam_id : cam_ids) {
          mCams.emplace_back(std::make_shared<op::WebcamReader>( cam_id ));
      }
    }

    void initializationOnThread() {}

    std::shared_ptr<std::vector<std::shared_ptr<op::Datum>>> workProducer()
    {
        try
        {
          std::lock_guard<std::mutex> g(lock);
          if(mBlocked.empty()) {
              for (size_t i = 0; i < mCams.size(); i++) {
                  auto datumsPtr = std::make_shared<std::vector<std::shared_ptr<op::Datum>>>();
                  // Create new datum
                  datumsPtr->emplace_back();
                  auto& datum = datumsPtr->back();
                  datum = std::make_shared<op::Datum>();

                  // Fill datum
                  datum->cvInputData = mCams[i]->getFrame();
                  datum->cvOutputData = datum->cvInputData;

                  // If empty frame -> return nullptr
                  if (datum->cvInputData.empty())
                  {
                      this->stop();
                      return nullptr;
                  }

                  mBlocked.push(datumsPtr);
              }
          }

          auto ret = mBlocked.front();
          mBlocked.pop();
          return ret;
        }
        catch (const std::exception& e)
        {
            this->stop();
            op::error(e.what(), __LINE__, __FUNCTION__, __FILE__);
            return nullptr;
        }
    }

private:
    std::vector<std::shared_ptr<op::WebcamReader>> mCams;
    std::queue<std::shared_ptr<std::vector<std::shared_ptr<op::Datum>>>> mBlocked;
    std::mutex lock;
};

with this python script to rename so that they were processed in the right order

import os

for file in sorted(os.listdir('.')):
        parts = file.split('_')
        id = int(parts[0])
        os.system("mv {} {:012d}_{}".format(file, id, '_'.join(parts[1:])))

And here is my class for rendering the images.

class WUserInput : public op::WorkerProducer<std::shared_ptr<std::vector<std::shared_ptr<op::Datum>>>>
{
public:
    WUserInput(std::vector<uint32_t> cam_ids, bool use3d)
      : mUse3d(use3d),
        mParamReader(std::make_shared<op::CameraParameterReader>())
    {
      for(const auto& cam_id : cam_ids) {
          mCams.emplace_back(std::make_shared<op::WebcamReader>( cam_id ));
      }
      if (use3d) {
          mParamReader->readParameters(FLAGS_calibration_dir);
          mIntrinsics = mParamReader->getCameraIntrinsics();
          mExtrinsics = mParamReader->getCameraExtrinsics();
          mMatrices = mParamReader->getCameraMatrices();
      }
    }

    void initializationOnThread() {}

    std::shared_ptr<std::vector<std::shared_ptr<op::Datum>>> workProducer()
    {
        try
        {
          std::lock_guard<std::mutex> g(lock);
          if(mBlocked.empty()) {

              for (size_t i = 0; i < mCams.size(); i++) {
                  auto datumsPtr = std::make_shared<std::vector<std::shared_ptr<op::Datum>>>();
                  // Create new datum
                  datumsPtr->emplace_back();
                  auto& datum = datumsPtr->back();
                  datum = std::make_shared<op::Datum>();

                  // Fill datum
                  datum->cvInputData = mCams[i]->getFrame();
                  datum->cvOutputData = datum->cvInputData;
                  datum->subId = i;
                  datum->subIdMax = mCams.size() - 1;
                  if(mUse3d) {
                      datum->cameraIntrinsics = mIntrinsics[i];
                      datum->cameraExtrinsics = mExtrinsics[i];
                      datum->cameraMatrix = mMatrices[i];
                  }

                  // If empty frame -> return nullptr
                  if (datum->cvInputData.empty())
                  {
                      this->stop();
                      return nullptr;
                  }

                  mBlocked.push(datumsPtr);
              }
          }

          auto ret = mBlocked.front();
          mBlocked.pop();
          return ret;
        }
        catch (const std::exception& e)
        {
            this->stop();
            op::error(e.what(), __LINE__, __FUNCTION__, __FILE__);
            return nullptr;
        }
    }

private:
    bool mUse3d;
    std::shared_ptr<op::CameraParameterReader> mParamReader;
    std::vector<cv::Mat> mIntrinsics;
    std::vector<cv::Mat> mExtrinsics;
    std::vector<cv::Mat> mMatrices;
    std::vector<std::shared_ptr<op::WebcamReader>> mCams;
    std::queue<std::shared_ptr<std::vector<std::shared_ptr<op::Datum>>>> mBlocked;
    std::mutex lock;
};

@djns99 Thanks for the amazing reply. I tried a similar thing couple of days ago ( some things are very similar to your code ). But I am still getting unusual high re-projection error. The code is as follows:

`std::shared_ptr> workProducer()
{
try
{
auto datums3d = std::make_shared>(1);

            // Check if the imageque still has unprocessed images. if so, return that.
            // else acquire image.
            if ( imageQueue.size() == 0 )  {
                    // Get image from each camera
                    const auto cvMats = acquireImages(cameraList, matList);

                    // Images to userDatum
                    for (auto i = 0u; i < cvMats.size(); i++)
                    {
                            auto aDatum3d = Datum3D();
                            aDatum3d.cvInputData = cvMats.at(i);
                            aDatum3d.subId = i;
                            aDatum3d.subIdMax = cvMats.size() - 1;
                            aDatum3d.cameraIntrinsics = INTRINSICS[0];
                            aDatum3d.cameraExtrinsics = EXTRINSIC_PS3EYE_1;
                            aDatum3d.cameraMatrix =  INTRINSICS[0] *  EXTRINSIC_PS3EYE_1;

                            imageQueue.push(aDatum3d);
                    }
             }

             auto aDatum3d = imageQueue.front();

             datums3d->at(0).cvInputData = aDatum3d.cvInputData;
             datums3d->at(0).subId = aDatum3d.subId;
             datums3d->at(0).subIdMax = aDatum3d.subIdMax;
             datums3d->at(0).cameraIntrinsics = INTRINSICS[0];
             datums3d->at(0).cameraExtrinsics = EXTRINSIC_PS3EYE_1;
             datums3d->at(0).cameraMatrix =  INTRINSICS[0] *  EXTRINSIC_PS3EYE_1;

             imageQueue.pop();

             // Return Datum
             return datums3d;
    }
    catch (const std::exception& e)
    {
            this->stop();
            op::error(e.what(), __LINE__, __FUNCTION__, __FILE__);
            return nullptr;
    }

}
`

The code where your and mine differs are:

aDatum3d.cameraIntrinsics = INTRINSICS[0];
aDatum3d.cameraExtrinsics = EXTRINSIC_PS3EYE_1;
aDatum3d.cameraMatrix = INTRINSICS[0] * EXTRINSIC_PS3EYE_1;

I got the intrinsic matrix by running the calibration module for camera intrinsics ( my camera is PS3EYE ).
The extrinsic is just a 3 by 4 matrix:
const cv::Mat EXTRINSIC_PS3EYE_1= (cv::Mat_(3, 4) <<
1.0, 0.0, 0.0, 0.0,
0.0, 1.0, 0.0, 0.0,
0.0, 0.0, 1.0, 0.0);

I did not use the calibration module to get camera extrinsic. The reason is, what extrinsic matrix is saying is the amount of rotation and translation needed to bring a world coordinate into camera coordinate. But this is going to change as soon as I move the camera. and both of my cameras are moving.

So what am I doing wrong here. Is it absolutely necessary to get camera extrinsics? But that extrinsic matrix will change as soon as camera is lightly moved isn't it?

A somewhat relevant link:
https://stackoverflow.com/questions/27486847/extrinsic-parameters-of-camera-are-they-constant

Update:
OK, I found the following link:
https://forums.ni.com/t5/Machine-Vision/Extrinsic-parameters-rotation-matrix/td-p/3065842

It says,

During the calibration, while the cameras setup is constant, you are acquiring the grid from different positions in order to minimize the error of extrinsic/intrinsic parameters calculation.

Also, you need translation in addition to rotation to align the images/coordinate systems.

YES, WHEN YOU CALIBRATE YOUR SYSTEM THE ROTATION MATRIX AND TRANSLATION VECTOR WILL REMAIN CONSTANT AS LONG AS THE CAMERAS ARE IN THE SAME RELATIVE POSITION TO EACH OTHER.

Consider your calibration as a stereo setup.

So, which means that I have to somehow keep the relative position of the cameras constant.
BTW what camera are you using? Are you using 2 physically separate cameras?
I am using two PS3Eye cameras.

[pritorius]

Hello Guys,
I need a clarification about the unit of camera intrinsics and camera translation values.
I am using 3 Sony Ps3Eye camera. I did the calibration. Examples of intrinsics and extrinsics are:

Intrinsics:
526.205413 0 0
-0.510150 531.263273 0
318.127497 212.243599 1

Extrinsics:
0.949561 -0.033976 -0.3117378 -512.07335
0.0357485 0.999361 -2.7854991 20.44102
0.3115395 -0.0111177 0.9501682 191.92707

The values in intrinsic matrix and the 4th column of extrinsic matrix is in millimeter.
Should I convert these values to meter before using ?

Thanks,
Regards,

[gineshidalgo99]

@pritorius No, if you provide it in mm, the 3D JSON results will be in mm. If you provide it in m, the result will be in m. OpenPose simply uses whatever measure you are using. The only potential problem is that nothing might be visually rendered (because the rendering code currently requires a hard-coded value of expected values, so the 3D points should have a value of 0.5-5 for correct visualization.

[pritorius]

No, if you provide it in mm, the 3D JSON results will be in mm

Can you clarify a little what do you mean when you say "you provide". I am just putting the values into openpose/3d/cameraParameters.hpp as cv::Mat and using it during starting openpose.
Do I need to specify the unit of measure somewhere?

BTW I am using matlab to calculate the intrinsic/extrinsic matrices of my setup. I could not make calibration module work.
Thanks.
Regards,

[Edison-xiao927]

I just delete the wQueueAssembler thread and the data can pass through now, but get a new warning now,
Unusual high re-projection error (averaged over #keypoints) of value 4.392997 pixels, while the average for a good OpenPose detection from 4 cameras is about 2-3 pixels. It might be simply a wrong OpenPose detection. If this message appears very frequently, your calibration parameters might be wrong.
And the 3-D display window is always black.

I am using the dataset 150821_dance2 01_01 , 06_15 , 14_03

The 3-D pose output is 0 all the time.

Have you solved this problem？I have the same problem，Unusual high re-projection error。Is there something wrong with my xml files？or Is my calibration parameter issue？

[pritorius]

Have you solved this problem？I have the same problem，Unusual high re-projection error。Is there something wrong with my xml files？or Is my calibration parameter issue？

Hi, I have not yet able to solve the issue. 3D keypoints in output json is also 0 for me.
--> openpose is correctly identifying the 2D keypoints. So, I have a strong guess that I am messing up in during 3d reconstruction.
--> I am going to try a couple of things,

1. I will convert translation vector (4th column) into meters. RIght now it is in millimeters.
2. Try the opencv::triangulatePoints function instead of openpose triangulation and see what result that gives me.
   BTW my cameraparameters are as follows:

Middle camera:
Intrinsics:
526.205413 0 0
-0.510150 531.263273 0
318.127497 212.243599 1
Extrinsics:
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1

Left Camera:
Intrinsics:
532.338192 0 0
-0.362866 536.560859 0
291.500725 233.526212 1
Extrinsics w.r.t Middle Camera:
0.949561 -0.033976 -0.3117378 -512.07335
0.0357485 0.999361 -2.7854991 20.44102
0.3115395 -0.0111177 0.9501682 191.92707
Right Camera:
Intrinsics:
520.801688 0 0
0 526.255718 0
310.620930 234.217933 1
Extrinsics w.r.t Middle Camera:
0.979391 0.0374783 0.198468 599.204522
-0.011845 0.9915999 -0.128799 29.621879
-0.201628 0.123794 0.971608 107.433003

[gineshidalgo99]

I can see your Intrinsics seem weird, they should be an upper triangular matrix (I will add this into the doc, I know in some toolboxes they are returned in a different way, but it should be an upper triangular matrix for OpenCV and therefore OpenPose).

[pritorius]

I can see your Intrinsics seem weird, they should be an upper triangular matrix (I will add this into the doc, I know in some toolboxes they are returned in a different way, but it should be an upper triangular matrix for OpenCV and therefore OpenPose).

Yes you are right. The camera instrinsics are flipped. I did camera calibration using matlab and I did not do sanity check on the result. Thanks a lot for pointing this out.

[pritorius]

I just delete the wQueueAssembler thread and the data can pass through now, but get a new warning now,
Unusual high re-projection error (averaged over #keypoints) of value 4.392997 pixels, while the average for a good OpenPose detection from 4 cameras is about 2-3 pixels. It might be simply a wrong OpenPose detection. If this message appears very frequently, your calibration parameters might be wrong.
And the 3-D display window is always black.
I am using the dataset 150821_dance2 01_01 , 06_15 , 14_03
The 3-D pose output is 0 all the time.

Have you solved this problem？I have the same problem，Unusual high re-projection error。Is there something wrong with my xml files？or Is my calibration parameter issue？

Hi,
I was able to generate 3d pose keypoints in the output json file. ( I have not yet looked into how to render the 3d keypoints yet). Here is what I did:

1. I am using 3 ps3eye cameras.
2. I used MATLAB and this link to generate my camera intrinsic and extrinsic parameters. Sadly, I could not make the calibration module work due to this issue.
3. The intrinsic parameters generated by MATLAB needs to be transposed ( as pointed out by @gineshidalgo99) in order to be used in openpose. The same goes to 3 by 3 rotation matrix of the extrinsics. After taking transpose of rotation matrix, add the translation vector as 4th column to get full extrinsics.
4. After step 3, I have the following:
   For middle camera: a 3x3 intrinsic matrix (upper triangular) and a 3x4 identity matrix.
   For Left camera: a 3x3 intrinsic matrix and a 3x4 extrinsic matrix ( rotation and translation w.r.t the middle camera).
   For right camera: a 3x3 intrinsic matrix and a 3x4 extrinsic matrix ( rotation and translation w.r.t the middle camera).
5. I loaded these values into include/openpose/3d/cameraParameters.hpp.
6. Then I ran the _./build/examples/tutorial_wrapper/4_user_synchronous_all.bin --net_resolution 64x64 --3d_ ( which contains the custom code to use ps3eye cameras as producers).

[Edison-xiao927]

Thank you for your reply。I will try the method you use

[FranzAlbers]

I was able to generate 3d pose keypoints in the output json file. ( I have not yet looked into how to render the 3d keypoints yet).

Thanks for your help! The extrinsic calibration using MATLAB worked fine for me. The translation vector of the extrinsic calibration matrix was specified in mm.

The openpose keyPoints3D seemed to be out of range for the 3d GUI. By changing the scale (see also #559 ) in renderHumanBody() to

const auto xScale = 2000.f;   // before: 43.f;
const auto yScale = 1500.f;   // before: 24.f;
const auto zScale = 1500.f;   // before: 24.f;

and setting the gluPerspective in initGraphics() to

gluPerspective( /* field of view in degree */ 50.0,
                /* aspect ratio */ 1.0,
                /* Z near */ 1.0, /* Z far */ 15000.0);    // Increased far clipping plane to 15000.0

rendering the keypoints with the 3D openpose GUI worked for me.

[gineshidalgo99]

Everything seems fixed then, closed!