Openmvg: Question: Automatically Scale

[Ntweat]

Hi @pmoulon .
Is there a way to automatically scale the model? (either with openMVG or openMVS)
Currently, I keep a checkerboard next to the object and use that as baseline to scale the model manually in meshlab.
Is there anyway by which the model can get scaled automatically?

[pmoulon]

If you want to scale the scene automatically you must detect object of known scale.
In your case you are using a checkerboard to manually register by hand the scene.

• You can detect the checkerboard, infer the pose of the image and then rescale your scene.

Or you can use simpler target like QR codes for this purpose.

I can guide you on how to to that if you want.

[Ntweat]

Hi

I would love your help in this!

I also wanted to ask if you have tried making models of something like a car, I tried it a few times never worked.

[pmoulon]

Car are hard cases since they are reflective (metallic car paint).
It will be true for every photogrammetry toolsuite.

Often people that deal with such curve based metallic object put markers on the car skin and then build curves in 3d. Then they link the 3D curves into surface in a 3D softwares.

You can build car model by photogrammetry only if you have a nice texture on the car (or some dirt...).

[Ntweat]

Ok, thought so.

How to start with automatic scaling

[Ntweat]

@pmoulon How should I start for automatic scaling?

[pmoulon]

1- are you able to detect your checkboard corners in many images?

• if so:

1. triangulate the corresponding points (you can take the 4 extreme corners),

2. then compute the 3D distances between some of them and compute the rescaling factor you have to apply rescale the scene accordingly. When you have your rescaling factor, you create a geometry::Similarity3 and apply it to your sfm_data scene ->

https://github.com/openMVG/openMVG/blob/6ce820b2844a98941a5d2543050f4e278a323800/src/openMVG/sfm/sfm_data_transform.hpp#L24

[Ntweat]

I am able to. But not always the whole checker board is reconstructed in the final mesh.
This is my pipeline:

1. User takes a video of object with the checkerboard next to it (can be a QR code)
2. Uploads it to the server.
3. The server extracts the images from the video and creates a model. Which the user can download.
   Right now I am manually rescaling the models.

[Ntweat]

Hi @pmoulon

Can you tell me if you know any papers or codes I can read for guidance on how to triangulate the points and compute the 3D distance?

[pmoulon]

• Triangulation: -> https://github.com/openMVG/openMVG/blob/master/src/openMVG/multiview/triangulation_nview.hpp

• 3d distance (substract the 3d coordinates)

[Ntweat]

Thank you 👍

[Ntweat]

Hi

I have been working on this. I started using QRCodes, since their detection is much simpler. I am now able to detect the corners and calculate the scale for 2D images (by detecting the 3 corners of the QR code). I am struggling on their triangulation in the 3D point cloud. I am using python with C++ (for openMVG)

Any help to triangulate the QRCode to 3D is very appreciated 👍

[pmoulon]

Given the camera position and intrinsics you can use the multiview triangulation functions.
https://github.com/openMVG/openMVG/blob/master/src/openMVG/multiview/triangulation_nview.hpp

[Ntweat]

I went through triangulation_nview.hpp as you had told earlier. But I don't know the positions or intrinsics. I am using videos to generate 3D models by extracting frames. For focal length, I am using the formula you had told a few months back f = 1.2*max(w,h)

[Ntweat]

Is there any way to locate the QR code in the completed 3D model? (get the 3D coordinates of the 3 points)

[pmoulon]

You can estimate the camera position from your QR codes.
(you know the position of 4 points in the scenes and so you can compute a global camera pose.)

Is there any way to locate the QR code in the completed 3D model? (get the 3D coordinates of the 3 points)

• you can detect the QR code in the image and then project the camera ray to the QR code corner or center to the mesh to compute an intersection and know its 3d position.

[Ntweat]

I am trying to use the QR codes to calculate the camera positions, translation and rotation matrices.
Am I still new to 3D modeling, so I am struggling with simple things. Looking up equations on how to do it :)

[Ntweat]

Hi

I now able to calculate the Homography matrices between the frames which have the QRcode. But now I have the following questions (some might be very basic):

1. Is there a way to know which is the reference image for globalSfM? If my reference frame is different than that of SfM then the scale may be different? (First frame might now have the QRCODE)
2. Can we know the camera intrinsic which globalSfM calculates.? (I am using the f=1.2*max(w,h) )

[pmoulon]

1. There is no particular reference frame. But you can fix one by applying the inverse pose of one of the picture to your scene.

2. For sure. You can use main_ConvertSfM_DataFormat in order to export the scene in json with for example only the intrinsic information:

3. main_ConvertSfM_DataFormat -i <your sfm_data.bin> -o scene.json -I.
4. Then you can read the refined focal length in the json file.

[Ntweat]

Thanks 👍
Now, I have the camera intrinsics, the QR corners on all the images and the homography matrices.

Just need a little more guidance on this.
Can you tell me how to plug all of them into https://github.com/openMVG/openMVG/blob/master/src/openMVG/multiview/triangulation_nview.hpp
to get the 3D coordinates?
I am using Python to call all the openMVG libraries. I am calculating Homographies also using python.

[pmoulon]

You need camera pose matrices from your QRCorners.

• Since you know the size of your pattern you can compute the camera poses thanks to P3P (Absolute Pose Estimation).

Once you have the camera pose matrices, you can send the 2d QR corners as points tracks and projection matrices and obtain the 3d points.

[Ntweat]

I used the solvePnP function in openCV to get the rotation and translation matrices.
Now to use the triangulation_nview.hpp

[pmoulon]

FYI, P3P solver are also available in OpenMVG ;-)

[Ntweat]

Where? I would like to use OpenMVG only it will maintain a consistency in the code.
Sorry, I seem to be asking you a lot.

[Ntweat]

Is there any detailed manual on how to use solver_resection_p3p.hpp and triangulation_nview.hpp . I tried reading the code to understand what is happening but sadly couldn't get it.

[pmoulon]

The best place to understand how to use the code is to look to the corresponding unit test that use it.
So you can figure it out how it must be used.

[pmoulon]

Sorry, I seem to be asking you a lot.

No problem, we are happy to help and provide guidance.

[Ntweat]

I could not find solver_resection_p3p_test.cpp

[pmoulon]

All the absolute pose estimation algorithm are tested there https://github.com/openMVG/openMVG/blob/master/src/openMVG/multiview/solver_resection_kernel_test.cpp

[Ntweat]

@pmoulon Thank you!
Looking at your code I can see that the scale calculations can be done in less than 100 lines of code, but still am unable to grasp my head around it (may because I have become too comfortable with python). Being stuck on this for nearly a month has made my head spin. But I will persevere.

[Ntweat]

@pmoulon
Question: I saw that sfm_data.jso also has the extrinsic parameters. Can I use them for triangulation? If yes, then the rotation matrices have a "key" how to map them, are all the matrices with respect to the image in "key 0" or they are with respect to the previous key? Also, are the centers for each key the translation matrics?

[pmoulon]

The poses are [R|C]. The C value is the camera position in the world space.
You can use them for triangulation.
There is not reference image, all the matrices express the global coordinates.

[Ntweat]

Nearly done :100:
Preliminary attempts have been successful, just have to automate it.
Will close this as soon as it is done.

[pmoulon]

Great. Would be nice to find a way to contribute it to the project ;-)

[Ntweat]

Compiling issues.
The code works on Windows, but won't compile on Linux. Using the following command

gcc -I/home/ntweat/openMVG/src/ -I/home/ntweat/openMVG/src/third_party/eigen/ -L/home/ntweat/openMVG_build/Linux-x86_64-RELEASE -llibeasyexif -llibfast -lliblemon -lliblib_clp -lliblib_CoinUtils -lliblib_Osi -lliblib_OsiClpSolver -llibopenMVG_features -llibopenMVG_image -llibopenMVG_kvld -llibopenMVG_lInftyComputerVision -llibopenMVG_matching -llibopenMVG_matching_image_collection -llibopenMVG_multiview -llibopenMVG_multiview_test_data -llibopenMVG_numeric -llibopenMVG_sfm -llibopenMVG_system -llibstlplus -llibvlsift ScaleFinderQR.cpp main.cpp -o scale

Many errors
/home/ntweat/openMVG/src/openMVG/numeric/numeric.h:74:7: error: expected nested-name-specifier before ‘EigenDoubleTraits’ using EigenDoubleTraits = Eigen::NumTraits<double>; ^ /home/ntweat/openMVG/src/openMVG/numeric/numeric.h:77:7: error: expected nested-name-specifier before ‘Vec3’ using Vec3 = Eigen::Vector3d; ^ /home/ntweat/openMVG/src/openMVG/numeric/numeric.h:80:7: error: expected nested-name-specifier before ‘Vec2i’ using Vec2i = Eigen::Vector2i; ^ /home/ntweat/openMVG/src/openMVG/numeric/numeric.h:83:7: error: expected nested-name-specifier before ‘Vec2f’ using Vec2f = Eigen::Vector2f;

What am doing wrong in compiling

[pmoulon]

You must enable CXX11 compiler flags.

[Ntweat]

Thank
but now am getting
/home/ntweat/openMVG/src/third_party/eigen/Eigen/src/Core/util/StaticAssert.h:32:40: error: static assertion failed: YOU_MIXED_MATRICES_OF_DIFFERENT_SIZES
it worked on windows

[pmoulon]

Can you try to locate when the message comes from? The compiler must tell you the pyramid of calls that leads to this error.

[Ntweat]

`
In file included from /home/ntweat/openMVG/src/third_party/eigen/Eigen/Core:254:0,
from /home/ntweat/openMVG/src/openMVG/numeric/numeric.h:35,
from /home/ntweat/openMVG/src/openMVG/multiview/projection.hpp:31,

`

[pmoulon]

Can you show the line thanks to github?
Since I don't know if you are using master or develop branch I cannot rely the the line.

[Ntweat]

I am using master branch

https://github.com/openMVG/openMVG/blob/6ce820b2844a98941a5d2543050f4e278a323800/src/third_party/eigen/Eigen/src/Core/util/StaticAssert.h#L32

I think it may be because the linux machine is using an older version of openMVG

in my projection.hpp the line 31 is #include "openMVG/numeric/numeric.h"

I just checked in github the line 32 is #include "openMVG/numeric/eigen_alias_definition.hpp"

can that be a factor?

[Ntweat]

I was able to compile it on codeblocks. (by pulling a fresh code)

Now,
TriangulateNView(xs, Ps, &X) is causing a segmentation fault (core dump)

It was causing the same error in windows under release but worked with Debug
on Linux it is giving segmentation fault with both Release and Debug

[Ntweat]

xs is Mat3X
Ps is Mat34
X is Vec4

Am I passing wrong data types?

[pmoulon]

Are you using OpenMVG master or develop?

-> You are using the right types.
Certainly an alignment issue but this is strange. Did the OpenMVG unit tests fun fine on your computer?

[Ntweat]

Hi

I have tried both master and develop. Both are giving the same issue.
Yes, make test ran with no issues.

[pmoulon]

So it unit test ran without issues, it is just a matter of compilation flag and ensure that the same version of Eigen is used.

Did you used cmake to build your external project?
https://github.com/openMVG/openMVG/blob/master/BUILD.md#using-openmvg-as-a-third-party-library-dependency-in-cmake

[Ntweat]

I rebuild it as external project (just to be sure)
I am using Eigen which is inbuilt into openMVG

[Ntweat]

Got the issue
The issue is this line. For some reason it is not able to allocate memory

https://github.com/openMVG/openMVG/blob/6ce820b2844a98941a5d2543050f4e278a323800/src/openMVG/multiview/triangulation_nview.cpp#L27

[Ntweat]

Update:
make test in Debug mode (both master and develop) fails.

[Ntweat]

Solved:
Used -mavx flag while compiling

[Ntweat]

I was able to automatically scale successfully.
I am not doing it inside openMVG, but using the scale later. That is, after creating the mesh (with openMVS).
The whole thing is a mix of python (to read the json file), C++ and meshlab scripts.

Thank you @pmoulon for all your help

[pmoulon]

When you are using develop the compilation flag must come with the target automatically.
See here: https://github.com/openMVG/openMVG/blob/develop/src/openMVG/matching/CMakeLists.txt#L31

BTW, I don't understand why the test are failing on your computer.

Looking forward to see you back ;-)

[Ntweat]

I was compiling my code as a standalone and linking it with openMVG libraries. I was not using CMakeLists. To compile my code I had to use the flag.

[pmoulon]

Ok, using CMake is advised and can avoid trouble like this ;-)