Openmvg: Incremental and Global SFM.

[naveenjafer]

Hi, I have started using the application for a sparse point cloud reconstruction and I seem to be running into a few issues.
I am pretty sure that the geometric matches has been formed as expected, I had used a set of sequential images of an object and the adjacency matrix of the geometric matches more or less has matches along the diagonal of the matrix, which is what would be expected if the images were in sequence.

However, I am running into issues with running the Incremental SFM. Out of the 67 high quality images that I have used, after multiple runs without setting any seed initial pairs it always only gives me 2 camera poses and re-sectioning fails for almost all the images that have been listed in the report. Viewing the mesh in meshlab always gives me all the point(both structure and camera points) in the same plane. I have re ran this experiment at least 10 times and I run into similar behaviour everytime.
Surprisingly the global SFM on the other hand just fails with a "Invalid matches file".

All images of mine were taken with one single camera and they have all been mapped to one single intrinsic mapped with the key '0', the details are as follows.

"intrinsics": [
{
"key": 0,
"value": {
"polymorphic_id": 2147483649,
"polymorphic_name": "pinhole_radial_k3",
"ptr_wrapper": {
"id": 2147483716,
"data": {
"width": 2992,
"height": 2000,
"focal_length": 1.0,
"principal_point": [
1496.0,
1000.0
],
"disto_k3": [
0.0,
0.0,
0.0
]
}
}
}
}
],

Now, when I ran the same program with a explicit initial pair which had good matches, I managed to receive an output with 4 camera poses and much more 3D points detected than earlier and also a couple of camera point's resectioning's passed OK. Would you happen to know what could cause this issue with global SFM when the incremental SFM is working fine? Any help will be appreciated.

[naveenjafer]

This is one of the final report of the sequential SFM.

SequentialSfMReconstructionEngine
Dataset info:Views count: 67

Essential Matrix.

-- Robust Essential matrix: <2,3> images: DSC_3786.JPG,DSC_3787.JPG
-- Threshold: 1.53267
-- Resection status: OK
-- Nb points used for robust Essential matrix estimation: 1048
-- Nb points validated by robust estimation: 899

-- % points validated: 0.857824

Residual of the robust estimation (Initial triangulation). Thresholded at: 1.53267
Histogram of residuals

Resection of Image index: <1> image: DSC_3785.JPG

-- Robust Resection of camera index: <1> image: DSC_3785.JPG
-- Threshold: 0.168325
-- Resection status: OK
-- Nb points used for Resection: 861
-- Nb points validated by robust estimation: 463

-- % points validated: 0.537747

Resection of Image index: <0> image: DSC_3784.JPG

-- Robust Resection of camera index: <0> image: DSC_3784.JPG
-- Threshold: 0.135572
-- Resection status: OK
-- Nb points used for Resection: 809
-- Nb points validated by robust estimation: 334

-- % points validated: 0.412855

Resection of Image index: <19> image: DSC_3803.JPG

-- Robust Resection of camera index: <19> image: DSC_3803.JPG
-- Threshold: 1.10417e-08
-- Resection status: FAILED
-- Nb points used for Resection: 776
-- Nb points validated by robust estimation: 0

-- % points validated: 0

Resection of Image index: <18> image: DSC_3802.JPG

-- Robust Resection of camera index: <18> image: DSC_3802.JPG
-- Threshold: 6.86143e-08
-- Resection status: FAILED
-- Nb points used for Resection: 750
-- Nb points validated by robust estimation: 0

-- % points validated: 0

Resection of Image index: <20> image: DSC_3804.JPG

-- Robust Resection of camera index: <20> image: DSC_3804.JPG
-- Threshold: 1.44583e-06
-- Resection status: FAILED
-- Nb points used for Resection: 711
-- Nb points validated by robust estimation: 0

-- % points validated: 0

Resection of Image index: <47> image: DSC_3837.JPG

-- Robust Resection of camera index: <47> image: DSC_3837.JPG
-- Threshold: 0.00162763
-- Resection status: FAILED
-- Nb points used for Resection: 130
-- Nb points validated by robust estimation: 0

-- % points validated: 0

Resection of Image index: <48> image: DSC_3838.JPG

-- Robust Resection of camera index: <48> image: DSC_3838.JPG
-- Threshold: 0.00200571
-- Resection status: FAILED
-- Nb points used for Resection: 78
-- Nb points validated by robust estimation: 0

-- % points validated: 0

Resection of Image index: <65> image: DSC_3855.JPG

-- Robust Resection of camera index: <65> image: DSC_3855.JPG
-- Threshold: 0.000942098
-- Resection status: FAILED
-- Nb points used for Resection: 73
-- Nb points validated by robust estimation: 0

-- % points validated: 0

Resection of Image index: <46> image: DSC_3836.JPG

-- Robust Resection of camera index: <46> image: DSC_3836.JPG
-- Threshold: 0.00639174
-- Resection status: FAILED
-- Nb points used for Resection: 46
-- Nb points validated by robust estimation: 0

-- % points validated: 0

Resection of Image index: <63> image: DSC_3853.JPG

-- Robust Resection of camera index: <63> image: DSC_3853.JPG
-- Threshold: 0.000346258
-- Resection status: FAILED
-- Nb points used for Resection: 45
-- Nb points validated by robust estimation: 0

-- % points validated: 0

Resection of Image index: <49> image: DSC_3839.JPG

-- Robust Resection of camera index: <49> image: DSC_3839.JPG
-- Threshold: 0.00722628
-- Resection status: FAILED
-- Nb points used for Resection: 25
-- Nb points validated by robust estimation: 0

-- % points validated: 0

Resection of Image index: <5> image: DSC_3789.JPG

-- Robust Resection of camera index: <5> image: DSC_3789.JPG
-- Threshold: 0.0103123
-- Resection status: FAILED
-- Nb points used for Resection: 24
-- Nb points validated by robust estimation: 0

-- % points validated: 0

Resection of Image index: <66> image: DSC_3856.JPG

-- Robust Resection of camera index: <66> image: DSC_3856.JPG
-- Threshold: 0.130689
-- Resection status: FAILED
-- Nb points used for Resection: 15
-- Nb points validated by robust estimation: 0

-- % points validated: 0

Resection of Image index: <22> image: DSC_3806.JPG

-- Robust Resection of camera index: <22> image: DSC_3806.JPG
-- Threshold: 0.267343
-- Resection status: FAILED
-- Nb points used for Resection: 15
-- Nb points validated by robust estimation: 0

-- % points validated: 0

Resection of Image index: <4> image: DSC_3788.JPG

-- Robust Resection of camera index: <4> image: DSC_3788.JPG
-- Threshold: 0.0111247
-- Resection status: FAILED
-- Nb points used for Resection: 14
-- Nb points validated by robust estimation: 0

-- % points validated: 0

Resection of Image index: <64> image: DSC_3854.JPG

-- Robust Resection of camera index: <64> image: DSC_3854.JPG
-- Threshold: 1286.71
-- Resection status: FAILED
-- Nb points used for Resection: 6
-- Nb points validated by robust estimation: 0

-- % points validated: 0

Resection of Image index: <6> image: DSC_3790.JPG

-- Robust Resection of camera index: <6> image: DSC_3790.JPG
-- Threshold: 866.152
-- Resection status: FAILED
-- Nb points used for Resection: 5
-- Nb points validated by robust estimation: 0

-- % points validated: 0

Resection of Image index: <45> image: DSC_3835.JPG

-- Robust Resection of camera index: <45> image: DSC_3835.JPG
-- Threshold: 0
-- Resection status: FAILED
-- Nb points used for Resection: 2
-- Nb points validated by robust estimation: 0

-- % points validated: 0

Resection of Image index: <24> image: DSC_3808.JPG

-- Robust Resection of camera index: <24> image: DSC_3808.JPG
-- Threshold: 0
-- Resection status: FAILED
-- Nb points used for Resection: 1
-- Nb points validated by robust estimation: 0

-- % points validated: 0

Structure from Motion process finished.

-- Structure from Motion (statistics):
-- #Camera calibrated: 4 from 67 input images.

-- #Tracks, #3D points: 1219

[naveenjafer]

@pmoulon I just realized something, the focal length is 1.0 according to the intrinsics that were calculated. Is this normal?

[naveenjafer]

I checked the actual intrinsics of the image using the exif utility in ubuntu and it gave me a focal length of 18 mm. The camera I used was listed in the camera width database, I would like to know how I should proceed if my camera is not listed in the database. Where exactly is this camera width information being used in the creation of the sfm_data.json?

[naveenjafer]

I managed to get rid of the Global SFM problem of "No matches" with this #586 . Now I am running into this. I used Visual SFM to try out a sparse point cloud reconstruction and there was no issue with it. I also ran computing features with "ULTRA" and still ended with similar results.

This is mostly not an issue of insufficient matches too, I have got 590 geometric matches with 67 images that have been given.

One thing consistent through all runs of the incremental SFM is that it always results in all points in a single plane. Could someone point me at some good dataset that I can use to check how the expected behaviour and values ideally would be?

Open Source implementation of the paper:
"Global Fusion of Relative Motions for Robust, Accurate and Scalable Structure from Motion."

Pierre Moulon, Pascal Monasse and Renaud Marlet. ICCV 2013.

• Features Loading -
  0% 10 20 30 40 50 60 70 80 90 100%
  |----|----|----|----|----|----|----|----|----|----|

---

CleanGraph_KeepLargestBiEdge_Nodes():: => connected Component: 1
Connected component of size: 67

• Relative pose computation -
  0% 10 20 30 40 50 60 70 80 90 100%
  |----|----|----|----|----|----|----|----|----|----|

---

---

Global rotations computation:
#relative rotations: 1
#global rotations: 2

Statistics about rotation triplets:

 min: 2.30089e-38
 mean: -3.57548e-22
 median: 4.58225e-41
 max: 0

Triplets filtering based on composition error on unit cycles

Triplets before: 0

Triplets after: 0

#Edges removed by triplet inference: 1

CleanGraph_KeepLargestBiEdge_Nodes():: => connected Component: 0
Found #global_rotations: 0
Timing: 0 seconds
GlobalSfM:: Rotation Averaging failure!

[rperrot]

Is it possible to share us the dataset ? Could you give use the command line you use ? Did you use the python scripts or all commands by hand ?

If your camera is not in the database, you could add it to the DB file (located at src/openMVG/exif/sensor_width_database/sensor_width_camera_database.txt), the width can be found in various sites, for example here : http://www.digicamdb.com

[pmoulon]

Your problem is related to the fact that an invalid focal length is set.
It's not normal that you have a focal length with a value equal to 1.

Does your camera model is set in the camera sensor database?

• else you can try to set the focal length value to -f=1.1*max(imageW,imageH) => -f 3291 in your case.

Once you have fixed the focal length value, everything will run fine, just reran ComputeMatches and the SfM (Incremental or Global).

You can play with this tiny dataset:
https://github.com/openMVG/ImageDataset_SceauxCastle

Or the collection that is saved here (big repo):
https://github.com/rperrot/ReconstructionDataSet

[naveenjafer]

Yeah my camera is added in the camera width database(I would also like to know what is to be done in case there is no info on camera model from exif information for any of the images in the dataset) and The exif information from one of the images is as follows(it is more or less the same for all of them in parameters of our interest, I dont get why the focal length is listed as 18mm in the database and is not anywhere close to the 3291 that was done from the estimation. Am i missing something here?

Manufacturer |NIKON CORPORATION
Model |NIKON D7100
Orientation |Top-left
X-Resolution |300
Y-Resolution |300
Resolution Unit |Inch
Software |Ver.1.01
Date and Time |2015:01:09 03:07:08
Artist |
YCbCr Positioning |Co-sited
Copyright |[None] (Photographer) - [None] (Editor)
Compression |JPEG compression
X-Resolution |300
Y-Resolution |300
Resolution Unit |Inch
YCbCr Positioning |Co-sited
Exposure Time |1/80 sec.
F-Number |f/4.5
Exposure Program |Not defined
ISO Speed Ratings |400
Exif Version |Exif Version 2.3
Date and Time (Origi|2015:01:09 03:07:08
Date and Time (Digit|2015:01:09 03:07:08
Components Configura|Y Cb Cr -
Compressed Bits per | 4
Exposure Bias |0.00 EV
Maximum Aperture Val|3.60 EV (f/3.5)
Metering Mode |Pattern
Light Source |Unknown
Flash |Flash did not fire, auto mode
Focal Length |18.0 mm
Maker Note |38940 bytes undefined data
User Comment |
Sub-second Time |10
Sub-second Time (Ori|10
Sub-second Time (Dig|10
FlashPixVersion |FlashPix Version 1.0
Color Space |sRGB
Pixel X Dimension |2992
Pixel Y Dimension |2000
Sensing Method |One-chip color area sensor
File Source |DSC
Scene Type |Directly photographed
CFA Pattern |8 bytes undefined data
Custom Rendered |Normal process
Exposure Mode |Auto exposure
White Balance |Auto white balance
Digital Zoom Ratio | 1
Focal Length in 35mm|27
Scene Capture Type |Standard
Gain Control |Low gain up
Contrast |Normal
Saturation |Normal
Sharpness |Normal
Subject Distance Ran|Unknown
GPS Tag Version |2.3.0.0
Interoperability Ind|R98
Interoperability Ver|0100

[naveenjafer]

Thank you, the global SFM gave very promising results.However I have a few things that need to cleared
So according to the code the calculation for the focal length is carried out as
focal = std::max ( width, height ) * exifReader->getFocal() / ccdw;

So in my case Nikon D7100;23.5 the ccdw value is 23.5? the focal length is 18mm.
I have 2 questions
1) So shouldnt the calculation for focal have been 0.76 * max(width, height)?
2) What is this focal that is being calculated?(what is the focal from the exif information then?). Excuse me for these doubts, I am a little new to these photography terms.

[pmoulon]

Since your camera D7100;23.5 is in the database the focal length must be well computed.
The exif focal length (is the focal in mm). In photogrammetry we use the focal length in pixels, so you need the following formula:

focal = std::max ( width, height ) * exifReader->getFocal() / ccdw;

So in your case 2992 * 18 / 23.5.

I wonder why your got this "focal_length": 1.0, in your initial test...

[naveenjafer]

I will try checking with some logs as to how it ended up that way, will keep you updated. I will close this issue once I have confirmed what the issue is.

[naveenjafer]

@pmoulon Is there a quicker way to rebuild the project to reflect any changes that I have made in the code? Rebuilding each time does not seem to be the right way going about it. TIA

[pmoulon]

I did not catch your last point. You can compile only one binary by sending it to the make command line.

make -j openMVG_main_SfMInit_ImageListing

[naveenjafer]

@pmoulon I figured out the issue that had happened with the focal length being set to 1. With huge commands being moved around I had accidentally used the "-f 1" option from compute features in this. Blunder on my part. But yeah this helped me understand a lot more now. Thank you. Yeah regarding the build, I was building everything all over again. Thanks for clearing that out. All my queries have been answered. :)

[pmoulon]

Impatient to see some of your reconstructions ;-)

[naveenjafer]

In the event the camera does not exist in the database, the intrinsic is mapped to some key "4294967295" and the entire intrinsic object is empty as expected. To tackle the same, I wanted to go ahead and add the camera sensor width to the database, but in the future if I plan on creating a automated system that could at least make approximations for the camera model and add it to the database as it proceeds with calculations. I came across this link for approximations, would this be a good way to go about it? I am basically interested in reconstruction primarily from smartphone cameras and the database seems to be missing most smartphone cameras information. If you do happen to know some database for smartphone cameras, could you point me to it? TIA

Edit: The EXIF Information of my images from a very good smartphone does not have what is required to estimate camera width unfortunately. I am kind of stuck on how to resolve this issue when the camera width is not already present.

[alkavan]

@pmoulon Hey, I have question about the focal calculation. I have a GoPro Hero 5 camera with the following exif data:

ExifTool Version Number         : 10.80
Camera Model Name               : HERO5 Black
Focal Length                    : 3.0 mm
Image Size                      : 4000x3000
Circle Of Confusion             : 0.005 mm
Field Of View                   : 93.3 deg
Focal Length                    : 3.0 mm (35 mm equivalent: 17.0 mm)
Hyperfocal Distance             : 0.61 m

Sensor Dimensions:

6.17 mm x 3.47 mm (0.243 in x 0.137 in)

So question is, what values should be used for the calculation?
focal = std::max (4000, 3000) * <3.0 or 17.0 ?> / 6.17; would that be the correct calculation?

[pmoulon]

As you see here:
https://github.com/openMVG/openMVG/blob/master/src/software/SfM/main_SfMInit_ImageListing.cpp#L355

focal = std::max ( width, height ) * exifReader->getFocal() / ccdw;

So you can use 4000 * 3.0 / 6.17 = 1944 as a starting point by using the -f 1944 option on SfMInit_ImageListing.

Since GoPro camera have a short focal length and some distortion, sometimes it's better to perform a tiny calibration and then provide the existing calibration to the pipeline.

Or you can also add HERO5 Silver;6.17 to the file https://github.com/openMVG/openMVG/blob/master/src/openMVG/exif/sensor_width_database/sensor_width_camera_database.txt#L1111

[alkavan]

@pmoulon So yeah, after adding the following value to the .txt file from link:

HERO5 Black;7.00

It worked, resulting:

"intrinsics": [
        {
            "key": 0,
            "value": {
                "polymorphic_id": 2147483649,
                "polymorphic_name": "pinhole_radial_k3",
                "ptr_wrapper": {
                    "id": 2147484619,
                    "data": {
                        "width": 4000,
                        "height": 3000,
                        "focal_length": 1714.2857142857143,
                        "principal_point": [
                            2000.0,
                            1500.0
                        ],
                        "disto_k3": [
                            0.0,
                            0.0,
                            0.0
                        ]
                    }
                }
            }
        }
    ],

and that's exactly like the value I calculated 1714.285714 before by hand. so everything seems okay now. thanks!