Openmvg: Adding views to an existing reconstruction

[sebastienchappuis]

Hello,

First, thank you for this awesome software and for making it opensource !

I was wondering if there was a way to add cameras/views to an already reconstructed dataset.
Basically, if I have a reconstruction of a 1000 images and I want to add a 100 images to the scene, is it possible to do so without having to recompute the first 1000 camera resections ?
My question is more relevant for sequential reconstruction than global reconstruction I guess.

Thanks !

[pmoulon]

In order to localize some new images inside a reconstruction you can use the Localization module.
See the $ openMVG_main_SfM_Localization binary.
It will try to find corresponding 2D-3D matches to the existing scene structure and to compute camera resection (it will not add new 3D points).
https://openmvg.readthedocs.io/en/latest/software/localization/localization/

The tool take only one image as input, but you can easily change it to take a series of images or a directory.

Feel free to try it a to continue the discussion and close the discussion once you tested it with success.
Feel also free to help in order to enhance the localization module and it's usage.

[sebastienchappuis]

Thanks !

I'll do a few tests and come back to you with my results.

[pmoulon]

I forgot to mention that for the moment only a PLY file is logged, I can work more with you next week to extend the binary to add the View to the existing scene and so complete the starting sfm_data with the new poses & intrinsic & view information.

[sebastienchappuis]

Ok, so I tested the Sfm_Localization_Single_3DTrackObservation_Database localizer used in the openMVG_main_SfM_Localization binary. I wrote an alternative binary that takes multiple images as input and outputs a new sfm_data file containing both the original and the newly computed poses/views/intrinsics. All seems to work well. Once I get all the new poses from the localizer I can run something like openMVG_main_ComputeStructureFromKnownPoses on the dataset in order to get some reconstruction from all the cameras together.

Now of course, this is not equivalent to resuming sequential reconstruction where it was. In order to get closer to this I would need to add a triangulation step and a BA step after each new camera resection. I will start implementing something in this direction. Maybe you have some advices on how to approach this ?

Thanks for your help !

[pmoulon]

@sebastienchappuis Sorry for my late answer.

If you want extend the scene, the task is not trivial, since you have also to update the localization interface at each round.

I would like to do something like this as a first draft:

1. localize the image or a round of images
2. compute to which view the new image(s) are linked => build a list of pair
3. compute guided matches for those pairs
4. triangulate the tracks (check you only add new one)
5. rebuild the localizer interface and loop over 1.

Hoping it can provide inspiration for you.

BTW, would you be interested into contributing back your update binary that is taking multiple images as input?

[sebastienchappuis]

@pmoulon Thank you for your answer.

Yes I figured it would not be that straight forward to implement. I tried to work around the problem by modifying the SequentialSfMReconstructionEngine to be able to work also from an existing reconstruction but I did not get any successful results yet. I will try to implement something a bit more in the line of your suggestion in the next days. Ill keep you updated of course.

BTW, would you be interested into contributing back your update binary that is taking multiple images as input?

Sure ! What is the best way to do this ? Do I fork openMVG and then submit a PR to the develop branch ? I would have to do some cleanup first but it should not take me too much time.

[pmoulon]

Best way for contributions:

• fork,
• build a new branch from develop with the name of your feature (i.e develop_localization_multiple_images)
• update the code, (optional: squash the commit if many and if the squashing is feasible)
• then do a PR where the community can comment/test.
• once the code review and modification was done the merging can be considered.

[pmoulon]

@sebastienchappuis
Any feedback about progress or help request on this?

[sebastienchappuis]

Hello!

Sorry for not giving you an update sooner but I did not work as much as I originally wanted to on this problem last week.

Nonetheless, I have made some good progress recently. As already mentioned I tried to approach the problem from two different sides :

1. by following your suggestion above, i.e. creating an executable that rebuilds a localizer interface every time I add a new image to the reconstruction.
2. by adding a few methods in the SequentialSfMReconstructionEngine class designed to grow an existing reconstruction. In short, I call from an executable a SequentialSfMReconstructionEngine::ProcessGrow method that differs a bit from the Process method. Instead of the usual InitLandmarkTracks -> FindInitialPair and MakeInitialPair3D -> Resection/BA for all images left to be reconstructed, ProcessGrow does something like GrowLandmarkTracks -> Resection/BA for all images left to be reconstructed where GrowLandmarkTracks grows the tracks coming from the landmarks of the initial scene with the extended matches computed with the new images. This way, the tracks ids coming from the reconstruction are the same as the tracks ids considered by the reconstruction engine.

As of today, I managed to make only the second approach work. I am not sure it is the best approach for this problem as it implies modifying things deeper into the code but it surely has its advantages. I still have to do some more testing with it but it seems to be pretty good.

For both approaches, I struggle the most with fusing tracks coming from the original reconstruction with new/expanded tracks coming from either a match file or a Image_Localizer_Match_Data object. Typically this problem is still not solved for the first approach. In the second approach « GrowLandmarkTracks » does this but it is still a bit clunky and not super safe.

Hopefully I will have some time to work a bit more on this in the next days ! I would definitely appreciate your opinion on this in the meantime.
Thanks !

[pmoulon]

Hi @sebastienchappuis,
I was thinking about this modification:

• " I wrote an alternative binary that takes multiple images as input and outputs a new sfm_data file containing both the original and the newly computed poses/views/intrinsics."

I will look more to your message and answer to it later.

[sebastienchappuis]

@pmoulon ah yes, this is pretty much ready, I will make a PR in the next days.

[happyfrank]

Hello @pmoulon. I have a reconstruction of Spherical images and I want to add one Pinhole image(same scene captured) to the scene. Do you think that can be achieved?

[pmoulon]

@happyfrank You can use the main_SfM_Localization binary to find where they are.

With this binary you can either try to localize pinhole or spherical images (use the develop branch).

Sidenote:
You can also perform the whole SfM stage with the pinhole and the spherical images (as you can see here https://plus.google.com/photos/108029268028983704703/album/6497284784744122273/6497284784510244418). For this step you just need to create the right sfm_data.json that describe your scene with the two camera intrinsic group (One for your spherical images, one for your pinhole images).

[happyfrank]

Dear @pmoulon thank you. It helps me a lot. ^v^

[pmoulon]

Impatient to see you share a screenshot of your reconstruction if you can ;-)

[happyfrank]

Dear @pmoulon, sorry for the late reply. Today I test it. The result is brilliant.
Thank your for provide this amazing feature.

Pinhole image:

spherical image:

result:

[happyfrank]

Hello @sebastienchappuis. I have the same issue as you said.

For both approaches, I struggle the most with fusing tracks coming from the original reconstruction with new/expanded tracks coming from either a match file or a Image_Localizer_Match_Data object. Typically this problem is still not solved for the first approach. In the second approach « GrowLandmarkTracks » does this but it is still a bit clunky and not super safe.

I just test the first approach. Now I wanna test the Second approach.
By the way, can you share the GrowLandmarkTracks code?
Thanks. ^v^

[sebastienchappuis]

Hello @happyfrank !
Unfortunately, I looked into it and it seems that the code I wrote at the time is lost in limbo (that was almost 2years ago !)...It's probably not much of a loss since it was very prototype-y.
Hopefully it should not be too difficult for you to rewrite this, please ask if you need some advices!

Cheers

[happyfrank]

@sebastienchappuis That's all right. I already rewrite it. Thanks. ^v^