Openmvg: [geodesy] Add a geodesy module to OpenMVG

[pmoulon]

Will provide a basic coordinates for GPS registration.

Add ability to read and use EXIF GPS data

• [x] Add basic function to convert WGS84 lat,lon,alt from/to ECEF X,Y,Z value
• [x] Deal with EasxyExif uninitialized GPS (long,lat,alt) values (https://github.com/openMVG/openMVG/commit/f558720e37bb43fcfbbc32a034d93fc714b47487#commitcomment-17338526)
  
  
  
  • Enhance EasyExif library:
  
  
  • Since 0 is a valid value, use a tag to mark the GPS value as uninitialized (infinity)
  
  
• [x] Add an unit test for checking existence and validity of GPS data
• [x] Add a sample to show how to parse GPS (lat, long, alt) EXIF metadata & compute XYZ positions & export the resulting positions as a PLY points file.

Fix usage of large XYZ coordinates

• [x] Fix SfM_Data PLY logging (force fixed notation and change the precision to fit float precision). Avoid quantification effect when data are registered (since they can have large value once they are in a geodesy coordinates system)

Add a binary to perform the SfM_Data rigid registration to GPS metadata

The registration is performed by computing a 3D similarity:

• [x] from all points
• [x] thanks to a robust estimation method, to be robust to outlier GPS measure.

Pose center positions can be read from:

• [x] image EXIF metadata
• [x] from the View pose priors

Navigation prior

• add a new view type with prior data
  
  
  
  • [x] translation
  
  
  • rotation -> this feature will be delayed since using many translation will align rotations too.
  
  
• [x] Implement the usage of navigation prior in the BA
  
  
  
  • [x] Add unit test to test BA with navigation prior
  
  

Enhance image matching time

• [x] Add a new binary that can generate a matching pair file for main_ComputeMatches.
  
  
  
  • Add a -G mode to use the GPS pose center prior and a neighbor value to limit the number of pair to compare for putative matches (particularly useful with UAV like scene that have a single height flight-plane).
  
  

Documentation

• [x] Add new documentation for the geodesy module

• [x] merge to main develop branch

[pmoulon]

Ex of the sample binary added by https://github.com/openMVG/openMVG/commit/337980a4d2f33ce3b7dd049b5f5e0ff5406b7930 on the SenseFly Quarry dataset:

This tool allows to check the "UAV flight" path :+1:

[pmoulon]

Here a capture of the actual results:
Show the GPS metadata position + the registration & a measure

[pmoulon]

@ORNis @mitjap @morgand536 @rory @whuaegeanse => If you can test this feature I would be happy.
I have still to add a GPS/XYZ data file import (csv/txt file?) but the main feature is done.

[mitjap]

Awesome, will definitely test the new feature. If I checked correctly, at this moment you have implemented a rigid transformation to fit camera positions to exif GPS data. Do you plan to use GPS information when doing bundle adjustment with ceres?

[pmoulon]

@mitjap Yes I plan to use GPS prior for BA, still trying to figure out how I will implement it.
I think I will a new VIEW type by inheritance that will embed "navigation" prior (rotation/translation).

[json87]

hi,
it is an useful tools. and when does these modules could be accessed? @pmoulon

[pmoulon]

Hi @json87
Rigid registration is already available in the WIP develop_geodesy branch (but undocumented for the moment, but it just consist into running a binary: openMVG_main_geodesy_registration_to_gps_position)
Navigation prior will be implemented in the coming weeks.

[kafeiyin00]

hi,pmoulon,thanks for openning source of openmvg.
now, i use openmvg to handle the panoramic images. and i use the findSRT function to registrate the points to gps points to find the absolute position. because of the car route is just a line, so the transform is not a rigid transform.so i hope the next version you can add the gps information int BA to solve the problem

[pmoulon]

Hi @kafeiyin00
Thanks for your feedback.
I will make a first draft soon in order to support GPS data in the BA.

As you will see in this thread https://groups.google.com/forum/#!topic/ceres-solver/WCW2T-l7Dmg since the magnitude of residuals is different between the image re-projection error and the GPS-cam distance the settings can be tricky.
BTW, I will make a first draft and then we will improve it.
Any dataset sharing for testing & evaluation would be a great contribution (even a tiny scene).

[kafeiyin00]

thank you @pmoulon

[pmoulon]

@kafeiyin00 Now rigid & non rigid registration can be performed.
Here the documentation.

[kafeiyin00]

great work! i will test it soon

[whuaegeanse]

Hi @pmoulon
It's exciting to see Navigation prior module is in above plan. Does it all finished？

[pmoulon]

@whuaegeanse
Yes, please check the doc Here the documentation.
Soon I will merge the branch "develop_geodesy" to the develop branch to have this feature in the next release.

[whuaegeanse]

@pmoulon
Thanks. Great feature!
BA will be accelerated because of this feature. I will test it soon.
Thanks again.

[whuaegeanse]

@pmoulon
I have seen the module.
Is there any solution to accelerate the process Global SFM with the help of navigation prior info?
How about to combine the navigation prior info with the relative rations to get initial values of tracks's landmarks and camera poses?

[pmoulon]

@whuaegeanse

"Is there any solution to accelerate the process Global SFM with the help of navigation prior info?"

• Yes -> See the example in the documentation openMVG_main_ListMatchingPairs -G -n 8 is used to reduce the number of pair for image matching. It make the process faster since only 8 pair per view will be used.

"How about to combine the navigation prior info with the relative rotations to get initial values of tracks's landmarks and camera poses?"

• You can think about skip the relative translations estimation and translation averaging step in GlobalSfM and configure directly the pose prior as pose center. Then, initialize the structure with the pairwise correspondences as tracks and perform a blind triangulation & BA with it. Feel free to test this possibility. (Note that results will be dependent of your prior accuracy).

[whuaegeanse]

@pmoulon
Thanks.
I will try this in next week.
If each image has a prior pose , the camera's pose can be replaced by the the prior pose.
With non rigid prior, the poses of outliers images should be computed from the inliers's poses with the help of the relative rotations .After this step, the initial poses and the tracks can be got .

[sunmengmeng]

Hi, my images have no exif GPS data,so does it mean I can't use this module.

Now I have GPS txt file ,how can I use it to realize the same function?

[pmoulon]

You can either use a script to add your GPS data in the corresponding read EXIF field or modify the existing SfMImageListing binary to look for your corresponding GPS data in your text file.

[slembcke]

Just wanted to say that this is fantastic, and exactly what I needed. I've been trying to figure out how to use OpenMVG to increase the telemetry accuracy of an existing flight, and this looks like it might help me calculate exactly what I needed to know.

The OpenMVG code has been very nice to work with so far!

[pmoulon]

Thank you @slembcke for your enthusiasm!
I invite you to take a look to the step by step example provided here
Yes, please check the doc Here the documentation.

[slembcke]

Thanks. I was able to figure out some of that from just guessing and browsing around, but that's super helpful.

Are you interested in having more datasets for testing? I can probably give you some from our DJI flights that have GPS in the exif. They are sort of large though.

[pmoulon]

You can share your dataset (1 or 2) to openmvg-team[AT]googlegroups.com (a download link for example)
Will sun some expriment on it and make you a feedback.

[jpwhitney]

Hi,

pmoulon commented on Nov 21
Thank you @slembcke for your enthusiasm!
I invite you to take a look to the step by step example provided here
Yes, please check the doc Here the documentation.

This link no longer works.

Is there a step by step example elsewhere?

[pmoulon]

Since I have merged the modification in develop branch for future integration in the next release you can find the information here

Look at the end for the step by step example that reduce the pair count and use the non rigid registration.

[jpwhitney]

Thanks, so far so good. Found one small typo here at the beginning of the example.

// Initialize the scene
// Pose prior for each view will be set thanks to the GPS data
openMVG_main_SfMInit_ImageListing \
-P \
-d sensor_width_camera_database.txt \
-i /media/pierre/SenseFly/Quarry/geotagged-images/ \
-o /media/pierre/SenseFly/Quarry/quary_output/

I believe the -P is spurious, as when I run the above I get the message:

Unrecognized option -P

[pmoulon]

Hi @jpwhitney
-P must work since it is accessible here, this is q switch option
https://github.com/openMVG/openMVG/blob/develop/src/software/SfM/main_SfMInit_ImageListing.cpp#L144

[jpwhitney]

Sorry I think I have old code from the develop branch. However upon attempts at recompilation I reach about 70% and it fails.

Currently I'm trying to get the windows compilation environment going so I can try there instead of Linux.

[pmoulon]

Which compilation error do you have on your Linux?

• build on for GCC and the CI is ok.
• build on windows must be ok too according the CI

[jpwhitney]

I have attached the exact steps taken to reproduce this problem:
build_fail.txt

23:41:43 [App ] CPU: Intel(R) Core(TM) i7-4700MQ CPU @ 2.40GHz
23:41:43 [App ] RAM: 15.59GB Physical Memory 15.91GB Virtual Memory
23:41:43 [App ] OS: Linux 4.4.0-57-generic (x86_64)
23:41:43 [App ] SSE & AVX compatible CPU & OS detected
23:41:43 [App ] Command line: -h
23:41:43 [App ] Available options:

It gets to here:

[ 67%] Building CXX object openMVG/matching/CMakeFiles/openMVG_matching.dir/regions_matcher.cpp.o
In file included from /home/sdsu/openMVG/src/third_party/eigen/Eigen/Core:276:0,
                 from /home/sdsu/openMVG/src/openMVG/types.hpp:10,
                 from /home/sdsu/openMVG/src/openMVG/matching/indMatch.hpp:11,
                 from /home/sdsu/openMVG/src/openMVG/matching/matching_interface.hpp:11,
                 from /home/sdsu/openMVG/src/openMVG/matching/matcher_brute_force.hpp:10,
                 from /home/sdsu/openMVG/src/openMVG/matching/regions_matcher.cpp:8:
/home/sdsu/openMVG/src/third_party/eigen/Eigen/src/Core/Functors.h:973:28: warning: ‘template<class _Operation> class std::binder2nd’ is deprecated [-Wdeprecated-declarations]
 struct functor_traits<std::binder2nd<T> >
                            ^
In file included from /usr/include/c++/5/bits/stl_function.h:1128:0,
                 from /usr/include/c++/5/string:48,
                 from /usr/include/c++/5/bits/locale_classes.h:40,
                 from /usr/include/c++/5/bits/ios_base.h:41,
                 from /usr/include/c++/5/ios:42,
                 from /usr/include/c++/5/istream:38,
                 from /usr/include/c++/5/sstream:38,
                 from /usr/include/c++/5/complex:45,
                 from /home/sdsu/openMVG/src/third_party/eigen/Eigen/Core:28,
                 from /home/sdsu/openMVG/src/openMVG/types.hpp:10,
                 from /home/sdsu/openMVG/src/openMVG/matching/indMatch.hpp:11,
                 from /home/sdsu/openMVG/src/openMVG/matching/matching_interface.hpp:11,
                 from /home/sdsu/openMVG/src/openMVG/matching/matcher_brute_force.hpp:10,
                 from /home/sdsu/openMVG/src/openMVG/matching/regions_matcher.cpp:8:
/usr/include/c++/5/backward/binders.h:143:11: note: declared here
     class binder2nd
           ^
In file included from /home/sdsu/openMVG/src/third_party/eigen/Eigen/Core:276:0,
                 from /home/sdsu/openMVG/src/openMVG/types.hpp:10,
                 from /home/sdsu/openMVG/src/openMVG/matching/indMatch.hpp:11,
                 from /home/sdsu/openMVG/src/openMVG/matching/matching_interface.hpp:11,
                 from /home/sdsu/openMVG/src/openMVG/matching/matcher_brute_force.hpp:10,
                 from /home/sdsu/openMVG/src/openMVG/matching/regions_matcher.cpp:8:
/home/sdsu/openMVG/src/third_party/eigen/Eigen/src/Core/Functors.h:977:28: warning: ‘template<class _Operation> class std::binder1st’ is deprecated [-Wdeprecated-declarations]
 struct functor_traits<std::binder1st<T> >
                            ^
In file included from /usr/include/c++/5/bits/stl_function.h:1128:0,
                 from /usr/include/c++/5/string:48,
                 from /usr/include/c++/5/bits/locale_classes.h:40,
                 from /usr/include/c++/5/bits/ios_base.h:41,
                 from /usr/include/c++/5/ios:42,
                 from /usr/include/c++/5/istream:38,
                 from /usr/include/c++/5/sstream:38,
                 from /usr/include/c++/5/complex:45,
                 from /home/sdsu/openMVG/src/third_party/eigen/Eigen/Core:28,
                 from /home/sdsu/openMVG/src/openMVG/types.hpp:10,
                 from /home/sdsu/openMVG/src/openMVG/matching/indMatch.hpp:11,
                 from /home/sdsu/openMVG/src/openMVG/matching/matching_interface.hpp:11,
                 from /home/sdsu/openMVG/src/openMVG/matching/matcher_brute_force.hpp:10,
                 from /home/sdsu/openMVG/src/openMVG/matching/regions_matcher.cpp:8:
/usr/include/c++/5/backward/binders.h:108:11: note: declared here
     class binder1st
           ^
In file included from /home/sdsu/openMVG/src/openMVG/features/regions_factory.hpp:33:0,
                 from /home/sdsu/openMVG/src/openMVG/features/image_describer_akaze.hpp:16,
                 from /home/sdsu/openMVG/src/openMVG/features/features.hpp:16,
                 from /home/sdsu/openMVG/src/openMVG/matching/indMatchDecoratorXY.hpp:11,
                 from /home/sdsu/openMVG/src/openMVG/matching/regions_matcher.hpp:13,
                 from /home/sdsu/openMVG/src/openMVG/matching/regions_matcher.cpp:11:
/home/sdsu/openMVG/src/dependencies/cereal/include/cereal/types/polymorphic.hpp:100:3: error: expected constructor, destructor, or type conversion before ‘namespace’
   namespace cereal {                                         \
   ^
/home/sdsu/openMVG/src/openMVG/features/regions_factory.hpp:37:1: note: in expansion of macro ‘CEREAL_REGISTER_TYPE_WITH_NAME’
 CEREAL_REGISTER_TYPE_WITH_NAME(openMVG::features::AKAZE_Float_Regions, "AKAZE_Float_Regions");
 ^
In file included from /home/sdsu/openMVG/src/openMVG/features/features.hpp:16:0,
                 from /home/sdsu/openMVG/src/openMVG/matching/indMatchDecoratorXY.hpp:11,
                 from /home/sdsu/openMVG/src/openMVG/matching/regions_matcher.hpp:13,
                 from /home/sdsu/openMVG/src/openMVG/matching/regions_matcher.cpp:11:
/home/sdsu/openMVG/src/openMVG/features/image_describer_akaze.hpp:119:33: error: expected declaration before end of line
openMVG/matching/CMakeFiles/openMVG_matching.dir/build.make:62: recipe for target 'openMVG/matching/CMakeFiles/openMVG_matching.dir/regions_matcher.cpp.o' failed
make[2]: *** [openMVG/matching/CMakeFiles/openMVG_matching.dir/regions_matcher.cpp.o] Error 1
CMakeFiles/Makefile2:2617: recipe for target 'openMVG/matching/CMakeFiles/openMVG_matching.dir/all' failed
make[1]: *** [openMVG/matching/CMakeFiles/openMVG_matching.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

[pmoulon]

Thx for the log sharing.
Which Linux platform and GCC version are you using?

[rperrot]

Just a question. Did you update the submodules after updating the develop branch ? (I had similar issue around the same lines, updating the submodules solve the issue).

Do a clean repository (i.e.: just after a clone) compiles fine ?

[jpwhitney]

@pmoulon
Ubuntu 16.04.1 LTS
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4)

@rperrot I believe that it would compile fine, however I believe I need the develop branch to use the geodesy functions. I go through the steps from a fresh git clone to an attempted compilation of the develop branch and if you look at the build_fail.txt file in my previous comment you will see the exact steps taken and should answer your first question which to be honest I don't fully understand.

When you say update the develop branch do you mean "git checkout develop"?

I updated the submodules after cloning. If I am doing something strange in that process please advise.
I tried to follow the directions of pmoulon in these comments on this recent issue.

pmoulon commented 2 days ago
Can you try the code that is in the develop branch?

git checkout develop
git submodule init
git submodule
then rerun cmake and build from scratch

Here are the relevant command lines in the order I ran them, I know the log file is hard to look at:

$ git clone --recursive https://github.com/openMVG/openMVG.git
$ cd openMVG/

$ git checkout develop
$ git submodule init
$ git submodule

$ cd ..
$ mkdir openMVG_Build
$ cd openMVG_Build
$ cmake -DCMAKE_BUILD_TYPE=RELEASE . ../openMVG/src/

@rperrot What additional lines are you suggesting I run?

[pmoulon]

Do git submodule update rather than just git submodule
my fault, I must update the other post ;-)

[jpwhitney]

Ohhh, okay, that makes sense now with what @rperrot was saying. I'm still learning what all this stuff means. Thanks guys!

[pmoulon]

@lixx2938 You will see that you can use GPS data as a non rigid prior for Bundle Adjustment in OpenMVG.
See here for some documentation.

[pmoulon]

@is03wlei, @lixx2938 You can use non rigid registration to GPS data. The prior can be used to reduce the pair to match, and during the BA. You can also perform a rigid alignment if you want.See the previous post for the doc.