openMVG speed decreasing when using machines > 32 cores

[ssampang]

Hi,

I'm trying to profile openMVG's performance on machines with varying numbers of cores. I noticed that performance degrades when using machines with more than 32 cores. For example, on a set of 147 images, feature extraction takes about 20 seconds when using 32 cores, but degrades to upwards of 1 minute with 64 cores, and even more with 128 cores.

In htop, it looks like 80-90% of each processor's time is being taken by the kernel (most of the process bar is red).

This is in contrast to running it on a 16 core machine, which looks like:

Any ideas what's going on when we exceed 32 cores?

Please let me know if there's additional info I can provide

[pmoulon]

Hi @ssampang

Thank you for your feedback.

1. Can you please tell us if you are using the branch master or develop?

2 (Threading Testing environment)

• If you are using develop I would advise you to use the -n X option on the ComputeFeatures binary. X is the number of core you want to use. It will be far more efficient than the default threading that is used.
  
  
  
  • Default threading mode run a single feature extractor and use threading inside it.
  
  
  • -n X will use X feature extractor and so will be more efficient but use more memory.
  
  

[ssampang]

Hi Pierre,

I'm using the master branch, but I'm also using the -n option to set the cores. I ran the ComputeFeatures binary on a 128 core machine, and simply varied the number of cores that ComputeFeatures used by supplying {16,32,64,128} to the -n option.

Also, I noticed that the performance issues (high number of kernel processing time) with other binaries as well (feature matching and IncrementalSfM).

[pmoulon]

I would like to suggest you to rerun your experiment by using the develop branch. I remember that we fixed an issue for threaded feature extractions.

I would suggest you the following:

1. re-run this experiment using the develop branch version?
2. since OpenMVG have various feature extraction method you can be interested to do this profiling for -m AKAZE_FLOAT and/or -m SIFT_ANATOMY and see if the behavior is the same.

The bottleneck in your case can be the disk (perhaps up to a limit the reading of the images cannot be done at the processing speed...)

PS: If we understand deeply what's going on the best way would be to run some profiling tools like Intel VTune to understand where are the bottlenecks.

[ssampang]

Will do that and get back to you. Thanks for the prompt response! Do you have any comments on debugging/accelerating later stages in the pipeline, since they a) don't have command line options for specifying cores and b) take the majority of processing time as well?

[pmoulon]

a) You can control the max number of cores by using the OPENMP_NUM_THREADS environment variable https://gcc.gnu.org/onlinedocs/libgomp/OMP_005fNUM_005fTHREADS.html

• If you don't have command line option you can modify the main binaries to call omp_set_num_threads(int num_threads); in the c++ client. https://gcc.gnu.org/onlinedocs/libgomp/omp_005fset_005fnum_005fthreads.html#omp_005fset_005fnum_005fthreads

b) any processing speed need to have a baseline. So I would suggest first to find the bottleneck (VTune analysis on a given dataset), share here the outcomes and so then we can work on understand what's going on and make the step faster or at least enhance its scalability ;-)

[pmoulon]

I would like to be curious to see the new performance test now that you are using the develop branch.
Did you made any progress on that?

[ssampang]

I haven't had time to do this yet - will try and do a simple test within the next week or so, but won't be able to look into VTune for maybe 1-2 months

[ssampang]

Moving to develop branch fixed this issue. Forgot to let you know here!