Openmvg: Is there state-of-art Descriptor and dense Correspondence algorithm in OpenMVG?

[wguo68]

I found there are some good dense Correspondence algorithm which wil make SFM in OpenMVG more robust and suitable for more kinds of objects such as human face. I didn't see these in OpenMVG.
Is there any plan to do this?

[rperrot]

Could you give us more information, tests and results about you descriptor ? It sounds interesting.

openMVG uses SIFT or AKAZE as full feature pipeline, proposes FAST, MSER and TBMR as detector LIOP or "Dissociated Dipole" as descriptor. All of these methods are general purpose and seems to corresponds to some of the state of art methods in the community. You're right that we're lacking some descriptors (binary descriptors like BRIEF/BRISK), we're also lacking time to do it ;-) (remember openMVG is not funded by any organisation, and code is written by people on their spare time)

BTW, it seems that you've implemented something robust, let's try to work together to include it in openMVG ?

[pmoulon]

You can also take a look to DAISY (its a SIFT like descriptor that is efficient to compute on a whole image).
Or take a look to KVLD (a graph based feature matcher) (see the sample included in OpenMVG).

PS: you can also take a look at the "new" approaches: "End-to-end 3D face reconstruction with deep neural networks"

[wguo68]

Yes. I am talking about things such as DAISY. Is it better than SIFT? I am not sure and didn't test. I tested deep matching and also not sure it is better than SIFT. The author told me EpicFlow(https://thoth.inrialpes.fr/src/epicflow/) is better for dense correspondence . What about your opinion?

[rperrot]

There's no closed answer related to descriptors.

1) There is no descriptor that is better than all the others in all the situations ;

2) (for now), you must use a descriptor for your application.

• What are the structure you want (description of a key point, of a region, dense, sparse ) ?
• Make a list of invariant you want (rotation, scale, lighting, ...) ?
• Decide if speed is relevant ?
• Do you allow patented descriptor ?

3) Daisy is designed to work as dense description (i.e.: one descriptor per pixel), this is different from sift (one descriptor per key point). The goal is not the same, so you cannot compare it without context. Daisy is known to work well for MVS, but you need calibrated rig (which is better handled with sift).

4) Deep learned descriptor are highly active topic and may be ill conditioned for your specific problem. As all learning method it highly depends on the learning set. If the set is absolutely different from the specific task you want to apply it, it will not work.

4) EpicFlow is mainly designed to compute optical flow, while this is close, it is not exactly the same context of descriptor we use.

[wguo68]

Thank you pmoulon and rperrot.

Why Daisy need calibrated rig?

My problem is to do 3d Face reconstruction and I need enough correspondence among a few images so that I can do SFM to find their poses. I don't if the 68 landmarks is enough because contour landmarks will not useful and some faces are not facing-front.

If I can do SFM using landmarks, then I can try another method to reconstruct 3d face similar to MVS

[rperrot]

Why Daisy need calibrated rig?

We never said that.

I don't if the 68 landmarks is enough because contour landmarks will not useful and some faces are not facing-front.

I think you talked about 68 landmarks because it's what you can find on face regression ? You need to understand few things :

• These landmarks are far less precise and stable than sift or any key point you will find in general literature (they are linked in a sort of elastic structure so it's not image based but image AND model based) ;
• In (standard) SfM you need some invariance to make a good reconstruction. Among theses invariance, you need rigid objects (i.e.: they doesn't move regarding the time of the capture process). Using faces, it's difficult to achieve this staticness (don't imagine any standard SfM could reconstruct face of a person talking, you need other methods). There are some variation of SfM methods like non-rigid SfM that alleviate this requirement. However openMVG doesn't (yet ?) include non rigid SfM methods. It would be great to have non rigid SfM in openMVG and you seem interested, feel free to propose a PR, it would be a great thing for the community !

[wguo68]

I found this site :http://www.kind-of-works.com/CODE_matching.html . Their method may produce more feature matchings. What do you think about it? I want to use it for ASIFT feature matching. But the program can't out matching.txt as he said. And I can't use their lib. Is there anyone can implement a algorithm similar to the method? I think it will be help to texture less object such as human faces.

[wguo68]

Hi, rperrot:
I know some people have used the mobile phone and MVS do human face reconstruction. I also want to do this. Please see this link and his paper: https://www.youtube.com/watch?v=xXfeja8evJk&t=29s&spfreload=10. The problem is I can't get good result as his. I think I need more feature matching between two images so that I can get good SFM result and I found this link :http://www.kind-of-works.com/CODE_matching.html . But I can't use their exe and lib to do feature matching as he said in this video: https://www.youtube.com/watch?v=xXfeja8evJk&t=29s&spfreload=10. No out file "matching.txt" if I run his program: "CODE -pwis -rept -indb image.txt".

[pmoulon]

The problem is not really the baseline but the lack of feature on the face.

ASIFT can help since it will simulate some image orientation that do not have.

• You can try ASIFT online here http://www.ipol.im/pub/art/2011/my-asift/ on the online demo (you can submit your image) and see how it react.

• CODE is interesting (and my wish list) but it will only provide you an alternative pre-segmentation of the corresponding features (ransac will still be required).

[wguo68]

Thank you.

I think CODE can provide more feature matchings which will make SFM works better. With good SFM result (good camera poses) ,I think I can do good 3d face reconstruction.

I have tried ASIFT online and the result is :

Do you think is it okay to do SFM using ASIFT(SIFT seems has no enough matchings)? Should I use only the face region to do SFM or use the whole scene to do SFM? How to use ASIFT in OpenMVG?

[pmoulon]

Thanks for the sharing.
Using point in all the static scene will help the SfM registration.

In order to use ASIFT matching in OpenMVG you must create features files and matches files.

• For each images you must export all the features points -> imageName.feat,
• Then for each pair you must export the matches as corresponding feature index -> matching.f.txt.

Then you run directly Incremental SfM on the produced file (since matches are already established you don't need to compute the matches).

[wguo68]

Thank you. Can I just export all the features points for each image and let OpenMVG to do matching ?

[pmoulon]

Yes, you need to export feature points and descriptors.

[wguo68]

Thanks

[pmoulon]

Will close the issue since we was able to find some method that can compute matches on your image dataset.