Odm: Rolling shutter correction

[kikislater]

It should be good to implement rolling shutter under ODM.

Link to have idea about process :
https://pix4d.com/rolling-shutter-correction/
http://www.eurocow2016.org/files/Vautherin-EuroCOW.pdf
https://pix4d.com/wp-content/uploads/2016/05/pix4d-isprs-paper-rolling-shutter-final-edited.pdf

[dakotabenjamin]

This is an interesting problem space. As I understand it, we could use a rolling shutter model (the equation in the last link) to adjust the images using the drone's velocity? That could be pulled from flight logs or simply by estimate.

I don't completely understand the math behind it, but would this translation (?) would happen during the orthophoto creation step?

[smathermather]

It seems like the translation affects everything from matching onward. If my intuition is correct, one could treat each scan as a separate image.

[KommandorKeen]

Any progress on this?
Happy to help.
Simon

[kikislater]

smathermather have a good suggestion : separate each frame of a rolling shutter in relation to shutter speed. It could be done with imagemagick.
What is difficult is to know sensor readout with sport camera like gopro or others ...

Example : with repage here =
http://www.fmwconcepts.com/imagemagick/tidbits/image.php#crop1
Crop Vertically
convert monet.jpg -crop 100%x50% +repage monet_vertical_%d.jpg

[smathermather]

It seems there's an opportunity to measure the rolling shutter effect. Is there any project, similar to lensfun for addressing the nature of rolling shutter for different cameras?

[kikislater]

updated post above with method in imagemagick.

IMO, very slow readout with rolling shutter like gopro @30ms sensor readout without stabilisation is problematic for photogrammetry.
Since CMOS sensor are rolling shutter without exception at this time. Like Sony A6000, Nikon D810, Canon 6D have a sensor readout of around 4ms or 1/250s, it seems not be an issue.
Small CMOS sensor have better readout like 1" on g9x or rx100 (may be the same sensor and of course same as parrot soda or dji phantom4 pro) or Canon S100, Sony WX220 around 0.5ms or 1/2000s

[yjmenezes]

if it acts as systematic effect, i think, for multirotor, it could be modeled by Imagemagick skew as a preprocessing stage, using a mean values of flight speed and height.
It should be done or not, depending of flight plan and camera type as pointed by @kikislater.

http://www.fmwconcepts.com/imagemagick/skew/index.php

[KommandorKeen]

My trawl of info:
https://pix4d.com/rolling-shutter-correction/
Lots of other sites focus on this as a source.

Looks like the DJI Phantom 4 has a 30 - 33 ms rolling shutter, so at 5m/s this takes 15cm along my line of flight, but at 10m/s this is 30cm

If I am flying at 100m with a 3000x4000 camera (94degFOV) my GSD is 5.3cm

Assuming that the time for the image taken is at the start of the image taking process and drone is flying oriented predominantly along track lines.

A quick fix may be to compress the vertical dimension of the image by my alongtrack speed / GSD and to offset the image by half the rolling shutter delay * along track speed

will play with the above and get back to you

Simon

[smathermather]

This is really interesting, @KommandorKeen. Do let us know what you discover.

[CarlosGrohmann]

This could be interesting: https://github.com/shimrcv/SfSM

[rionlerm]

@smathermather , @dakotabenjamin , @KommandorKeen any progress on this matter?

We might look into Parrot Sequoia+ sensor and it employs a rolling shutter. Before making this expensive purchase, we would like this issue #313 to be addressed before deciding to purchase Pix4D or remain with ODM.

I will be at a conference this week where the matter of which software to use with which expensive UAV hardware, will be discussed.

[kikislater]

Rolling shutter on sequoia(+) is only on RGB sensor. RGB sensor on sequoia doesn't make any value added output ...
Rolling shutter doesn't matter if you have mechanical shutter which is not the case of sequoia

[smathermather]

@smathermather , @dakotabenjamin , @KommandorKeen any progress on this matter?

We might look into Parrot Sequoia+ sensor and it employs a rolling shutter. Before making this expensive purchase, we would like this issue #313 to be addressed before deciding to purchase Pix4D or remain with ODM.

I will be at a conference this week where the matter of which software to use with which expensive UAV hardware, will be discussed.

For what you spend on Pix4D, you might be able to create some momentum behind adding this feature to ODM. :)

[smathermather]

And by "this feature" I mean this one: https://github.com/OpenDroneMap/OpenDroneMap/issues/500#issuecomment-423524570

[rionlerm]

@smathermather , @dakotabenjamin , @KommandorKeen any progress on this matter?
We might look into Parrot Sequoia+ sensor and it employs a rolling shutter. Before making this expensive purchase, we would like this issue #313 to be addressed before deciding to purchase Pix4D or remain with ODM.
I will be at a conference this week where the matter of which software to use with which expensive UAV hardware, will be discussed.

For what you spend on Pix4D, you might be able to create some momentum behind adding this feature to ODM. :)

Very true RE cost of Pix4D but, I don't have the skills currently to add this to ODM! Seems I might have some self-teaching skills coming up but would appreciate any help in order to improve ODM. Just difficult imagine that someone else using these types of sensors haven't added it yet.

[smathermather]

@smathermather , @dakotabenjamin , @KommandorKeen any progress on this matter?
We might look into Parrot Sequoia+ sensor and it employs a rolling shutter. Before making this expensive purchase, we would like this issue #313 to be addressed before deciding to purchase Pix4D or remain with ODM.
I will be at a conference this week where the matter of which software to use with which expensive UAV hardware, will be discussed.

For what you spend on Pix4D, you might be able to create some momentum behind adding this feature to ODM. :)

Very true RE cost of Pix4D but, I don't have the skills currently to add this to ODM! Seems I might have some self-teaching skills coming up but would appreciate any help in order to improve ODM. Just difficult imagine that someone else using these types of sensors haven't added it yet.

We take all sorts of contributions, including monetary ones, is what I am saying. I understand the skill-set limitation. Take a look at the code contributions. Mine are now overwhelmed by folks much better at this than I am.

This need is something that would also be beneficial to my organization and maybe some others. Reach out at svm[[[[[@]]]]]clevelandmetroparks.com and we can have a discussion about how we might finance it.

[Saijin-Naib]

From my prior usage of Pix4D with a 3DR Solo with GoPro Hero4 Black for agricultural field mapping (few thousand+ acres mapped), the usage of the rolling shutter compensation was really important. It made some really early flights (with sub-optimal overlap/sidelap) stitch properly, and for properly designed flights, greatly improved the quality of the point cloud and thereby the products (DSM, ortho, mesh) quite visibly. Accuracy measurements during the Pix4D accuracy report improved markedly, as well.

I'm a HUGE proponent of having this capability (likely on by default), if possible.

[kikislater]

Some recents documents on this subject :
http://openaccess.thecvf.com/content_cvpr_2018/papers/Lao_A_Robust_Method_CVPR_2018_paper.pdf

https://arxiv.org/pdf/1903.03943.pdf

It seems to be available in MicMac ...