Odm: Overflow with larger image sizes with odm_orthophoto

[smathermather]

How did you install OpenDroneMap? (Docker, natively, ...)?

Natively

What's your browser and operating system? (Copy/paste the output of https://www.whatismybrowser.com/)

irrelevant

What is the problem?

Line 512 of modules/odm_orthophoto/src/OdmOrthoPhoto.cpp causes issues with sufficiently large images, e.g. 52700 x 45976 but not e.g. 49510 x 38436.

Edit: the issue clearly isn't with that line, but with the allocation.

With the following message:

Couldn't allocate enough memory to render the orthophoto (50191x43787 cells = 8790853268 bytes). Try to increase the --orthophoto-resolution parameter to a larger integer or add more RAM.
Traceback (most recent call last):
  File "run.py", line 57, in <module>
    app.execute()
  File "/home/odm/ODM/stages/odm_app.py", line 92, in execute
    self.first_stage.run()
  File "/home/odm/ODM/opendm/types.py", line 464, in run
    self.next_stage.run(outputs)
  File "/home/odm/ODM/opendm/types.py", line 464, in run
    self.next_stage.run(outputs)
  File "/home/odm/ODM/opendm/types.py", line 464, in run
    self.next_stage.run(outputs)
  File "/home/odm/ODM/opendm/types.py", line 464, in run
    self.next_stage.run(outputs)
  File "/home/odm/ODM/opendm/types.py", line 464, in run
    self.next_stage.run(outputs)
  File "/home/odm/ODM/opendm/types.py", line 464, in run
    self.next_stage.run(outputs)
  File "/home/odm/ODM/opendm/types.py", line 464, in run
    self.next_stage.run(outputs)
  File "/home/odm/ODM/opendm/types.py", line 464, in run
    self.next_stage.run(outputs)
  File "/home/odm/ODM/opendm/types.py", line 464, in run
    self.next_stage.run(outputs)
  File "/home/odm/ODM/opendm/types.py", line 464, in run
    self.next_stage.run(outputs)
  File "/home/odm/ODM/opendm/types.py", line 445, in run
    self.process(self.args, outputs)
  File "/home/odm/ODM/stages/odm_orthophoto.py", line 89, in process
    '-outputCornerFile {corners} {bands}'.format(**kwargs))
  File "/home/odm/ODM/opendm/system.py", line 76, in run
    raise Exception("Child returned {}".format(retcode))
Exception: Child returned 1

What should be the expected behavior? If this is a feature request, please describe in detail the changes you think should be made to the code, citing files and lines where changes should be made, if possible.

Big images: good.

How can we reproduce this? (What steps did you do to trigger the problem? What parameters are you using for processing? If possible please include a copy of your dataset uploaded on Google Drive or Dropbox. Be detailed)

Any sufficiently high-resolution dataset of large enough extent should trigger. Can share dataset offline if necessary.

The threshold is suspiciously near 2000000000....

[pierotofy]

It could be a problem of insufficient memory; how much RAM have you allocated (and is this a multispectral dataset?)

[ivangayton]

Hi Piero, it's Ivan (it's my dataset that Steve is helping manage). It's a Digital Ocean droplet with 182 GB of RAM and another 192 GB of swap.

Standard Ebee photos from the SODA camera (RGB).

[ivangayton]

@pierotofy I can share access to the machine if poking around it might shed light on the problem (that would also give you access to the dataset and console logs; we ran ODM using nohup and saved console logs).

[pierotofy]

Hey @ivangayton :hand: access to the machine would make it easier yes.

SSH key: https://gist.github.com/pierotofy/02f30a3ff2dcab7128042d4e7e32407c

[ivangayton]

Done, added you. Access by ssh [email protected]

Steve will send you the sudo password

[smathermather]

Hi Piero, it's Ivan (it's my dataset that Steve is helping manage). It's a Digital Ocean droplet with 182 GB of RAM and another 192 GB of swap.

Standard Ebee photos from the SODA camera (RGB).

Actually, I changed that to 300GB swap... . :)

[pierotofy]

What was the command (how was ODM invoked) that caused the error? (I see it's a split-merge workflow, so not sure which submodel failed).

Was it python run.py submodel_0000 --depthmap-resolution 1000 --orthophoto-resolution 5.4 --dem-resolution 15 --pc-las --dsm ?

[pierotofy]

Probably not;

Material 79 rendered.
Material 80 rendered.
Material 81 rendered.
... model rendered

...
Orthophoto generation done.

Let me know how to reproduce it.

[ivangayton]

No, that one succeeded! The original command was:

nohup python run.py BackupMsimbazi --split 0 --split-overlap 0 --ignore-gsd --depthmap-resolution 1000 --orthophoto-resolution 5 --dem-resolution 15 --pc-las --dsm &

That ran until submodel_0000, which crashed.
Error was:

[INFO]    Finished odm_dem stage
[INFO]    Running odm_orthophoto stage
[INFO]    running /home/odm/ODM/build/bin/odm_orthophoto -inputFiles /mnt/odmphotos/BackupMsimbazi/submodels/submodel_0000/odm_texturing_25d/odm_textured_model_geo.obj -logFile /mnt/odmphotos/BackupMsimbazi/submodels/submodel_0000/odm_orthophoto/odm_orthophoto_log.txt -outputFile /mnt/odmphotos/BackupMsimbazi/submodels/submodel_0000/odm_orthophoto/odm_orthophoto_render.tif -resolution 20.0  -outputCornerFile /mnt/odmphotos/BackupMsimbazi/submodels/submodel_0000/odm_orthophoto/odm_orthophoto_corners.txt 
Couldn't allocate enough memory to render the orthophoto (52700x45976 cells = 9691740800 bytes). Try to increase the --orthophoto-resolution parameter to a larger integer or add more RAM.
Traceback (most recent call last):
  File "/home/odm/ODM//run.py", line 57, in <module>
    app.execute()
  File "/home/odm/ODM/stages/odm_app.py", line 92, in execute
    self.first_stage.run()
  File "/home/odm/ODM/opendm/types.py", line 464, in run
    self.next_stage.run(outputs)
  File "/home/odm/ODM/opendm/types.py", line 464, in run
    self.next_stage.run(outputs)
  File "/home/odm/ODM/opendm/types.py", line 464, in run
    self.next_stage.run(outputs)
  File "/home/odm/ODM/opendm/types.py", line 464, in run
    self.next_stage.run(outputs)
  File "/home/odm/ODM/opendm/types.py", line 464, in run
    self.next_stage.run(outputs)
  File "/home/odm/ODM/opendm/types.py", line 464, in run
    self.next_stage.run(outputs)
  File "/home/odm/ODM/opendm/types.py", line 464, in run
    self.next_stage.run(outputs)
  File "/home/odm/ODM/opendm/types.py", line 464, in run
    self.next_stage.run(outputs)
  File "/home/odm/ODM/opendm/types.py", line 464, in run
    self.next_stage.run(outputs)
  File "/home/odm/ODM/opendm/types.py", line 464, in run
    self.next_stage.run(outputs)
  File "/home/odm/ODM/opendm/types.py", line 445, in run
    self.process(self.args, outputs)
  File "/home/odm/ODM/stages/odm_orthophoto.py", line 89, in process
    '-outputCornerFile {corners} {bands}'.format(**kwargs))
  File "/home/odm/ODM/opendm/system.py", line 76, in run
    raise Exception("Child returned {}".format(retcode))
Exception: Child returned 1
Traceback (most recent call last):
  File "run.py", line 57, in <module>
    app.execute()
  File "/home/odm/ODM/stages/odm_app.py", line 92, in execute
    self.first_stage.run()
  File "/home/odm/ODM/opendm/types.py", line 464, in run
    self.next_stage.run(outputs)
  File "/home/odm/ODM/opendm/types.py", line 445, in run
    self.process(self.args, outputs)
  File "/home/odm/ODM/stages/splitmerge.py", line 150, in process
    system.run(" ".join(map(quote, argv)), env_vars=os.environ.copy())
  File "/home/odm/ODM/opendm/system.py", line 76, in run
    raise Exception("Child returned {}".format(retcode))
Exception: Child returned 1

Then I ran all other submodels successfully (_0000 had the largest number of images). Steve then tried rerunning _0000 and it crashed until he set resolution to 5.4 (less than that and it still crashes)

[ivangayton]

So to reproduce @pierotofy try running submodel_0000 at resolution 5

[smathermather]

Correct. Run at 5.5 cm and it completes. Run at <5.4 cm and it fails quite quickly.

[pierotofy]

The fix in #1086 is already applied to the machine. Double-check if it works, then close?

Thanks!

(I was able to process it):

Material 79 rendered.
Material 80 rendered.
Material 81 rendered.
... model rendered

...
Orthophoto generation done.

In particular, check that the orthophoto looks normal (check for artifacts).

[smathermather]

Reviewed the patch: it looks good.