Odm: Segfault while merging depthmaps

[bstempi]

Hi,
I'm running the latest ODM Docker image (pulled last night) on a VPS running Debian with 24GB of RAM and 8 CPU cores. I'm currently trying to stitch together a survey that consists of around 270 photos, totaling 1-1.5GB. During processing, I get this:

2017-03-10 13:48:42,183 Cleaning depthmap for image DJI_0062.JPG
2017-03-10 13:48:43,404 Cleaning depthmap for image DJI_0021.JPG
2017-03-10 13:48:44,682 Cleaning depthmap for image DJI_0060.JPG
2017-03-10 13:48:46,072 Cleaning depthmap for image DJI_0162.JPG
2017-03-10 13:48:47,415 Cleaning depthmap for image DJI_0236.JPG
2017-03-10 13:48:52,428 Cleaning depthmap for image DJI_0050.JPG
2017-03-10 13:48:53,772 Cleaning depthmap for image DJI_0052.JPG
2017-03-10 13:48:55,157 Cleaning depthmap for image DJI_0228.JPG
2017-03-10 13:48:56,464 Cleaning depthmap for image DJI_0087.JPG
2017-03-10 13:48:57,821 Cleaning depthmap for image DJI_0124.JPG
2017-03-10 13:48:59,103 Cleaning depthmap for image DJI_0118.JPG
2017-03-10 13:49:00,459 Cleaning depthmap for image DJI_0079.JPG
2017-03-10 13:49:01,845 Cleaning depthmap for image DJI_0241.JPG
2017-03-10 13:49:06,852 Cleaning depthmap for image DJI_0051.JPG
2017-03-10 13:49:08,309 Merging depthmaps
Segmentation fault (core dumped)
Traceback (most recent call last):
File "/code/run.py", line 55, in
plasm.execute(niter=1)
File "/code/scripts/opensfm.py", line 90, in process
(context.pyopencv_path, context.opensfm_path, tree.opensfm))
File "/code/opendm/system.py", line 28, in run
raise Exception("Child returned {}".format(retcode))
Exception: Child returned 139
Any ideas? I'm fairly new to the project and I'm not sure where to dig in.

[dakotabenjamin]

Do you get this on a rerun as well? What is your docker command?

[bstempi]

I did, and it still fails.

Command: docker run -it --rm -v $(pwd)/drone-images:/code/images -v $(pwd)/odm_orthophoto:/code/odm_orthophoto -v $(pwd)/odm_texturing:/code/odm_texturing opendronemap/opendronemap

As you can see, the only part that I'm modifying is the path that I'm mapping for the images.

I'm currently building a docker container against the latest code to see if this is still an issue.

[pierotofy]

Would you be able to share your dataset so that we could try to run it also?

[bstempi]

Go nuts!

https://s3.amazonaws.com/public.brianstempin.com/odm_test_data/facilities.zip

[pierotofy]

So I was able to process the images using the latest node-opendronemap docker image (which is built upon opendronemap/opendronemap):

That's really odd. Could you share details about the machine you are running this on (is it on EC2? If so which instance type?)

[pierotofy]

Ah I noticed you already shared some machine details in the issue description. I wonder if it's some deeper issue (maybe a race condition) in opensfm related to multithreading (just a guess). The only significant difference between my machine and yours is that my CPU has only two physical cores. I also have 24GB of RAM.

What happens if you pass the --opensfm-processes 1 option?

[bstempi]

Nice! I'll give your approach a try.

I'm using a 24Gb Linode with 8 cores. When I try on the 2Gb Linode, my
process crashes with a similar message, except that it also says that it
can't allocate memory (despite having swap space available).

I'll retry the the opensfm-process option tomorrow.

On Fri, Mar 10, 2017 at 9:28 PM, Piero Toffanin [e-mail]
wrote:

Ah I noticed you already shared some machine details in the issue
description. I wonder if it's some deeper issue (maybe a race condition) in
opensfm related to multithreading (just a guess). The only significant
difference between my machine and yours is that my CPU has only two
physical cores, I have 24gb of RAM).

What happens if you pass the --opensfm-processes 1 option?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/OpenDroneMap/OpenDroneMap/issues/509#issuecomment-285834934,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABOXADUHoykdqXs4QQV0C6MhkVeDnsVvks5rkga9gaJpZM4MZbcb
.

[smathermather]

Similar to what @pierotofy suggests, I would try reducing your number of CPUs you are using to 6 to balance out the number of cores with the amount of memory: --opensfm-processes 6

[bstempi]

When trying the --opensfm-processes 1 option, I get the following output:

2017-03-12 09:06:11,478 Merging depthmaps
[DEBUG]   running PYTHONPATH=/code/SuperBuild/install/lib/python2.7/dist-packages /code/SuperBuild/src/opensfm/bin/export_bundler /code/opensfm
[INFO]    Running ODM OpenSfM Cell - Finished
[INFO]    Running ODM Meshing Cell
[DEBUG]   Writing ODM Mesh file in: /code/odm_meshing/odm_mesh.ply
[DEBUG]   running /code/build/bin/odm_meshing -inputFile /code/opensfm/depthmaps/merged.ply -outputFile /code/odm_meshing/odm_mesh.ply -logFile /code/odm_meshing/odm_meshing_log.txt -maxVertexCount 100000 -octreeDepth 9  -samplesPerNode 1.0 -solverDivide 9
[INFO]    Running ODM Meshing Cell - Finished
[INFO]    Running MVS Texturing Cell
[DEBUG]   Writing MVS Textured file in: /code/odm_texturing/odm_textured_model.obj
[DEBUG]   running /code/SuperBuild/install/bin/texrecon /code/opensfm/reconstruction.nvm /code/odm_meshing/odm_mesh.ply /code/odm_texturing/odm_textured_model -d gmi -o gauss_clamping -t none     
/code/SuperBuild/install/bin/texrecon (built on Mar 10 2017, 18:01:53)
Load and prepare mesh: 
PLY Loader: comment PCL generated
Reading PLY: 100048 verts... 199844 faces... done.
Generating texture views: 
NVM: Loading file...
NVM: Number of views: 241
NVM: Number of features: 0
        Loading 46%...libpng error: Write Error                                 
Aborted (core dumped)
Traceback (most recent call last):
  File "/code/run.py", line 55, in <module>
    plasm.execute(niter=1)
  File "/code/scripts/mvstex.py", line 109, in process
    '{keepUnseenFaces}'.format(**kwargs))
  File "/code/opendm/system.py", line 28, in run
    raise Exception("Child returned {}".format(retcode))
Exception: Child returned 134

Trying again with 6.

[smathermather]

New error, so that's good. That's an error I've only seen a couple times. How many images do you have in your dataset? My hunch is that the mesh has gotten so large that texturing is now the barrier, but it'd be good to know more about the dataset.

[bstempi]

Sure!

The dataset is posted above if you'd like to do some manual inspections.
Here are some statistics:

Data size: 1.3Gb
Number of images: 260
Area size photographed: ~4 acres
Drone: Phantom 3 basic

On Sun, Mar 12, 2017 at 11:15 AM, Stephen Mather [e-mail]
wrote:

New error, so that's good. That's an error I've only seen a couple times.
How many images do you have in your dataset? My hunch is that the mesh has
gotten so large that texturing is now the barrier, but it'd be good to know
more about the dataset.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/OpenDroneMap/OpenDroneMap/issues/509#issuecomment-285951178,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABOXAPrkayChO8tP1ql0X5csAmMWHOiKks5rlAwEgaJpZM4MZbcb
.

[pierotofy]

@smathermather The thing is that I've processed the dataset (260 images) just fine, and my PC specs are lower than his VPS.

[bstempi]

@pierotofy When was the last time you pulled the opendronemap/opendronemap? It looks like versions aren't labeled, so the only way to know what you were running is to look at pull date. I had pulled mine shortly before opening this ticket.

Also, would it be worth me trying to run the data through the node-opendronemap as you did?

[pierotofy]

5 days ago. The last image built was 6 days ago, so I'm using the same version as you. https://hub.docker.com/r/opendronemap/opendronemap/

It will not make a difference to run via node-opendronemap, that image is based on opendronemap.

But it would be interesting to see if you can process the images on another linode instance (maybe with fewer CPUs). These seg faults smell like race conditions.

[pierotofy]

Also, if we can retrieve the core dump, we might be able to dig further. Try to mount a docker volume at /code/SuperBuild/install/bin/, and when the process fails with Loading 46%...libpng error: Write Error (core dumped) copy/paste the resulting core file in that directory here.

[bstempi]

I have a run in progress. When it dies, I'll start a new one and try to capture the dump.

[bstempi]

Running with --opensfm-processes 6 resulted in the last error that I posted.

I'm going to resize my VPS so that it only has 4 cores and I'll attempt to catch the core dump.

[bstempi]

I ran the following Docker command to try to retrieve the coredump:

docker run -it --rm -v $(pwd)/bin:/code/SuperBuild/install/bin -v $(pwd)^Crone-images:/code/images -v $(pwd)/odm_orthophoto:/code/odm_orthophoto -v $(pwd)/odm_texturing:/code/odm_texturing my_odm_image

I didn't get a coredump in my local bin/ directory.

I'll rerun without the --rm option.

[bstempi]

I got a new error when trying on a machine with 4 cores and 8Gb of RAM:

2017-03-13 01:54:42,983 Merging depthmaps
Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/compute_depthmaps.py", line 25, in run
    dense.compute_depthmaps(data, graph, reconstructions[0])
  File "/code/SuperBuild/src/opensfm/opensfm/dense.py", line 40, in compute_depthmaps
    merge_depthmaps(data, graph, reconstruction, neighbors)
  File "/code/SuperBuild/src/opensfm/opensfm/dense.py", line 162, in merge_depthmaps
    points, normals, colors = dm.merge()
MemoryError
Traceback (most recent call last):
  File "/code/run.py", line 55, in <module>
    plasm.execute(niter=1)
  File "/code/scripts/opensfm.py", line 90, in process
    (context.pyopencv_path, context.opensfm_path, tree.opensfm))
  File "/code/opendm/system.py", line 28, in run
    raise Exception("Child returned {}".format(retcode))
Exception: Child returned 1
root@debian:/home/bstempi/current_stitch_proj

No mention of a core dump in this one.

[smathermather]

That's a very clear out of memory error on merging of depthmaps. With 8GB, I'd use max 2 cores only.

[pierotofy]

Yup, not enough memory there. Enable a swapfile. https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-ubuntu-14-04

[bstempi]

I had an instance with 8Gb of RAM and a 10Gb swap file. It's less than the 24Gb of RAM you were using so perhaps that still wasn't enough.

I'm retrying on an instance with 60Gb of RAM and 4 cores to see if that will do it.

[smathermather]

Cool. Let us know how it goes. The depthmap merging is killing us on memory usage -- that's our current memory bottleneck. It's something we'll need to address in the next round of core work (cc @paulinus).

[bstempi]

I got a segfault again (same as the error in my first post) while running on 4 cores and 60Gb of RAM, but the core dump is empty. I saved the container -- here's the listing:

root@6eee3757e109:/code# ls -lha       
total 100K
drwxr-xr-x 17 root root 4.0K Mar 13 16:15 .
drwxr-xr-x 22 root root 4.0K Mar 13 16:20 ..
drwxr-xr-x  8 root root 4.0K Mar 10 18:02 .git
-rw-r--r--  1 root root  198 Mar 10 17:33 .gitignore
-rw-r--r--  1 root root   97 Mar 10 17:33 .gitmodules
-rw-r--r--  1 root root  547 Mar 10 17:33 CMakeLists.txt
drwxr-xr-x  7 root root 4.0K Mar 10 17:58 SuperBuild
drwxr-xr-x  5 root root 4.0K Mar 10 18:24 build
-rw-r--r--  1 root root  654 Mar 10 17:33 ccd_defs_check.py
-rwxr-xr-x  1 root root 3.0K Mar 10 17:33 configure.sh
-rw-------  1 root root    0 Mar 13 16:15 core
drwxr-xr-x  2 root root 4.0K Mar 13 13:24 images
drwxr-xr-x  2 root root  12K Mar 13 13:26 images_resize
drwxr-xr-x  8 root root 4.0K Mar 10 17:57 modules
drwxr-xr-x  2 root root 4.0K Mar 13 16:21 odm_meshing
drwxr-xr-x  2 root root 4.0K Mar 13 13:24 odm_orthophoto
drwxr-xr-x  2 root root 4.0K Mar 13 13:24 odm_texturing
drwxr-xr-x  2 root root 4.0K Mar 13 13:24 opendm
drwxr-xr-x  7 root root 4.0K Mar 13 16:13 opensfm
drwxr-xr-x  3 root root 4.0K Mar 10 17:57 patched_files
drwxr-xr-x  2 root root 4.0K Mar 13 13:26 pmvs
-rw-r--r--  1 root root 1.7K Mar 10 17:33 run.py
drwxr-xr-x  2 root root 4.0K Mar 13 13:24 scripts
drwxr-xr-x  2 root root 4.0K Mar 10 17:58 tests
root@6eee3757e109:/code# 

I'm assuming that core is the core dump. Am I looking at the right thing?

[pierotofy]

Yes core is the file.

What's the output of (from within the docker container) of:

ulimit -c and find /code -name core?

Perhaps we need to set ulimit -c unlimited. I didn't mentioned that because usually having a ulimit -c of 0 should suppress the "core dumped" string, but that might not be the case with Ubuntu (see http://stackoverflow.com/a/18368068)

[bstempi]

root@debian:~# docker run -it --entrypoint=/bin/bash bstempi/odm_core_dump
root@05ff176fe8c9:/code# ulimit -c
unlimited
root@05ff176fe8c9:/code# find /code -name core
/code/core
/code/SuperBuild/install/include/opencv2/core
/code/SuperBuild/src/cmvs/program/thirdParty/miniBoost/boost/core
/code/SuperBuild/src/opencv/modules/gpu/src/nvidia/core
/code/SuperBuild/src/opencv/modules/core
/code/SuperBuild/src/opencv/modules/core/include/opencv2/core
/code/SuperBuild/src/opencv/modules/java/android_test/src/org/opencv/test/core
/code/SuperBuild/src/opencv/samples/cpp/tutorial_code/core
/code/SuperBuild/src/opencv/doc/tutorials/core
/code/SuperBuild/src/pdal/vendor/pdalboost/boost/core
/code/SuperBuild/src/pdal/java/core
/code/SuperBuild/build/opencv/modules/core

It looks like ulimit is already unlimited.

[pierotofy]

It looks like core dumps are handled by systemd. Is there a /var/lib/systemd/coredump directory in there? If so, the core dumps might be there.

[bstempi]

root@debian:~# docker run -it --entrypoint=/bin/bash bstempi/odm_core_dump
root@082a71c82ef1:/code# ls /var/lib/
alsa/            dpkg/            initscripts/     locales/         misc/            plymouth/        sgml-base/       ucf/             ureadahead/      
apt/             git/             insserv/         logrotate/       ntpdate/         python/          sudo/            update-rc.d/     vim/             
dhcp/            initramfs-tools/ libuuid/         man-db/          pam/             python-support/  systemd/         urandom/         xml-core/        
root@082a71c82ef1:/code# ls /var/lib/systemd/                           
deb-systemd-helper-enabled
root@082a71c82ef1:/code# 

It doesn't appear so.

[pierotofy]

Ahhh, my mistake, docker containers use the host's kernel, so systemv is the one I currently have on my machine.

What's the output of: cat /proc/sys/kernel/core_pattern?

You might find the core dump in a directory on your host. I found a core dump for a test program I built and run within the docker container on my host's /var/lib/systemd/coredump (that's because I use systemv), but your host's kernel configuration is different.

[bstempi]

root@debian:~# docker run -it --entrypoint=/bin/bash bstempi/odm_core_dump
root@3b81d5118476:/code# cat /proc/sys/kernel/core_pattern
core
root@3b81d5118476:/code# exit
exit
root@debian:~# ls /var/lib/systemd/
catalog/                    coredump/                   deb-systemd-helper-enabled/ deb-systemd-helper-masked/  random-seed                 
root@debian:~# ls /var/lib/systemd/coredump/
root@debian:~# 

No dice.

[pierotofy]

Well, I'm out of ideas.

I wrote this program (test.c):

#include <stdio.h>

int main(void){
    int *ptr = 0;
    printf("%d", *ptr);
}

Then from my host:

sudo bash -c 'echo core > /proc/sys/kernel/core_pattern'

Then from the container:

/code# g++ test.c
/code# ./a.out
/code# ls
-rw-rw-r-- 1 root root    547 Dec 24 00:00 CMakeLists.txt
drwxr-xr-x 1 root root     84 Mar  6 00:26 SuperBuild
-rwxr-xr-x 1 root root   8520 Mar 13 19:36 a.out
drwxr-xr-x 1 root root    122 Mar  6 02:09 build
-rw-rw-r-- 1 root root    654 Nov 26 15:49 ccd_defs_check.py
-rwxrwxr-x 1 root root   2975 Feb 28 00:00 configure.sh
-rw------- 1 root root 266240 Mar 13 19:38 core.27
drwxr-xr-x 1 root root     24 Mar  6 00:26 docker
drwxr-xr-x 1 root root    170 Mar  6 00:26 modules
drwxr-xr-x 1 root root    132 Mar  6 00:26 opendm
drwxr-xr-x 1 root root      6 Mar  6 00:26 patched_files
-rw-rw-r-- 1 root root   1693 Jan 18 00:00 run.py
drwxr-xr-x 1 root root    378 Mar  6 00:26 scripts
-rw-r--r-- 1 root root     74 Mar 13 19:36 test.c
drwxr-xr-x 1 root root     42 Mar  6 00:26 tests

core.27 is being generated. So I have no idea why the core file by opensfm is not being generated (perhaps it's too large?), but without being able to reproduce this error or get hands on a coredump, I wouldn't be able to troubleshoot much :)

Perhaps when I have some time I could try to start a similar VPS and try there.

[bstempi]

If I process my images with WebODM, it works.

Perhaps there's something happening with the way that ODM is invoked?

[pierotofy]

WebODM uses by default the image at opendronemap/node-opendronemap, which in turn is based on opendronemap/opendronemap. So it shouldn't make a difference. There are no special differences between using opendronemap and WebODM.

[bstempi]

What else could it be? I'm using the latest ODM image. Perhaps the
defaults are different when calling run.py?

On Mar 14, 2017 5:57 PM, "Piero Toffanin" [e-mail] wrote:

WebODM uses by default the image at opendronemap/node-opendronemap, which
in turn is based on opendronemap/opendronemap. So it shouldn't make a
difference. There are no special differences between using opendronemap and
WebODM.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/OpenDroneMap/OpenDroneMap/issues/509#issuecomment-286573838,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABOXAEzaElproRuaZAQm_l5VKUS7oh7oks5rlw05gaJpZM4MZbcb
.

[pierotofy]

Being that the segmentation fault looks to be caused by a race condition, you might have gotten lucky and the dataset processed without errors (maybe some CPUs were busy with running WebODM). It's not guaranteed that processing it another time (either via WebODM or directly from opendronemap) will work again. Race conditions can be tricky to reproduce consistently.

There's definitely a problem in opensfm's code, so I think we should leave this issue open until somebody finds time to attach a debugger / generate a core dump and find out exactly what's going on.

[bstempi]

Those are valid points.

I'm willing to start up a Linode and to let somebody have at it with my dataset since it consistently produced errors. I'm a developer, but it's been years since I've touched C/C++, so I don't think I'm in a good position to be the once attaching debuggers, etc.

[pierotofy]

I could try to take a stab at it. Here's my public SSH key:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC3hhFfUo5YUcHSR6CB+Ic2KNaTXcSVgbSVTWD6R3WUsxze2jIHjrrGICsG0VHjnIEZuJte5lpBNVqcHJPAeMyHF2OGQ4S7Y+4Okg1gRcBR7qkOjo3UNTA7+riUzA8j2FWloonsOMaqH3pwrRle+ewr3ZW2AOB55XUb4YGRx+oLE9v/zWEQQ+KtQZqAwxjIkAXTe6LKBTYf5l1v+ba57s+tT4+erEzP5bDH9Sk01du4dr4RShukSlk/g9eeCZhpjiYaLNiANrtn8QLGLtrmZ3idkfp7WUymbnw2Ry6bB4e+NNdszfA/vXGQkCNcLhQIFvvgMLdMw0QpeuuCCkfzYpP7 piero@piero-PC

[bstempi]

Added. You have root access to that machine. Everything on it is ephemeral, so there's no risk of accidentally destroying anything. Here's what I have there:

• Docker with a bunch of ODM related images
• A giant swap-file to prevent OOM errors

Let me know if you need anything else!

[pierotofy]

Host/ip address and port of the VPS? :)

[bstempi]

hahahaha, herp derp!

li586-234.members.linode.com

SSH is on the standard port

On Fri, Mar 17, 2017 at 8:59 AM, Piero Toffanin [e-mail]
wrote:

Host/ip address and port of the VPS? :)

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/OpenDroneMap/OpenDroneMap/issues/509#issuecomment-287347333,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABOXAA3Y-8W2WO6Y5CK9StwSsMnwoo8Vks5rmoO6gaJpZM4MZbcb
.

[pierotofy]

Mm, just processed the dataset without issues:

[INFO]    Creating GeoTIFF
[DEBUG]   running gdal_translate -a_ullr 497123.495338 4395274.15748 497364.496643 4395068.13737 -co TILED=yes -co COMPRESS=DEFLATE -co PREDICTOR=2 -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co NUM_THREADS=ALL_CPUS -a_srs "EPSG:32618" /data/odm_orthophoto/odm_orthophoto.png /data/odm_orthophoto/odm_orthophoto.tif > /data/odm_orthophoto/gdal_translate_log.txt
[INFO]    Running ODM OrthoPhoto Cell - Finished
[INFO]    OpenDroneMap app finished - Fri Mar 17 19:13:58  2017

Resulting assets are in /root/data.

Commands I ran:

docker run -ti -d -p 3000:3000 -v `pwd`/data:/data opendronemap/node-opendronemap
docker exec -ti <sha-1> /bin/bash
cd /code
python run.py --project-path /data

Took a few hours to process, but no errors.

[bstempi]

All of my errors were on the opendronemap/opendronemap image.

On Fri, Mar 17, 2017 at 3:37 PM, Piero Toffanin [e-mail]
wrote:

Mm, just processed the dataset without issues:

[INFO] Creating GeoTIFF
[DEBUG] running gdal_translate -a_ullr 497123.495338 4395274.15748 497364.496643 4395068.13737 -co TILED=yes -co COMPRESS=DEFLATE -co PREDICTOR=2 -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co NUM_THREADS=ALL_CPUS -a_srs "EPSG:32618" /data/odm_orthophoto/odm_orthophoto.png /data/odm_orthophoto/odm_orthophoto.tif > /data/odm_orthophoto/gdal_translate_log.txt
[INFO] Running ODM OrthoPhoto Cell - Finished
[INFO] OpenDroneMap app finished - Fri Mar 17 19:13:58 2017

Resulting assets are in /root/data.

Commands I ran:

run -ti -d -p 3000:3000 -v pwd/data:/data opendronemap/node-opendronemap
docker exec -ti /bin/bashcd /code
python run.py --project-path /data

Took a few hours to process, but no errors.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/OpenDroneMap/OpenDroneMap/issues/509#issuecomment-287451498,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABOXABdrVXW5xla5yZrqYuErDmxzlj-wks5rmuDcgaJpZM4MZbcb
.

[pierotofy]

Should not matter. https://github.com/OpenDroneMap/node-OpenDroneMap/blob/master/Dockerfile

This line:

FROM opendronemap/opendronemap:latest

Simply instructs docker to use opendronemap/opendronemap to build the opendronemap/node-opendronemap. They have the same base image.

[pierotofy]

I should add, perhaps this is just another "lucky" case where the race condition did not occur. It's possible that if I run it again, maybe the error will happen. I'll try that.

[pierotofy]

Nop, nothing. Repeated the process 4 times, can't get it to seg fault.

[bstempi]

Try checking out the ODM project and build the image.

On Sat, Mar 18, 2017 at 6:43 PM, Piero Toffanin [e-mail]
wrote:

Nop, nothing. Repeated the process 4 times, can't get it to seg fault.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/OpenDroneMap/OpenDroneMap/issues/509#issuecomment-287580325,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABOXAL6xfc_F-4FXWcorUMqQyj9NeZvlks5rnF4UgaJpZM4MZbcb
.

[pierotofy]

Built from sources (/root/OpenDroneMap), re-run dataset, processed without seg fault. @bstempi were you able to reproduce the issue on this particular linode instance? If so, what commands did you use?

[bstempi]

I performed all of my computation on the same instance you're using.

I'll retry and see if I can reproduce.

On Mon, Mar 20, 2017 at 9:29 AM, Piero Toffanin [e-mail]
wrote:

Built from sources (/root/OpenDroneMap), re-run dataset, processed without
seg fault. @bstempi https://github.com/bstempi were you able to
reproduce the issue on this particular linode instance? If so, what
commands did you use?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/OpenDroneMap/OpenDroneMap/issues/509#issuecomment-287758766,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABOXAKYrfUovAEJNOCD7wpcQB17YGxl3ks5rnn9AgaJpZM4MZbcb
.

[dakotabenjamin]

Is this a case where running with GDB would help to debug? I've run into segfaults before but never consistently enough to try. If you are able to reproduce the segfault, it wouldn't hurt to try.

[bstempi]

I'll retry tonight to see if I can reproduce.

[dakotabenjamin]

Any update?

[bstempi]

I've been lazy. I'll make time tonight.

On Mon, Apr 3, 2017 at 12:18 PM, Dakota Benjamin [e-mail]
wrote:

Any update?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/OpenDroneMap/OpenDroneMap/issues/509#issuecomment-291193607,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABOXAFgQeI1j65Ktetlp2tckhnkq8xNqks5rsRu-gaJpZM4MZbcb
.

[dakotabenjamin]

No worries, I'm just trying to keep up with the issue queue!

[bstempi]

I'm no longer able to replicate the issue. I tried firing up a brand new machine, getting a fresh install of everything, and used the same commands that I had used earlier.

The only thing that I did differently was use a very, very large swap file to try to prevent OOM errors. So, perhaps my segfault was an out of memory error or something in disguise.

Since I can no longer replicate this, I would say that this ticket is safe to close.

[dakotabenjamin]

Please open again if you run into it again.

[bstempi]

No problem; thanks for all of the help!

On Tue, Apr 11, 2017 at 8:13 AM, Dakota Benjamin [e-mail]
wrote:

Closed #509 https://github.com/OpenDroneMap/OpenDroneMap/issues/509.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/OpenDroneMap/OpenDroneMap/issues/509#event-1038166548,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABOXAApZibOd319CAevZrUE4b35MimpKks5ru25zgaJpZM4MZbcb
.

[furick1]

This was helpful for me to debug a similar issue. Thanks to all for the thorough troubleshooting and clear communication.

[mjevanmarle]

I have a similar problem running 69 photos in webodm. I already allocated 50 GB extra swap, but this doesn't help either. Maybe some of you can help me.

[INFO] Running georeferencing with generated coords file.
[DEBUG] running /opt/odm/OpenDroneMap_v0_3_1/build/bin/odm_georef -bundleFile /opt/odm/node-OpenDroneMap/data/5156a2cf-47df-463f-b990-c6e6bdb8f641/opensfm/bundle_r000.out -inputCoordFile /opt/odm/node-OpenDroneMap/data/5156a2cf-47df-463f-b990-c6e6bdb8f641/odm_georeferencing/coords.txt -inputFile /opt/odm/node-OpenDroneMap/data/5156a2cf-47df-463f-b990-c6e6bdb8f641/odm_texturing/odm_textured_model.obj -outputFile /opt/odm/node-OpenDroneMap/data/5156a2cf-47df-463f-b990-c6e6bdb8f641/odm_texturing/odm_textured_model_geo.obj -inputPointCloudFile /opt/odm/node-OpenDroneMap/data/5156a2cf-47df-463f-b990-c6e6bdb8f641/opensfm/depthmaps/merged.ply -outputPointCloudFile /opt/odm/node-OpenDroneMap/data/5156a2cf-47df-463f-b990-c6e6bdb8f641/odm_georeferencing/odm_georeferenced_model.ply -logFile /opt/odm/node-OpenDroneMap/data/5156a2cf-47df-463f-b990-c6e6bdb8f641/odm_georeferencing/odm_georeferencing_log.txt -georefFileOutputPath /opt/odm/node-OpenDroneMap/data/5156a2cf-47df-463f-b990-c6e6bdb8f641/odm_georeferencing/odm_georeferencing_model_geo.txt
Segmentation fault (core dumped)
Traceback (most recent call last):
File "/opt/odm/OpenDroneMap/run.py", line 46, in
plasm.execute(niter=1)
File "/opt/odm/OpenDroneMap_v0_3_1/scripts/odm_georeferencing.py", line 137, in process
'-logFile {log} -georefFileOutputPath {geo_sys}'.format(**kwargs))
File "/opt/odm/OpenDroneMap_v0_3_1/opendm/system.py", line 28, in run
raise Exception("Child returned {}".format(retcode))
Exception: Child returned 139

Best, Margreet

[pierotofy]

@mjevanmarle this look like a different error (not memory related). Could you open a new issue and perhaps share your dataset?

[mjevanmarle]

@pierotofy I created one: https://github.com/OpenDroneMap/OpenDroneMap/issues/730

[pierotofy]

@mjevanmarle Thank you!

[saira05]

hello, i am making my uni's lab pointcloud but i also get this error of segmentation fault(core dumped). My image size is very small in kbs(single image) and i am using only 30 images. can anybody help me about it why i got this kind of error? my system have 4gb ram and core i5.

[smathermather]

@Saira05 -- feel free to reach out at http://community.opendronemap.org