Odm: OSFMContext.name() breaks splitmerge

[acortelyou]

OSFMContext.name() returns parent folder name submodels intead of submodel_XXXX which breaks splitmerge.py submodel processing.

https://github.com/OpenDroneMap/ODM/blob/aa26e4aa643ab081d834e8f1d5709141d78ba8ac/opendm/osfm.py#L230-L231

https://github.com/OpenDroneMap/ODM/blob/aa26e4aa643ab081d834e8f1d5709141d78ba8ac/stages/splitmerge.py#L140-L150

Using latest docker image with

docker run -it --rm \
    -v "$(pwd)/images:/code/images" \
    -v "$(pwd)/odm_georeferencing:/code/odm_georeferencing" \
    -v "$(pwd)/odm_meshing:/code/odm_meshing" \
    -v "$(pwd)/odm_orthophoto:/code/odm_orthophoto" \
    -v "$(pwd)/odm_texturing:/code/odm_texturing" \
    -v "$(pwd)/opensfm:/code/opensfm" \
    -v "$(pwd)/mve:/code/mve" \
    opendronemap/odm \
    --verbose \
    --time \
    --split 400

....
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    Submodel submodels has already been aligned.
[INFO]    ========================
[INFO]    Processing submodels
[INFO]    ========================
[INFO]    running /code/run.py --verbose --time --fast-orthophoto --project-path /code/submodels --orthophoto-cutline --dem-euclidean-map --skip-3dmodel submodels
[INFO]    Fast orthophoto is turned on, automatically setting --skip-3dmodel
[INFO]    Initializing OpenDroneMap app - Wed Dec 04 22:20:05  2019
[INFO]    ==============
[INFO]    build_overviews: False
[INFO]    camera_lens: auto
[INFO]    cameras: {}
[INFO]    crop: 3
[INFO]    debug: False
[INFO]    dem_decimation: 1
[INFO]    dem_euclidean_map: True
[INFO]    dem_gapfill_steps: 3
[INFO]    dem_resolution: 5
[INFO]    depthmap_resolution: 640
[INFO]    dsm: False
[INFO]    dtm: False
[INFO]    end_with: odm_orthophoto
[INFO]    fast_orthophoto: True
[INFO]    force_gps: False
[INFO]    gcp: None
[INFO]    ignore_gsd: False
[INFO]    matcher_distance: 0
[INFO]    matcher_neighbors: 8
[INFO]    max_concurrency: 12
[INFO]    merge: all
[INFO]    mesh_octree_depth: 9
[INFO]    mesh_point_weight: 4
[INFO]    mesh_samples: 1.0
[INFO]    mesh_size: 100000
[INFO]    min_num_features: 8000
[INFO]    mve_confidence: 0.6
[INFO]    name: submodels
[INFO]    opensfm_depthmap_method: PATCH_MATCH
[INFO]    opensfm_depthmap_min_consistent_views: 3
[INFO]    opensfm_depthmap_min_patch_sd: 1
[INFO]    orthophoto_compression: DEFLATE
[INFO]    orthophoto_cutline: True
[INFO]    orthophoto_no_tiled: False
[INFO]    orthophoto_resolution: 5
[INFO]    pc_classify: False
[INFO]    pc_csv: False
[INFO]    pc_ept: False
[INFO]    pc_filter: 2.5
[INFO]    pc_las: False
[INFO]    pc_sample: 0
[INFO]    project_path: /code/submodels
[INFO]    rerun: None
[INFO]    rerun_all: False
[INFO]    rerun_from: None
[INFO]    resize_to: 2048
[INFO]    skip_3dmodel: True
[INFO]    sm_cluster: None
[INFO]    smrf_scalar: 1.25
[INFO]    smrf_slope: 0.15
[INFO]    smrf_threshold: 0.5
[INFO]    smrf_window: 18.0
[INFO]    split: 999999
[INFO]    split_overlap: 150
[INFO]    texturing_data_term: gmi
[INFO]    texturing_keep_unseen_faces: False
[INFO]    texturing_nadir_weight: 16
[INFO]    texturing_outlier_removal_type: gauss_clamping
[INFO]    texturing_skip_global_seam_leveling: False
[INFO]    texturing_skip_hole_filling: False
[INFO]    texturing_skip_local_seam_leveling: False
[INFO]    texturing_skip_visibility_test: False
[INFO]    texturing_tone_mapping: none
[INFO]    time: True
[INFO]    use_3dmesh: False
[INFO]    use_exif: False
[INFO]    use_fixed_camera_params: False
[INFO]    use_hybrid_bundle_adjustment: False
[INFO]    use_opensfm_dense: False
[INFO]    verbose: True
[INFO]    ==============
[WARNING] Directory submodels does not exist. Creating it now.
[INFO]    Running dataset stage
[INFO]    Project directory /code/submodels/submodels/images doesn't exist. Creating it now. 
[INFO]    Loading dataset from: /code/submodels/submodels/images
[ERROR]   Not enough supported images in /code/submodels/submodels/images
Traceback (most recent call last):
  File "/code/run.py", line 57, in <module>
    app.execute()
  File "/code/stages/odm_app.py", line 92, in execute
    self.first_stage.run()
  File "/code/opendm/types.py", line 370, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 351, in run
    self.process(self.args, outputs)
  File "/code/stages/splitmerge.py", line 150, in process
    system.run(" ".join(map(quote, argv)), env_vars=os.environ.copy())
  File "/code/opendm/system.py", line 76, in run
    raise Exception("Child returned {}".format(retcode))
Exception: Child returned 1

Additionally the tree.submodels_path appears to be incorrect: /code/submodels instead of /code/opensfm/submodels but I can work around that with volume mounts.

[acortelyou]

Changing the .. to a . produces the expected output and allows processing to continue. https://github.com/OpenDroneMap/ODM/blob/aa26e4aa643ab081d834e8f1d5709141d78ba8ac/opendm/osfm.py#L231

[pierotofy]

Hey @acortelyou :hand: thanks for the report; tree.submodels_path is actually correct (should be /code/submodels).

self.path("..") should return the parent directory (which is correct, if the OSFMContext is for /code/submodels/submodel_0000/opensfm the function should return submodel_0000).

Running on my dev environment I wasn't able to immediately reproduce this, so further investigation is needed.

[acortelyou]

Interesting.

My submodels ended up in /code/opensfm/submodels/submodel_XXXX not in /code/submodels/submodel_XXXX and the /code/submodels folder was empty.

The OSFMContext was being created for the submodel_XXXX folder not the opensfm subfolder which is apparently why my "fix" worked.

I am starting a fresh run completely from scratch having pruned all docker images just in case to sanity check. It's quite a large dataset so it may take some time (4000 images).

The only peculiarity about my setup is that I'm running Docker Desktop on Windows, but since it's running linux containers in HyperV I wouldn't expect that to cause this sort of issue.

Let me know what sort of digging/further info would be helpful.

[pierotofy]

Perhaps it's something different about the running environment, but I can't reproduce this.

Running:

docker run -it --rm \
    -v "$(pwd)/images:/code/images" \
    -v "$(pwd)/odm_georeferencing:/code/odm_georeferencing" \
    -v "$(pwd)/odm_meshing:/code/odm_meshing" \
    -v "$(pwd)/odm_orthophoto:/code/odm_orthophoto" \
    -v "$(pwd)/odm_texturing:/code/odm_texturing" \
    -v "$(pwd)/opensfm:/code/opensfm" \
    -v "$(pwd)/mve:/code/mve" \
    opendronemap/odm \
    --verbose \
    --time \
    --split 8

On a small dataset works as expected (it's the same command you posted, with just --split changed to a smaller value to fit the test dataset).

...
2019-12-05 04:24:09,958 INFO: Reconstruction 0: 18 images, 8837 points
2019-12-05 04:24:09,958 INFO: 1 partial reconstructions in total.
[INFO]    Aligning submodels...
[INFO]    /code/submodels/submodel_0000/opensfm/reconstruction.aligned.json is now /code/submodels/submodel_0000/opensfm/reconstruction.json
[INFO]    /code/submodels/submodel_0001/opensfm/reconstruction.aligned.json is now /code/submodels/submodel_0001/opensfm/reconstruction.json
[INFO]    /code/submodels/submodel_0002/opensfm/reconstruction.aligned.json is now /code/submodels/submodel_0002/opensfm/reconstruction.json
[INFO]    ========================
[INFO]    Processing submodel_0000
[INFO]    ========================
[INFO]    running /code/run.py --verbose --time --project-path /code/submodels --orthophoto-cutline --dem-euclidean-map --skip-3dmodel submodel_0000
[INFO]    Initializing OpenDroneMap app - Thu Dec 05 04:24:17  2019
[INFO]    ==============
...

Does your $(pwd) path contain spaces? Multiple volumes? Special file systems?

Perhaps some other users can try to reproduce this and post their experience as well.

[acortelyou]

$(pwd) does not contain spaces.

There is however a mount --bind /mnt/d /d so that the paths match between WSL and Docker Desktop. Perhaps that is an issue?

I am trying the suggestion from your initial comment before the edits of using a single docker volume binding -v /d/project:/datasets/code opendronemap/odm --project-path /datasets and after that will try the run on a linux host instead of WSL/Docker Desktop.

Hopefully that will clear up where the issue lies.

Thanks for looking into it!

[acortelyou]

When using the single docker volume binding for the dataset I see the submodels folder created where expected instead of inside the opensfm folder.

However now I'm experiencing an issue where instead of being split into 125 submodels as before it is only split into 20 submodels that each have 90% of the entire dataset, which crashes the "create_tracks" step on my machine (only 48GB of ram available to the container).

I'm not sure what to make of this result.

To recap: when I bound the folders individually the submodels folder was created in the wrong place but the submodels themselves correctly had roughly 400 images and now that I'm binding the root dataset folder the submodels folder is created in the right place but the submodels each have roughly 9000 images in them.

😮

I looked through the logs and the only odm option I had specified differently during the original pass was "fast-orthophoto" so I'll try that again from scratch just to see. Will update when it completes.

edit: On a future run this did not happen again: I suspect I was looking at it too soon and it spreads out as the solver progresses.

[pierotofy]

Mm, very odd. Note that split-merge uses symlinks for some of its logic, which may or may not trip up whatever compatibility layer WSL is using; I've had sporadic issues mounting volumes with ODM before on Windows (with and without WSL). Perhaps they are related. See https://github.com/OpenDroneMap/oats/blob/master/run#L87 and https://github.com/OpenDroneMap/oats#windows-10-wsl-quirks

[acortelyou]

This issue disappears when using a single mount point and after ditching WSL/Docker Desktop for a standard linux host my other issues were gone as well.

-v ~/project:/datasets/code opendronemap/odm --project-path /datasets

Thanks for looking into it and anyone running across this in the future: just use linux with a single bind and save yourself some trouble.