Odm: ClusterODM “[WARNING] LRE: submodel_0001 failed with”

[rumenmitrev]

Its not working somehow.
Sub models are only distributed to local node even that ClusterODM has 2 online nodes(1 local /1 remote).
And it fails with :slight_smile: [WARNING] LRE: submodel_0001 failed with: [{“name”:“verbose”,“error”:“Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean”}]

console output:

Click to expand

[code]
[INFO] Initializing OpenDroneMap app - Mon Oct 28 08:24:10 2019
[INFO] ==============
[INFO] build_overviews: False
[INFO] camera_lens: auto
[INFO] cameras: {}
[INFO] crop: 3
[INFO] debug: True
[INFO] dem_decimation: 1
[INFO] dem_euclidean_map: False
[INFO] dem_gapfill_steps: 3
[INFO] dem_resolution: 2.0
[INFO] depthmap_resolution: 1000.0
[INFO] dsm: True
[INFO] dtm: True
[INFO] end_with: odm_orthophoto
[INFO] fast_orthophoto: False
[INFO] force_gps: False
[INFO] gcp: None
[INFO] ignore_gsd: True
[INFO] matcher_distance: 0
[INFO] matcher_neighbors: 8
[INFO] max_concurrency: 4
[INFO] merge: all
[INFO] mesh_octree_depth: 12
[INFO] mesh_point_weight: 4
[INFO] mesh_samples: 1.0
[INFO] mesh_size: 300000
[INFO] min_num_features: 14000
[INFO] mve_confidence: 0.6
[INFO] name: 9a3aa963-a5f9-4742-a589-907945610be5
[INFO] opensfm_depthmap_method: PATCH_MATCH
[INFO] opensfm_depthmap_min_consistent_views: 3
[INFO] opensfm_depthmap_min_patch_sd: 1
[INFO] orthophoto_compression: DEFLATE
[INFO] orthophoto_cutline: False
[INFO] orthophoto_no_tiled: False
[INFO] orthophoto_resolution: 5
[INFO] pc_classify: True
[INFO] pc_csv: False
[INFO] pc_ept: False
[INFO] pc_filter: 2.5
[INFO] pc_las: False
[INFO] project_path: /www/data
[INFO] rerun: None
[INFO] rerun_all: False
[INFO] rerun_from: None
[INFO] resize_to: -1
[INFO] skip_3dmodel: False
[INFO] sm_cluster: True
[INFO] smrf_scalar: 1.25
[INFO] smrf_slope: 0.15
[INFO] smrf_threshold: 0.5
[INFO] smrf_window: 18.0
[INFO] split: 100
[INFO] split_overlap: 80.0
[INFO] texturing_data_term: gmi
[INFO] texturing_keep_unseen_faces: False
[INFO] texturing_nadir_weight: 16
[INFO] texturing_outlier_removal_type: gauss_clamping
[INFO] texturing_skip_global_seam_leveling: False
[INFO] texturing_skip_hole_filling: False
[INFO] texturing_skip_local_seam_leveling: False
[INFO] texturing_skip_visibility_test: False
[INFO] texturing_tone_mapping: none
[INFO] time: False
[INFO] use_3dmesh: True
[INFO] use_exif: False
[INFO] use_fixed_camera_params: False
[INFO] use_hybrid_bundle_adjustment: False
[INFO] use_opensfm_dense: False
[INFO] verbose: True
[INFO] ==============
[INFO] Running dataset stage
[INFO] Loading dataset from: /www/data/9a3aa963-a5f9-4742-a589-907945610be5/images
[DEBUG] /www/data/9a3aa963-a5f9-4742-a589-907945610be5/images
[INFO] Loading 493 images

[INFO] Wrote images database: /www/data/9a3aa963-a5f9-4742-a589-907945610be5/images.json
[INFO] Found 493 usable images
[INFO] Parsing SRS header: WGS84 UTM 51N
[INFO] Finished dataset stage
[INFO] Running split stage
[INFO] Setting max-concurrency to 3 to better handle remote splits
[INFO] Large dataset detected (493 photos) and split set at 100. Preparing split merge.
[INFO] Altitude data detected, enabling it for GPS alignment
[INFO] ['use_exif_size: no', 'feature_process_size: -1', 'feature_min_frames: 14000', 'processes: 3', 'matching_gps_neighbors: 8', 'matching_gps_distance: 0', 'depthmap_method: PATCH_MATCH', 'depthmap_resolution: 1000.0', 'depthmap_min_patch_sd: 1', 'depthmap_min_consistent_views: 3', 'optimize_camera_parameters: yes', 'undistorted_image_format: png', 'bundle_outlier_filtering_type: AUTO', 'align_orientation_prior: vertical', 'use_altitude_tag: yes', 'align_method: auto', 'local_bundle_radius: 0', 'submodels_relpath: ../submodels/opensfm', 'submodel_relpath_template: ../submodels/submodel_%04d/opensfm', 'submodel_images_relpath_template: ../submodels/submodel_%04d/images', 'submodel_size: 100', 'submodel_overlap: 80.0']
[INFO] running /code/SuperBuild/src/opensfm/bin/opensfm extract_metadata "/www/data/9a3aa963-a5f9-4742-a589-907945610be5/opensfm"

[INFO] running /code/SuperBuild/src/opensfm/bin/opensfm create_submodels "/www/data/9a3aa963-a5f9-4742-a589-907945610be5/opensfm"
[INFO] Dataset has been split into 5 submodels. Reconstructing each submodel...
[INFO] LRE: Initializing using cluster node 127.0.0.1:3001
[INFO] LRE: Node is online and running odm version 0.9.1
[INFO] LRE: Adding to queue /www/data/9a3aa963-a5f9-4742-a589-907945610be5/submodels/submodel_0000
[INFO] LRE: Adding to queue /www/data/9a3aa963-a5f9-4742-a589-907945610be5/submodels/submodel_0001
[INFO] LRE: Adding to queue /www/data/9a3aa963-a5f9-4742-a589-907945610be5/submodels/submodel_0002
[INFO] LRE: Adding to queue /www/data/9a3aa963-a5f9-4742-a589-907945610be5/submodels/submodel_0003
[INFO] LRE: Adding to queue /www/data/9a3aa963-a5f9-4742-a589-907945610be5/submodels/submodel_0004
[INFO] LRE: About to process submodel_0000 locally
[INFO] ==================================
[INFO] LRE: About to process submodel_0001 remotely
[INFO] Local Reconstruction submodel_0000
[INFO] ==================================
[INFO] running /code/SuperBuild/src/opensfm/bin/opensfm detect_features "/www/data/9a3aa963-a5f9-4742-a589-907945610be5/submodels/submodel_0000/opensfm"
[WARNING] LRE: submodel_0001 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO] LRE: Re-queueing submodel_0001 (retries: 1)
[INFO] LRE: About to process submodel_0002 remotely
[WARNING] LRE: submodel_0002 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO] LRE: Re-queueing submodel_0002 (retries: 1)
[INFO] LRE: About to process submodel_0003 remotely
[WARNING] LRE: submodel_0003 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO] LRE: Re-queueing submodel_0003 (retries: 1)
[INFO] LRE: About to process submodel_0004 remotely
[WARNING] LRE: submodel_0004 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO] LRE: Re-queueing submodel_0004 (retries: 1)
[INFO] LRE: About to process submodel_0001 remotely
[INFO] LRE: Waiting 10 seconds before processing submodel_0001
2019-10-28 08:24:22,718 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms2_117.JPG
2019-10-28 08:24:23,054 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms1_117.JPG
2019-10-28 08:24:23,414 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms2_055.JPG
2019-10-28 08:24:31,347 DEBUG: Found 18493 points in 8.16880202293s
2019-10-28 08:24:31,349 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms1_117.JPG, no features masked.
2019-10-28 08:24:31,611 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms3_69.JPG
2019-10-28 08:24:31,876 DEBUG: Found 18735 points in 8.70158004761s
2019-10-28 08:24:31,880 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms2_117.JPG, no features masked.
[WARNING] LRE: submodel_0001 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO] LRE: Re-queueing submodel_0001 (retries: 2)
[INFO] LRE: About to process submodel_0002 remotely
[WARNING] LRE: submodel_0002 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO] LRE: Re-queueing submodel_0002 (retries: 2)
[INFO] LRE: About to process submodel_0003 remotely
2019-10-28 08:24:32,044 DEBUG: Found 18237 points in 8.51156592369s
2019-10-28 08:24:32,048 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms2_055.JPG, no features masked.
[WARNING] LRE: submodel_0003 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO] LRE: Re-queueing submodel_0003 (retries: 2)
[INFO] LRE: About to process submodel_0004 remotely
2019-10-28 08:24:32,166 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms1_110.JPG
[WARNING] LRE: submodel_0004 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO] LRE: Re-queueing submodel_0004 (retries: 2)
[INFO] LRE: About to process submodel_0001 remotely
[INFO] LRE: Waiting 20 seconds before processing submodel_0001
2019-10-28 08:24:32,344 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms1_123.JPG
2019-10-28 08:24:40,034 DEBUG: Found 18739 points in 8.29558491707s
2019-10-28 08:24:40,037 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms3_69.JPG, no features masked.
2019-10-28 08:24:40,280 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms3_31.JPG
2019-10-28 08:24:40,478 DEBUG: Found 18499 points in 8.00401902199s
2019-10-28 08:24:40,482 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms1_123.JPG, no features masked.
2019-10-28 08:24:40,619 DEBUG: Found 17747 points in 8.33730101585s
2019-10-28 08:24:40,621 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms1_110.JPG, no features masked.
2019-10-28 08:24:40,744 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms2_010.JPG
2019-10-28 08:24:40,836 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms3_05.JPG
2019-10-28 08:24:48,584 DEBUG: Found 18830 points in 8.18423914909s
2019-10-28 08:24:48,587 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms3_31.JPG, no features masked.
2019-10-28 08:24:48,880 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms1_104.JPG
2019-10-28 08:24:48,982 DEBUG: Found 18861 points in 8.12652301788s
2019-10-28 08:24:48,985 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms2_010.JPG, no features masked.
2019-10-28 08:24:49,230 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms1_145.JPG
2019-10-28 08:24:49,709 DEBUG: Found 18264 points in 8.7456908226s
2019-10-28 08:24:49,712 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms3_05.JPG, no features masked.
2019-10-28 08:24:49,956 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms3_80.JPG
[WARNING] LRE: submodel_0001 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO] LRE: Re-queueing submodel_0001 (retries: 3)
[INFO] LRE: About to process submodel_0002 remotely
[WARNING] LRE: submodel_0002 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO] LRE: Re-queueing submodel_0002 (retries: 3)
[INFO] LRE: About to process submodel_0003 remotely
[WARNING] LRE: submodel_0003 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO] LRE: Re-queueing submodel_0003 (retries: 3)
[INFO] LRE: About to process submodel_0004 remotely
[WARNING] LRE: submodel_0004 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO] LRE: Re-queueing submodel_0004 (retries: 3)
[INFO] LRE: About to process submodel_0001 remotely
[INFO] LRE: Waiting 30 seconds before processing submodel_0001
2019-10-28 08:24:57,154 DEBUG: Found 18115 points in 8.14896893501s
2019-10-28 08:24:57,156 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms1_104.JPG, no features masked.
2019-10-28 08:24:57,396 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms1_141.JPG
2019-10-28 08:24:57,453 DEBUG: Found 18285 points in 8.08539485931s
2019-10-28 08:24:57,455 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms1_145.JPG, no features masked.
2019-10-28 08:24:57,602 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms2_041.JPG
2019-10-28 08:24:59,278 DEBUG: Found 18541 points in 9.20740008354s
2019-10-28 08:24:59,281 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms3_80.JPG, no features masked.
2019-10-28 08:24:59,523 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms4_002.JPG
2019-10-28 08:25:04,705 DEBUG: Found 18472 points in 7.18407392502s
2019-10-28 08:25:04,706 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms1_141.JPG, no features masked.
2019-10-28 08:25:04,936 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms2_018.JPG
2019-10-28 08:25:06,485 DEBUG: Found 17667 points in 8.79993796349s
2019-10-28 08:25:06,488 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms2_041.JPG, no features masked.
2019-10-28 08:25:06,732 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms2_054.JPG
2019-10-28 08:25:08,081 DEBUG: Found 18643 points in 8.44536280632s
2019-10-28 08:25:08,084 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms4_002.JPG, no features masked.
2019-10-28 08:25:08,311 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms2_129.JPG
2019-10-28 08:25:12,981 DEBUG: Found 18594 points in 7.92231798172s
2019-10-28 08:25:12,984 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms2_018.JPG, no features masked.
2019-10-28 08:25:13,125 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms2_108.JPG
2019-10-28 08:25:15,435 DEBUG: Found 18044 points in 8.58410215378s
2019-10-28 08:25:15,438 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms2_054.JPG, no features masked.
2019-10-28 08:25:15,576 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms2_074.JPG
2019-10-28 08:25:17,093 DEBUG: Found 18503 points in 8.69408607483s
2019-10-28 08:25:17,096 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms2_129.JPG, no features masked.
2019-10-28 08:25:17,341 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms3_71.JPG
2019-10-28 08:25:21,691 DEBUG: Found 17845 points in 8.48174595833s
2019-10-28 08:25:21,693 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms2_108.JPG, no features masked.
2019-10-28 08:25:21,826 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms2_052.JPG
[WARNING] LRE: submodel_0001 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO] LRE: Re-queueing submodel_0001 (retries: 4)
[INFO] LRE: About to process submodel_0002 remotely
[WARNING] LRE: submodel_0002 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO] LRE: Re-queueing submodel_0002 (retries: 4)
[INFO] LRE: About to process submodel_0003 remotely
[WARNING] LRE: submodel_0003 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO] LRE: Re-queueing submodel_0003 (retries: 4)
[INFO] LRE: About to process submodel_0004 remotely
[WARNING] LRE: submodel_0004 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO] LRE: Re-queueing submodel_0004 (retries: 4)
[INFO] LRE: About to process submodel_0001 remotely
[INFO] LRE: Waiting 40 seconds before processing submodel_0001
2019-10-28 08:25:23,617 DEBUG: Found 18620 points in 7.92439603806s
2019-10-28 08:25:23,620 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms2_074.JPG, no features masked.
2019-10-28 08:25:23,848 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms1_111.JPG
2019-10-28 08:25:25,596 DEBUG: Found 18747 points in 8.14708900452s
2019-10-28 08:25:25,598 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms3_71.JPG, no features masked.
2019-10-28 08:25:25,740 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms3_84.JPG
2019-10-28 08:25:30,005 DEBUG: Found 18434 points in 8.06343984604s
2019-10-28 08:25:30,006 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms2_052.JPG, no features masked.
2019-10-28 08:25:30,268 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms2_113.JPG
2019-10-28 08:25:31,894 DEBUG: Found 17620 points in 7.92464804649s
2019-10-28 08:25:31,896 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms1_111.JPG, no features masked.
2019-10-28 08:25:32,140 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms3_65.JPG
[/code]

engine version 0.9.1
Api version 1.5.3

[pierotofy]

Thanks for the report @rumenmitrev :+1:

If you remove the --verbose flag, does it process correctly?

[rumenmitrev]

493 photos
dataset:
https://community.opendronemap.org/t/2018-12-21-deforestation-in-shitan-miaoli-county-taiwan/2063

Whit this parameters :
sm-cluster: http://127.0.0.1:3001, split: 100, split-overlap: 50, dsm: true

error is slightly differnet

[WARNING] LRE: submodel_0001 failed with: [{"name":"dsm","error":"Cannot convert 38f1220d-7785-4716-82b8-f2cfa0462ea1 to boolean"}]

[bertramt]

I'm running into that issue too

: [{"name":"dsm","error":"Cannot convert fd124845-8ec3-43b4-93d0-168a043fd990 to boolean"}]

with the options

sm-cluster: http://192.168.0.165:3001, rerun-from: dataset, split: 500, dsm: true

[pierotofy]

Could you detail your setup?

• Is the task being sent from WebODM? (I think yes, but just want to make sure)
• Are you sending the task to a ClusterODM or NodeODM node?

[rumenmitrev]

Yes. The task is initiated through WebODM.
At this machine reside also ClusterODM and NodeODM

On the remote machine is only NodeODM and WebODM

All is native install 16.04.

[pierotofy]

When you create a task for processing, do you send it to ClusterODM or NodeODM?

[rumenmitrev]

Processing node is Auto.
The other option is local NodeODM

I didn’t add ClusterODM as processing node!

Apparently the task is send to local NodeODM.

[bertramt]

I initially tried to add the cluster as a processing node and for some reason it didn't work come online so I assumed that setting the sm-cluster option overrides the node. I back and restarted everything and now was able to add my cluster as a node. Trying to reprocess now.

[rumenmitrev]

if I initiate a task via ./odm command (CloudODM) with explicitly --merge all , no errors occur.
Models are distributed correctly to remote host and task is running fine.
--sm-cluster "http://127.0.0.1:3001" --split "200" --split-overlap "80" --merge all --dtm
(without /--merge all/ fails!!)

But if i put --dsm , task again returns error
WARNING] LRE: submodel_0001 failed with: [{"name":"dsm","error":"Cannot convert 38f1220d-7785-4716-82b8-f2cfa0462ea1 to boolean"}]

[wang1212]

I ran into the same problem, I tried it on ubuntu 18.04 server. And I create a task through WebODM and add ClusterODM to Process Nodes.

I processed 136 images and the options for the error were:

sm-cluster: http://192.168.1.120:3001, rerun-from: dataset, debug: true, split-overlap: 5, split: 70, fast-orthophoto: true, max-concurrency: 1

I tried setting verbose to true, but it caused the error to occur, then I set it to false and tried to set split-overlap to 3, 10, 15, still not solved.

In the end, I succeeded, but I don't know the reason at the moment. The options are as follows:

sm-cluster: http://192.168.1.120:3001, rerun-from: dataset, debug: true, split-overlap: 50, split: 70, fast-orthophoto: true, max-concurrency: 1

I just set split-overlap to 50 and the error disappears.

At the same time, I removed the dsm option and the resulting data does not contain dsm.

If anyone else meets the same problem, can you give a solution and a specific reason? Thank you!

[pierotofy]

These issues should have been solved with https://github.com/OpenDroneMap/ODM/commit/59edf1fd0f63551470e118c87817f0d768b2dbc0 a few months ago. Please re-open if the issue persists? :pray: