Meshroom: Windows7: Meshroom got stuck in FeatureExtraction

[LukeV1]

Meshroom always gets stuck in the FeatureExtraction node while the log shows that AliceVision actually _succesfully_ finished.

Doing the workaround described in #183 "solves" the problem and Meshroom continues then.

Im running Windows 7 64bit with CUDA 9.2

Log file (presumably not important):

Program called with the following parameters:
 * describerPreset = "high"
 * describerTypes = "sift"
 * forceCpuExtraction = 1
 * input = "P:/Photoscan/monument/MeshroomCache/CameraInit/8d797ccec8360a3492d5cb9e1587ac45a70b7c41/cameraInit.sfm"
 * maxThreads = 0 (default)
 * output = "P:/Photoscan/monument/MeshroomCache/FeatureExtraction/1420a3202080d02b4958234883833dfd1439f489"
 * rangeSize = 33
 * rangeStart = 0
 * verboseLevel = "trace"

[13:33:29.470469][debug] Job max memory consumption: 992092048 B
[13:33:29.470469][debug] Memory information: 
    - Total RAM:  7.9992 GB
    - Free RAM:   5.5797 GB
    - Total swap: 8192 GB
    - Free swap:  8191.89 GB

[13:33:29.470469][debug] # threads for extraction: 4
[13:33:31.186472][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3053.JPG' [cpu]
[13:33:31.233272][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3035.JPG' [cpu]
[13:33:31.264472][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3054.JPG' [cpu]
[13:33:31.358072][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3050.JPG' [cpu]
[13:33:51.546143][info]       34083 sift features extracted from view 'P:/Photoscan/monument/IMG_3054.JPG'
[13:33:52.482145][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3038.JPG' [cpu]
[13:33:54.826800][info]       33906 sift features extracted from view 'P:/Photoscan/monument/IMG_3053.JPG'
[13:33:55.624221][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3045.JPG' [cpu]
[13:33:56.622623][info]       42372 sift features extracted from view 'P:/Photoscan/monument/IMG_3035.JPG'
[13:33:57.943056][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3027.JPG' [cpu]
[13:34:01.343250][trace] Grid filtering -- Copy remaining points: 3478
[13:34:03.012854][info]       50000 sift features extracted from view 'P:/Photoscan/monument/IMG_3050.JPG'
[13:34:04.363910][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3047.JPG' [cpu]
[13:34:11.111183][info]       44360 sift features extracted from view 'P:/Photoscan/monument/IMG_3038.JPG'
[13:34:12.139199][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3026.JPG' [cpu]
[13:34:16.606915][info]       41774 sift features extracted from view 'P:/Photoscan/monument/IMG_3045.JPG'
[13:34:17.737936][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3046.JPG' [cpu]
[13:34:20.456092][info]       39359 sift features extracted from view 'P:/Photoscan/monument/IMG_3027.JPG'
[13:34:22.466745][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3023.JPG' [cpu]
[13:34:29.358574][info]       44898 sift features extracted from view 'P:/Photoscan/monument/IMG_3026.JPG'
[13:34:30.891662][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3025.JPG' [cpu]
[13:34:33.911321][info]       43705 sift features extracted from view 'P:/Photoscan/monument/IMG_3047.JPG'
[13:34:34.928380][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3042.JPG' [cpu]
[13:34:42.346777][info]       41025 sift features extracted from view 'P:/Photoscan/monument/IMG_3046.JPG'
[13:34:43.656236][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3033.JPG' [cpu]
[13:34:46.292640][info]       34674 sift features extracted from view 'P:/Photoscan/monument/IMG_3023.JPG'
[13:34:48.243290][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3041.JPG' [cpu]
[13:34:52.050995][info]       44191 sift features extracted from view 'P:/Photoscan/monument/IMG_3025.JPG'
[13:34:53.530670][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3037.JPG' [cpu]
[13:35:02.120945][info]       48636 sift features extracted from view 'P:/Photoscan/monument/IMG_3042.JPG'
[13:35:04.454079][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3039.JPG' [cpu]
[13:35:06.290184][info]       38763 sift features extracted from view 'P:/Photoscan/monument/IMG_3033.JPG'
[13:35:07.709816][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3049.JPG' [cpu]
[13:35:15.623941][info]       42825 sift features extracted from view 'P:/Photoscan/monument/IMG_3037.JPG'
[13:35:17.059143][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3036.JPG' [cpu]
[13:35:19.263450][trace] Grid filtering -- Copy remaining points: 9238
[13:35:21.717371][info]       50000 sift features extracted from view 'P:/Photoscan/monument/IMG_3041.JPG'
[13:35:23.904203][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3040.JPG' [cpu]
[13:35:32.647676][info]       44389 sift features extracted from view 'P:/Photoscan/monument/IMG_3039.JPG'
[13:35:34.559785][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3029.JPG' [cpu]
[13:35:35.269826][info]       43326 sift features extracted from view 'P:/Photoscan/monument/IMG_3036.JPG'
[13:35:35.982832][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3052.JPG' [cpu]
[13:35:38.696509][trace] Grid filtering -- Copy remaining points: 4261
[13:35:39.871576][info]       50000 sift features extracted from view 'P:/Photoscan/monument/IMG_3049.JPG'
[13:35:40.795629][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3030.JPG' [cpu]
[13:35:51.430697][trace] Grid filtering -- Copy remaining points: 7266
[13:35:53.858008][info]       50000 sift features extracted from view 'P:/Photoscan/monument/IMG_3040.JPG'
[13:35:55.308811][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3051.JPG' [cpu]
[13:35:55.565821][info]       37184 sift features extracted from view 'P:/Photoscan/monument/IMG_3052.JPG'
[13:35:56.352866][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3032.JPG' [cpu]
[13:35:59.210030][info]       39478 sift features extracted from view 'P:/Photoscan/monument/IMG_3029.JPG'
[13:35:59.905039][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3048.JPG' [cpu]
[13:36:03.748754][info]       38452 sift features extracted from view 'P:/Photoscan/monument/IMG_3030.JPG'
[13:36:05.183957][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3043.JPG' [cpu]
[13:36:11.760207][info]       38434 sift features extracted from view 'P:/Photoscan/monument/IMG_3032.JPG'
[13:36:20.092657][trace] Grid filtering -- Copy remaining points: 9386
[13:36:20.745695][info]       50000 sift features extracted from view 'P:/Photoscan/monument/IMG_3051.JPG'
[13:36:21.635746][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3034.JPG' [cpu]
[13:36:25.420449][trace] Grid filtering -- Copy remaining points: 8908
[13:36:26.035484][info]       50000 sift features extracted from view 'P:/Photoscan/monument/IMG_3048.JPG'
[13:36:26.637518][info]       41805 sift features extracted from view 'P:/Photoscan/monument/IMG_3043.JPG'
[13:36:27.302556][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3024.JPG' [cpu]
[13:36:27.400562][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3028.JPG' [cpu]
[13:36:39.510715][info]       39826 sift features extracted from view 'P:/Photoscan/monument/IMG_3034.JPG'
[13:36:40.441768][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3044.JPG' [cpu]
[13:36:44.558891][info]       38870 sift features extracted from view 'P:/Photoscan/monument/IMG_3024.JPG'
[13:36:45.713957][info] Extracting sift features from view 'P:/Photoscan/monument/IMG_3031.JPG' [cpu]
[13:36:45.902968][info]       38261 sift features extracted from view 'P:/Photoscan/monument/IMG_3028.JPG'
[13:36:55.638899][info]       40775 sift features extracted from view 'P:/Photoscan/monument/IMG_3044.JPG'
[13:37:00.087640][info]       41135 sift features extracted from view 'P:/Photoscan/monument/IMG_3031.JPG'
[13:37:00.098641][info] Task done in (s): 210.626000

---

To me it seems as if many people got the same issue also at other nodes..

[fabiencastan]

Yes, this problem seems to be present only on Windows 7.
The binaries have been created on Windows 10 and have not been tested on Windows 7.
I don't know why he doesn't work the same way.

[LukeV1]

Maybe this has something to do with the Windows permission management?

At least to my understanding there _are_ stange things happening when I change the execute permissions:

• If I start Meshroom explicitely as administrator, I can't add any images per drag and drop anmore (when dropping it just shows the crossed sign)
• If I enable "execute as administator" on aliceVision exe files (what means enabling the code of single nodes to be executed as admin), Windows opens a command promt window showing the progress (displaying the log). Some of this windows are automatically closing (i. e. aliceVision_cameraInit.exe) and some not (i. e. aliceVision_featureExtraction.exe). The windows which are not automatically closing are the ones which were leaving Meshroom in an idle/stuck state after finishing.

Hope you understand my point or at least what I'm trying to describe :grin:

I think the 2. point is showing the problem why it isn't working on Win 7 - some of the AliceVision executables didn't close properly.

[LukeV1]

Just found out that somebody else discovered this already in #145. To be more specific: here.

[sirwillard]

I limped it through on Win 7 by changing the status files up to the depthMap node, but that is just too much work for the 33 status files. It appears as you said.. AliceVision is not exiting properly. The GPU/CPU go idle, but the executable remains an active but idle process. The logs say it completed. If you kill the process, you can go back and edit the status file from error to success, and continue, but that's A LOT of babysitting.

[LukeV1]

but that's A LOT of babysitting.

I know. I did this once just to prove that it is working then..

[fake-name]

I'm hitting this too, both in feature extraction and now in structure from motion.

Interestingly, stopping and restarting the run via the UI seems to successfully resume at the point where it got wedged for the feature extraction, but it causes the StructureFromMotion step to crash when you resume.

I have a VS2015 install, so I'd be happy to help debug if someone can provide the appropriate PDB files. Attaching to the stuck process and pausing leaves me somewhere in OpenImageIO, so it needs the OpenImageIO.PDB debug database.

[Valouleloup]

Same problem for me.
OS : Win7
Meshroom : v2018.1.0
GPU : GTX 850M with last Nvidia drivers (08-2018)

The workaround I've found is (same workaround as @sirwillard actually link#1 and link#2 on an other thread) :

• Stop the process when FeatureExtraction is done but stuck (Task done in (s): xxx.xxx)
• Go to MeshroomCache/FeatureExtraction//status.log ( is the token you can find in the UI output something)
• Edit the status from ERROR to SUCCESS
• Go back to the UI and resume the computation

I have to repeat this process like 7-8 times, for StructureFromMotion, DepthMap, PrepareDenseScene, Meshing and MeshinFiltering. At the end, the results are great for the examples I've tested.

An other workaround is to do it from console, I will test it later.

[fabiencastan]

@fake-name When you say, it is stuck somewhere in OpenImageIO: Is it on the FeatureExtraction or on the StructureFromMotion step? Or both?

[yann-lty]

@fake-name Thanks for debugging and finding this.
It seems to be related to this issue: https://github.com/OpenImageIO/oiio/issues/1795

[fake-name]

@fabiencastan - both.

@yann-lty - Interesting.

[fabiencastan]

Can someone try to set OPENIMAGEIO_THREADS=1 as an environment variable to see if it fixes the problem?

[Valouleloup]

No change for me :/

[fabiencastan]

Thanks for testing it so quickly! ;)

Could you try to launch the aliceVision command line directly?
If you look into the status section of the Meshroom node, you can see the command line launched.
I'm not sure that if the env var is propagated to the final command line.

[Valouleloup]

If I launch the command line in a terminal, the process never ends (it shows the same Task done in (s) : xxx.xxx just before)

[sirwillard]

The only work around I can find was in the comments for the OIIO bug. They built with OIIO 1.7.17 for win7, and the recent build for win10.

[DerFetzer]

I have found another workaround by accident. I tried to find information about the processes that are not ending with Sysinternals Process Explorer. So when I open the properties of the hanging process (Right click --> Properties) it immediately exits successfully! This is still not an acceptable workaround but it is way more comfortable than the other workaround.

[LukeV1]

This may also be a sign that the issue is not even originated in the OIIO code but in some windows lib or even a compiler bug. Because viewing properties of a exe shouldn't affect any code in the running application :thinking:

[todd-koeck]

Here's a script I wrote for sort of working around this issue. It requires cygwin64 and I ran it under windoze 7 64. File uploading to attach didn't work, so I made a gist.

fix-meshroom-status

It's a complete hack, but it eliminates much of the manual work needed to use meshroom if you are seeing these errors. The script runs an infinite loop, looking for status files that have ended with a result of ERROR and a resultCode of 15. It then fixes those files and sleeps a few seconds before it looks again.

Run the script in a cygwin64 terminal in your MeshroomCache directory. Run meshroom and click the stop and start buttons. Just wait for the current "hung" step to turn green between clicks.

Type CTRL-C in the cygwin64 window to terminate the script when you're done with meshroom.

[hyperfx]

The workaround more or less worked for me, but my resultCode was never 15. Usually it's 255. Does that affect anything or make a difference in what the bug could be?

[fake-name]

Any chance someone could bake a release with https://github.com/OpenImageIO/oiio/pull/2013 merged into the OIIO dependency?

[fabiencastan]

We will try it in 2 or 3 weeks and upload new binaries if if it fixes the problem.

[hyperfx]

I also could never get it to create the mesh. Using the workaround, I'd get to the final node, and it would change to green, but the .obj never shows up in the correct folder, and I can't ever load a model in the UI, although I can see the point cloud.

[fake-name]

I have a project where I'd really like to see if I can use photogrammetry to avoid a bunch of messy measuring, but I have no windows 10 machines whatsoever (bc W10 is a privacy shitshow).

I looked at building it myself, but the dependencies are so complex it'd probably take a full day or so to get everything together.

[PersonWithIssue]

It does, in fact, get stuck at FeatureExtraction on Windows 7. Fix it, please.

[fake-name]

https://github.com/OpenImageIO/oiio/issues/1795 claims it has resolved this issue. Any chance of someone with the whole build toolchain doing a release?

[fabiencastan]

Yes, we managed to fix it and it is now included the latest oiio release.
It will be included in the next release of Meshroom which should be there in a few weeks.

[yann-lty]

A new download option for Windows 7 is now available on the release page, please see https://github.com/alicevision/meshroom/issues/183#issuecomment-442396682 for additional details.
Would be great if several people could confirm the problem is solved before closing all those related issues! @fake-name @PersonWithIssue @LukeV1 @sirwillard @Valouleloup

[sirwillard]

I have just confirmed that this is working! Threads now close properly across all nodes in win7.
Thank you!

[yann-lty]

Thanks for the feedback!

[fake-name]

Can report that it works for me, too.

[hyperfx]

Still getting stuck for me on the DepthMap node. Any ideas, or how can I figure out why it's getting stuck there?

It's not the CUDA problem, I don't think, I have a GeForce GTX Titan X

[fake-name]

@hyperfx - Are you using the windows-7 specific build? What's the log output?

[hyperfx]

Where do I find the log output?

[hyperfx]

Oh I see, in the panel at bottom left. It says I need a CUDA enabled GPUwith at least compute capability 2.0

[hyperfx]

I believe my card has compute capability 6.1

[LukeV1]

Would be great if several people could confirm the problem is solved before closing all those related issues!

The new build is working for me too!
Thanks for investigating and fixing this :+1: