Tesseract: CMake hangs at "Performing 71 checks using 8 threads"

[datalurkur]

Cmake hangs at this stage and does not advance, even if left overnight:

The C compiler identification is MSVC 19.0.24215.1
The CXX compiler identification is MSVC 19.0.24215.1
Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/amd64_x86/cl.exe
Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/amd64_x86/cl.exe -- works
Detecting C compiler ABI info
Detecting C compiler ABI info - done
Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/amd64_x86/cl.exe
Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/amd64_x86/cl.exe -- works
Detecting CXX compiler ABI info
Detecting CXX compiler ABI info - done
Detecting CXX compile features
Detecting CXX compile features - done
Check if the system is big endian
Searching 16 bit integer
Looking for sys/types.h
Looking for sys/types.h - found
Looking for stdint.h
Looking for stdint.h - found
Looking for stddef.h
Looking for stddef.h - found
Check size of unsigned short
Check size of unsigned short - done
Using unsigned short
Check if the system is big endian - little endian
-- Performing 71 checks using 8 threads
-- This process may take up to 5 minutes depending on your hardware

[zdenop]

IMO this check has nothing to do with tesseract.
@egorpugin can you have a look on this?

[egorpugin]

Tess uses lept and the process does appropriate checks for each dependent project.
@datalurkur Do you use the latest CPPAN client? If not try to update it. If yes I need to investigate this more closely, maybe with more help from you.

[datalurkur]

Just updated to latest cppan and tried again, same results:

D:\personal\tesseract>cppan --self-upgrade
Downloading checksum file
Downloading the latest client
Unpacking
Replacing client

D:\personal\tesseract\build>git status
On branch master
Your branch is up-to-date with 'origin/master'.

D:\personal\tesseract>cppan
Reading package specs... Ok
Generating build configs... Ok

D:\personal\tesseract>cd build

D:\personal\tesseract\build>cmake .. -DSTATIC=1
-- Building for: Visual Studio 14 2015
-- Check if the system is big endian
-- Searching 16 bit integer
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of unsigned short
-- Check size of unsigned short - done
-- Using unsigned short
-- Check if the system is big endian - little endian
-- Performing 71 checks using 8 threads
-- This process may take up to 5 minutes depending on your hardware

[datalurkur]

I should mention that I'm seeing this issue across multiple machines, and this is the only codebase I'm seeing this issue for (cmake works fine on other projects, example: opencv).

[egorpugin]

It's cppan issue. I'm investigating.

[egorpugin]

@datalurkur Is it possible to chat with you somewhere (maybe skype etc.) and also could you share your desktop with me (in skype again or teamviewer)?
Because it works for me on my machine.
And I need some infos from you:

1. cmake version
2. list of processes (maybe cppan or cmake hangs)
3. files on disk in c:\Users\u\AppData\Local\Temp\cppan\vars\

[datalurkur]

I doubt I'll have time to chat synchronously or share my screen today. At a glance, it looks like cppan.exe is running during the hang, so I suspect it's the culprit. Also see 2 cmake-gui processes and about 7 cmake processes. Other info:

D:\personal\tesseract>cmake --version
cmake version 3.6.2

Regarding files in that directory, are you looking for a recursive list? It looks like it just contains random folder names that contain session data. There are a lot of files in each directory, though - looking for something specific?

[egorpugin]

Regarding files in that directory, are you looking for a recursive list?

That would be nice. But before, please remove all stuff in c:\Users\u\AppData\Local\Temp\cppan\vars\, re-run cmake as you did earlier. When cmake hangs, check task manager (or process explorer) and make sure that cmake processes performs nothing (cpu usage will be 0 instead of 13% in case of 8 cores). So, when cmake children perform nothing do a recursive list of vars dir (maybe with tree command).

If cmake processes are running probably the issue somewhere in its scripts (they're generated by cppan). cppan process just waits for them.

[datalurkur]

Yep, the CPU is doing barely anything.

snip

[egorpugin]

From this text it's not clear where 3.6.2 folders reside. Are they under CMakeFiles dirs?
Please do tree /F. It will show files too.
The output should be something like that http://imgur.com/4bPuWJB

[egorpugin]

Looks terrible. :) And not useful. Could you please post a screenshot of tree output and remove that text from your comment?

[datalurkur]

This better?
tree /F
Folder PATH listing for volume OSDisk
Volume serial number is 00000054 5A53:90A7

snip

[egorpugin]

Not good. Could you post a screenshot like this? http://imgur.com/4bPuWJB

[datalurkur]

http://imgur.com/a/Y3qXr

[egorpugin]

It's ok now, thanks! Everything seems to be fine there (atleast in dir 0).

[egorpugin]

Could you also please send me a screenshot of task manager with list of processes (details tab on win10) when hang occurs? I'm interested in cmake/cppan processes with command lines (right click on columns at the top of the table - select columns - command line or command args).

[datalurkur]

http://imgur.com/a/RZLsF

[egorpugin]

Image is empty for me :(

[datalurkur]

http://imgur.com/VjOG70H

[egorpugin]

Is that all? Only 3 cmake processes?

Could you try to clear c:\Users\u\AppData\Local\Temp\cppan\ dir, re-run cmake .. -DSTATIC=1 and make same screenshot again? I'm asking this because I see the same dir name 0XektimK in your messages. Maybe fresh dir could say me something.

Please do two screens: one right after parallel checks start and after 3-5 mins (when there's 0% cpu load by cmake processes).

And more here - could you please then (after hang occurs) archive the whole c:\Users\u\AppData\Local\Temp\cppan\vars\SomeRandomLetters dir and send it to me (attach here).

[datalurkur]

http://imgur.com/a/LAfPN
qDUdTLdY.zip

[egorpugin]

Are there any 'cl.exe' processes hanging around? Maybe MSBuild.exe?
It's better to check this when no other Visual Studio instances exist.
I suspect that:

1. initial cmake call is waiting for cppan
2. cppan is waiting for cmake children
3. these hanged cmake processes are waiting for visual studio cl.exe or MSBuild.exe or even cmd.exe or conhost.exe.

[datalurkur]

No cl.exe, link.exe, MSBuild.exe, but there are a 6 conhost.exe's. The two cmd.exe's open are my command windows. Conhosts all look identical with the following command line:
??\C:\WINDOWS\system32conhost.exe 0x4

[egorpugin]

Try to kill all conhosts one by one. On some step you should see some progress with cmake processes (increased cpu usage or death of that cmake process or smth else).

[datalurkur]

Killed all the conhosts. Seems like one of them was driving the command window I executed cmake from, so that shut down. Still see 4 cmake processes and a cppan process running.

[egorpugin]

That's true - cmakes are waiting for someone or deadlocked inside.

[egorpugin]

Could you try Process Explorer to check out children of those cmake processes? https://technet.microsoft.com/en-us/sysinternals/processexplorer.aspx

You should see process tree like this http://imgur.com/FDhmOHQ
Please, attach a screenshot of cmake call tree (maybe re-run it to keep your main cmd.exe window and full process tree online).

[datalurkur]

[egorpugin]

No children? :) Looks like cmake issue.

[egorpugin]

Ok, next step:

1. Keep in mind the number of hanged cmake process. It's a number of a worker (0-7) - last digit or dir name in cmake command line (e.g. c:\Users\u\AppData\Local\Temp\cppan\vars\SomeRandomLetters\7 - 7 is the needed number).
2. Kill all hanged cmake processes.
3. Go to c:\Users\u\AppData\Local\Temp\cppan\vars\SomeRandomLetters\YouCmakeHangProcessLastDirNumber)
4. Run in that dir cmake .

[datalurkur]

That seemed to unstick it. So cmake is hanging on something.

[egorpugin]

I've already seen bugs in cmake while developing cppan. Fixes will appear in cmake-3.7. But those are not issues you hit.
The best way in your case is to build debug version of cmake, put in into c:\program files (or x86)\cmake\bin, run cmake on tess library as you're doing and attach to hanged cmake processes.

[datalurkur]

Seems fixed in 3.7. I don't see the "running tests" step at all anymore.

[egorpugin]

Try to go to c:\users\u\.cppan\storage\cfg and remove it completely (cfg dir). Then re-run cmake as you did.

[datalurkur]

Yeah, this brings the problem back. If the next step is to build a debug version of cmake and debug their code, it might be a while before I have the time to address it.

[rold2007]

I got exactly the same issue. After many trial & errors I was about to compile CMake in debug when I found the solution in this thread. Simply calling cppan --self-upgrade solved my issue. It seems like this client is buggy and not up-to-date.

[egorpugin]

Yes, you can try to remove the whole storage dir at c:\users\u\.cppan\storage\.

[egorpugin]

To keep you up to date.

I was hit twice by this deadlock or whatever it is.
On my system it's very rare and not repeatable.
Second time it was "unlocked" accidentally in 30-45 seconds without my actions but with error process cannot be started.
It seems the issue in cmake's startup process. In other words it hangs when it invokes msvc compiler.
And it also possible that this is not cmake fault but cl.exe or Windows issue itself.

[egorpugin]

Ok, I've hit by deadlock again and debugged cmake process.
I filed an issue in cmake tracker, but don't know if they'll be able to fix it in the near future.
https://gitlab.kitware.com/cmake/cmake/issues/16461

As a temporal solution here I will add an option for performing checks in single thread. This will prevent such errors (I hope). It will be available with the next cppan client release. I'll ping about it here.

[hyogase]

it seems that the work can go on if you kill the two or three cmake tasks with least memories.

[egorpugin]

Yes. And more to say if you kill cppan process (responsible for parallel var checks) the overall process won't fail, it will switch automatically to single thread mode.

I've added njobs flag already, but the client version is not uploaded yet. I'm fixing the rest of the bugs.

[unholyHub]

If you have problems @hyogase solution works perfectly.

[egorpugin]

Hi there!
0.2.0 is out!
To lower number of parallel jobs, put to the config in c:\users\u\.cppan\cppan.yml line var_check_jobs: N, N is your number.
But...
CMake devs told me that they also have such hangs, so even 1 thread may hang.

[nionis]

removing all cmake tasks except one (with the highest memory) worked for me, thanks @hyogase

[Shreeshrii]

@egorpugin If this issue is fixed, please close.