Tesseract: MSVC optimization /Ot (included in /O2) breaks tesseract release build

[MarkusGH]

Environment

• Tesseract Version: 4.00.00dev with current training data (tessdata_best).
• Commit Number: eba0ae3b88a46a93e981770caa0b148d65cc4468
• Platform: Windows 64-bit build, Visual Studio 2017 15.4.4

Current Behavior:

Test.zip

1) MinSizeRel build

Command:

tesseract.exe C:\Test.tif C:\Test -l deu+eng --psm 1 --oem 1 pdf

Error messages:

None

Result:

Correct PDF created, OCR results OK

2) Release build

Command:

tesseract.exe C:\Test.tif C:\Test -l deu+eng --psm 1 --oem 0 pdf

Error messages:

None

Result:

Correct PDF created, OCR results OK

3) Release build

Command:

tesseract.exe C:\Test.tif C:\Test -l deu+eng --psm 1 --oem 1 pdf

Error messages:

Multiple times:

Error in pixCreateHeader: height must be > 0
Error in pixCreateNoInit: pixd not made
Error in pixCreate: pixd not made
Error in pixClipRectangle: pixd not made

Result:

Broken PDF created.

Conclusion:

Release build shows erratic behavior when LSTM is used, errormessages and/or wrong output.
Debug And MinSizeRel builds work slowly but correctly.
As soon as the optimization /Ot is switched to /Os (or from /O2 to /O1) or LSTM is not used all problems go away.

Expected Behavior:

Release build behaves like MinSizeRel/Debug built.

Suggested Fix:

Problem is located in Tesseract::GetRectImage

As soon as speed optimization for this function is switched off with #pragma optimize( "t", off ) the problem goes away.

[zdenop]

How did you build tesseract?
Error messages you posted comes from leptonica, so...

[MarkusGH]

As I made some progress I updated the issue to better describe what is happening.
I built tesseract exactly like described here:
https://github.com/tesseract-ocr/tesseract/wiki/Compiling#windows
Develop Tesseract -> Building for x64 platform

[zdenop]

I would suggest you make sure that the all dependencies (at least leptonica) are build as 64bit and with the same version of compiler...

[MarkusGH]

It is a 64 bit version of leptonica - otherwise it would not work at all.
Why should the compiler used to build leptonica make any difference?
Due to my findings the problem is located in Tesseract::GetRectImage, and goes away as soon as that very function is not optimized for speed anymore.
I would not expect that the way tesseract interfaces with external libraries is changed by the optimization option.

[egorpugin]

Hi,

Not sure if this is your post https://groups.google.com/forum/#!topic/tesseract-ocr/lufafrAAEnk
It points to /Ob2 option while you state /Ot.

Did you try RelWithDebInfo configuration?

[MarkusGH]

Not my post and it does not look like the problem has anything to do with /Ob2.
RelWithDebugInfo shows the same behaviour as Release as both use /O2.

[MarkusGH]

Found similar problem in ColPartitionGrid::ComputeTotalOverlap with some input files.
A Release build hits the following assertion:
contains_unichar_id(unichar_id):Error:Assert failed:in file C:\Users\mw\Desktop\OCR\tesseract-master\ccutil\unicharset.h, line 513
As soon as I add #pragma optimize( "t", off ) directly in front of the function ColPartitionGrid::ComputeTotalOverlap the problem goes away.

[stweil]

@MarkusGH, did you also restore optimization after ColPartitionGrid::ComputeTotalOverlap and Tesseract::GetRectImage, so we can be sure that only those functions used code without optimization in your test?

[MarkusGH]

Yes - I put #pragma optimize( "", on ) directly after the functions.
It looks like the following have a strong effect on the problem:
Training files used, size of input file (pixel count), engine.
Strange enough I found at least one file which works with --oem 0 and --oem 1 but not with --oem 3.

[egorpugin]

To check if this is VS2017 issue, it's necessary to build on VS2015.
As I understand if we haven't any troubles on gcc/clang, it's problem of MSVC, not tesseract.

[MarkusGH]

I think it is too early to blame MSVC for this issue.
Could be a compiler error or could be sloppy programming which leads to unexpected behaviour when that kind of optimization is active. Again - to me it feels a lot like it has to do with signed integer overflow handling.
Signed integer overflow behaviour is undefined per C++11 standard.
Without optimizations signed integer overflow will simply wrap around (this is simply what the processor does). With optimizations anything can happen...
You actually do not really know, if the issue is not also present in gcc builds, because the effects are not necessarily obvious - could be there are only slight variations in OCR results.

As you use gcc I would kindly suggest to create a build with the gcc option -ftrapv (and maybe other options necessary to bring the debugger to break on integer overflows) and try the test case I provided to see if it actuall produces integer overflows. If so this makes a case to investigate further in that direction.

[stweil]

A Tesseract executable which was compiled with -ftrapv does not raise an overflow exception. What about debugging the buggy Windows executable and setting a conditional breakpoint on pixCreateHeader (waiting for height <= 0)?

[MarkusGH]

Oh - so it looks like the integer overflow hypothesis is dead.
About pixCreateHeader: I searched for that already. Unfortunately I did not build leptonica from source and it does not look like this function is called from anywhere outside leptonica.

[stweil]

I tested the latest binary from AppVeyor. It shows the same problem when running tesseract --oem 1 Test.tif stdout with LSTM.

It looks like pixCreateHeader is called with height == 0.

[stweil]

@egorpugin, I tried a local build on Windows 7 to debug the problem, but failed: cppan.exe crashed very often with SIGSEGV, and when it finally passed, I got a cmake failure similar to https://github.com/cppan/cppan/issues/36. Is there an alternative way to build Tesseract with VS 2017 community edition?

[egorpugin]

Could you try the latest build?

cppan.zip

[FaisalHussain95]

@stweil I have the same issue here with Visual 15 2017 Win64 Enterprise & Community editions

Thanks @egorpugin i will try that

Edit: not working... i will try overiding .cppan/helpers.cmake
Related post: cppan/cppan#36

[egorpugin]

@Salfai77210, @stweil
Please, try another one client.

cppan.zip

[FaisalHussain95]

@egorpugin okay i will try this tkx. Have you read my edited comment on cppan/cppan#36 ?

I have added my fix for this issue as well FIX

[egorpugin]

New client is a possible fix for your issue too. Try it.

[stweil]

Even with the latest cppan it takes about 20 iterations until cppan has fetched all required packages and passes without failure. It looks like these iterations don't get everything correctly set up, because the following cmake still complains about missing files and fails.

[egorpugin]

Are you trying this on windows?

[stweil]

Sure. I tried it on Windows 7 / 64 bit from the Visual Studio command line. PATH was fixed to include the CMake binary and cppan.

[egorpugin]

Maybe this one?
Sorry, I can't repro the issue locally, so we have this kinda "game".
Did you see any messages about crash dumps? Check c:\users\u\appdata\local\temp\cppan\dump folder.

cppan.zip

[stweil]

I got messages about crash dumps (minidumps), but did not find the dump files. I'll have a look at the indicated location later.

[stweil]

The cppan crash seems to occur while trying to free memory, maybe related to setlocale. Could you please create an executable with debug information (even better: a debug build)? Then I'll get more information about the exact code location.

[egorpugin]

Debug version:
https://www.dropbox.com/s/lja1dmzn6iwzxnx/cppand.zip?dl=1

[zdenop]

@MarkusGH @stweil @FaisalHussain95 Is this issue solved?

[stweil]

I just repeated the previous test with the latest binary, and it looks like the issue is fixed.

[FaisalHussain95]

@zdenop Yes it works !