Cryptopp: Random crashes on different computers because option -march=native is active 'by default'

[Marc--Olivier]

After compiling CRYPTOPP_5_6_5 and linking my program with the static library libcryptopp.a using mingw, this program was surprisingly crashing on some computers and not on others.

After many hours of investigation, I discovered that the flag -march=native was used and that was the reason why the program may not work when copied and pasted to other computers.
Note: it was extremely difficult to understand the reason of the crash, the stack being unhelpful (see the end of the email) and gdb was stopping immediately.

Having a dedicated option to activate or not this flag would be welcome - currently, the flag is set if ((NOT CRYPTOPP_CROSS_COMPILE) AND (CMAKE_CXX_COMPILER_ID MATCHES "Clang|GNU|Intel")).

Documenting the default use of the flag in README.txt could probably save a lot of time when investigating the problem.

I hope it helps!

```

Stack using catchsegv provided by drmingw

C:\Program Files (x86)\MyProgram>catchsegv MyProgram.exe
MyProgram.exe caused an Illegal Instruction at location 000000000089D6EC in module MyProgram.exe.

Registers:
eax=0000001e ebx=00d0fe60 ecx=4dfaeb02 edx=00000001 esi=00180000 edi=00000010
eip=0089d6ec esp=00d0fe50 ebp=00d0ff80 iopl=0 nv up ei pl nz na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202

AddrPC Params
0089D6EC 0024A000 747662A0 BF165E62 MyProgram.exe!CryptoPP::DetectX86Features
747662C4 0024A000 18364849 00000000 KERNEL32.DLL!@BaseThreadInitThunk@12
778D0FD9 FFFFFFFF 778F2EE9 00000000 ntdll.dll!__RtlUserThreadStart
778D0FA4 004014C0 0024A000 00000000 ntdll.dll!__RtlUserThreadStart@8
```

[noloader]

After compiling CRYPTOPP_5_6_5 and linking my program with the static library libcryptopp.a using mingw, this program was surprisingly crashing on some computers and not on others.

MinGW is no longer maintained by its authors, so we don't spend a lot of time with the platform nowadays. Its not regularly tested, and its hit or miss what will happen when using it.

For completeness, we call for testers for the platform before a release. In the case of MinGW, we usually don't get the testers.

---

Documenting the default use of the flag in README.txt could probably save a lot of time when investigating the problem.

Cmake is not documented in the README because its not the recommended way to build the library. The recommended way to build the library on Unix-like platforms is the GNUmakefile. On Windows, its the Visual Studio project files or MSBuild, which uses the VS project files.

There's some documentation for Cmake on the Crypto++ wiki.

For completeness, the documentation for the makefile can be found at GNUmakefile on the Crypto++ wiki.

---

After many hours of investigation, I discovered that the flag -march=native was used and that was the reason why the program may not work when copied and pasted to other computers.

In the regular makefile, you can do something like the following. Adding -march=x86-64 will signal you don't want the native capabilities of the host machine. It will build for a "least capable" machine, and its similar to the way distros build the library.

$ CXXFLAG="-DNDEBUG -g2 -O3 -march=x86-64" make -j 4
...

We regularly test both -march=i586 and -march=x86-64 because its the distro use case on IBM PCs.

I'm not sure what Cmake is doing. We added the Cmake files with the condition the community maintain it. We carry the files for convenience, but we don't do much more.

---

MyProgram.exe caused an Illegal Instruction at location 000000000089D6EC in module MyProgram.exe

This is expected as CPU features are probed. Its part of DetectX86Features, which can be found in cpu.cpp. The result of the probing is the preamble of cryptest.exe v:

Using seed: 1487764916

Testing Settings...

passed:  Your machine is little endian.
passed:  CRYPTOPP_ALLOW_UNALIGNED_DATA_ACCESS is not defined. Will restrict to aligned data access.
passed:  sizeof(byte) == 1
passed:  sizeof(word16) == 2
passed:  sizeof(word32) == 4
passed:  sizeof(word64) == 8
passed:  sizeof(hword) == 2, sizeof(word) == 4, sizeof(dword) == 8
passed:  hasMMX == 1, hasISSE == 1, hasSSE2 == 1, hasSSSE3 == 0, hasSSE4 == 0, hasAESNI == 0,
hasCLMUL == 0, hasRDRAND == 0, hasRDSEED == 0, hasSHA == 0, isP4 == 0, cacheLineSize == 32

---

Note: it was extremely difficult to understand the reason of the crash, the stack being unhelpful (see the end of the email) and gdb was stopping immediately.

Yeah, I think you went down a rabbit hole by suspecting the CPU probes. Your problem likely comes later.

[anonimal]

@Marc--Olivier @noloader

Introduced in db50b937660193bf19f737b17abfbbd5bbbed8dc (after 5.6.5), as implemented here: -D DISABLE_CXXFLAGS_OPTIMIZATIONS=ON.

[Marc--Olivier]

@noloader @anonimal
Many thanks for your detailed answers!

Cmake is not documented in the README because its not the recommended way to build the library.

I did not guess it. Maybe you should write it in the README.

[noloader]

@anonimal, @egorpugin, @nexussafe, @HeinrichJanzing, @alonbl, @steady286, @mouse07410, @morozovcookie, @ZahlGraf, @GamePad64, @Marc--Olivier

-D DISABLE_CXXFLAGS_OPTIMIZATIONS=ON

We added/updated the name to DISABLE_NATIVE_ARCH. DISABLE_NATIVE_ARCH is used in both the GNUmakefile and CmakefileList.txt. CmakefileList.txt responds to both DISABLE_CXXFLAGS_OPTIMIZATIONS and DISABLE_NATIVE_ARCH at the moment.

DISABLE_CXXFLAGS_OPTIMIZATIONS is marked as deprecated because it does not relate to optimizations (like -O0 or -O3). We will probably remove it at Crypto++ 6.1, after the upcoming Crypto++6.0 is released. We also documented the values at Cmake on the Crypto++ wiki.

Also see Commit 4bcaabbe2686f8b1. The 4bcaabbe commit is concerned with RDRAND and RDSEED, but the GNUmakefile overlap made it a convenient time to check it in. 4bcaabbe also bumped the Crypto++ version to 6.0 in CmakefileList.txt (it was already changed elsewhere).

[alonbl]

Hi,
Please make sure that any optimization flags are not added or downstream
can disable them if required.
No problem to use many of them as you outlined above.
Thanks!

[Romain-Geissler-1A]

Hi,

I came across the same issue on a set of Linux platforms. Our company uses many different open-source C/C++ libraries that we build ourselves, and AFAIK CryptoPP is the only library that builds with -march=native by default (at least, no other libraries as ever triggered a SIGILL instruction because the build machine and the run machine used different CPUs microarchitecture). The usual snippet "./configure; make; make install" (or just "make; make install" in your case) generates binaries without any -march flag on the vaste majority of projects, so I would consider this has become a de-facto standard. Seeing this issue we are not the only ones who spent time investigating this issue, and we will certainly not be the last ones. Would it be possible to switch the default to not using -march=native like the other open source projects ?

Cheers,
Romain

[noloader]

@Romain-Geissler-1A,

My gut tells me the common case should be increased performance. Those who want less performance should do something special.

With that said, here are some counter-arguments:

• Visual Studio builds with /machine:sse2, and not /machine:avx or /machine:avx2
• Folks like Uri (@mouse07410) have to do something special to get performance on OS X even with -march=native
• Its a pain in the ass to maintain -march=native because its missing on some platforms (ARM, Aarch32, Aarch64), and it causes segfaults on others (some GCC 43, some GCC4.6, SunCC on x86_64)
• Now that split sources are available from PR 461, it is easier to meet in the middle and keep everyone happy (or only mildly offend everyone)

The split sources mean we have a base implementation in say, sha.cpp and aes.cpp. The base can be compiled with -march=x64-64, or effectively no features. Then, we have accelerated implementation in files like sha-simd.cpp and aes-simd.cpp. The accelerated files get additional flags, like -msse4.2, -msha and -maes. The options are added by the makefile. Distros will have to rework their configure script (makefile.am?) and add the additional flags (re: Debain, Fedora, @gcsideal). At runtime, calls from base into accelerated are guarded as you would expect to avoid the SIGILLs.

If -march=native is removed by default, then we can readd it based on an envar, like CRYPTOPP_NATIVE_ARCH. If present, the makefile can revert back to CXXFLAGS += -march=native.

@mouse07410, @MarcelRaad, @denisbider - what do you guys think?

[alonbl]

Hi,
The decision what optimize flag to use is up to downstream maintainer.
Upstream can have sane default if it likes as long as downstream maintainer
can disable this easily and replace with downstream specific flags.
The native optimization is the most dangerous one as it makes it impossible
to use build machines.
Thanks,
Alon

[Romain-Geissler-1A]

I wanted to suggest you the "target" gcc attribute which would bring that sane middle between a single computer user seeking performances and a distro maintainer willing to ship the same binary to many different boxes. But apparently you had considered that already in PR 461, rejecting it for now since it requires too recent gcc. For me there are only two good options:

• either generate generic x64/arm binaries by default
• either use the gcc target attribute (which would make it work everywhere, and as fast as possible everywhere, without the user/maintainer even knows about it)

Requiring distro maintainers to explicitly disable your predefined optimization flags seems like a mistake for me (even more in that case since it is required in each and every case of distro maintainers). That would be fine if most other projects did the same, but that's definitely not the case, so by default people won't read your wiki to know about such optimization problems. Note that beyond the well known Fedoras and Debians distro, there are many other ones maintained by small groups of people who don't have time to read all documentations about all libraires they maintain, so keeping up with what the others are doing seems like the sanest default for me. (I personally work in a company hiring thousands of C++ developers, and only 2 or 3 people are maintaining the open source components).

Anyway, in the end the decision is yours. We have lost some time on this problem, now it's fixed on our side, so I would say we are fine with any decision you would take right now, this will not impact us anymore.

[denisbider]

@noloader - I think consistency across different products has an important economic value, which manifests by avoiding special cases that users need to learn about, and which end up consuming their time. If the expectation (based on use of other libraries) is that there is no -march flag, this should be the default.

Split-source compilation seems like a great way to enable optimizations at run-time that may not be present on all platforms where the library might be used.

[noloader]

@mouse07410, @MarcelRaad, @denisbider,

Ack, thanks.

I'll need a few days to yank -march=native. Intel ADX and OCB mode are the active tasks at the moment. Once they are cleared I can get into -march=native.

[noloader]

We started testing the removal at Commit 7b0b48af6665, Test remove -march=native.

ADX turned out to be an evolutionary dead-end, so I don't really need to investigate it further at the moment. If ADX ever takes a memory operand as the destination, then we can revisit it. Writing directly to memory will save two register writes, and it should overcome the performance loss.

[noloader]

@mouse07410, @MarcelRaad, @denisbider, @Romain-Geissler-1A, @alonbl, @anonimal, @Marc--Olivier,

Cleared at PR 465, Remove -march=native as default in Makefile and CMake.

We also cut-in the pattern the makefile uses for CMake. CMake will add flags like -maes and -march=armv8-a+crypto to certain files, just like the makefile does. Finally, CMake-based builds were added to AppVeyor and Travis.

The CMakeFileList.txt probably needs some cleaning up. If anyone wants to make a pass, then we would be much obliged.

[anonimal]

If -march=native is removed by default, then we can readd it based on an envar, like CRYPTOPP_NATIVE_ARCH. If present, the makefile can revert back to CXXFLAGS += -march=native.

@noloader will CRYPTOPP_NATIVE_ARCH be implemented?

[noloader]

@anonimal,

This should be cleared at Commit 0dbc2cc532eb. It should make it to master shortly.

[noloader]

Last call for CMake patches before removing it from the library and adding it to the patch page. If you have something to add, then now is the time to do it.

Also see Remove CMake from GitHub; add it to Patch Page on wiki? and Commit 1c740b0a097aecaa.

[ZahlGraf]

When I was using Crypt++ some years ago for an university project, one reason for choosing Crypt++ over other similar libraries was the availability of the CMake build system. Because my project had to be highly portable and must compile on Windows and Linux natively and for different Android targets.

If Crypt++ had used plain makefiles instead of CMake, I would have never considered it for my project, because using plain makesfiles is a pure pain, if you use them for another system as they are designed for.

However, my user experience was also, that the CMake system of Crypt++ was poorly maintained, which leads to some obvious and easy to fix errors on my first try. I had the impression, that the developers of this library are hardcore makefile users and CMake is just an alternative to the makefiles, which is not really maintained.

This is really strange, since it is clearly more effort to maintain many different makefiles and visual studio project files in parallel, than maintaining a single meta build system like CMake. Yes, CMake works differently and yes it adds a new abstraction to the project, which does not only come with advantages. So developers using CMake must be able to understand the way how CMake works and writing a modular and easy to maintain CMake system for a project requires some good software engineering skills. However writing a well maintainable makefile system, requires this too.

From my point of view, skipping CMake support is a step in the wrong direction. Commiting to CMake as the only build system and using continuous integration and unit-testing for all supported target platforms would be clearly the right direction which will lead to less effort in the long run.

[noloader]

@ZahlGraf,

CMake is still available at CMake on the wiki. Two things have changed. First, the community is responsible for CMake, which was always been part of the deal. Second, all the barriers for editing the CMake sources have been removed. Anyone can make a change and upload a new and improved version to the wiki.

I feel like CMake never achieved a good state; and its still in a half-baked state. I'm partially responsible for the state of affairs. I warned folks I had no experience with CMake, and the folks who wanted it had to maintain it. In hindsiht I never should have checked it in because I did not have the skills to maintain it.

About a year ago I tried to hire someone to build-out the CMake file for the platforms we support to our specifications. I was willing to pay a few thousand dollars for a small contract. The CMake guy I was talking with seemed to be knowledgeable but declined the work.

If you are interested in some of the problems we encountered when trying to maintain it, then see CMake Removal on the Crypto++ wiki. It details the reasons for yanking it from the library sources.

[ZahlGraf]

@noloader
Of course it is your decision as developer and I can completely understand, that there are good reasons on your side to not use CMake. CMake can be frustrating, but it is the same with plain makefiles, if you have no experience with it.

However, when I read the reasons on the wiki-page, the only real reason is the point about unfixed issues of CMake. This is really annoying and should be addressed in the CMake issue tracker.

All other points seems to be related to a lack of experience or a lack of testing. For example: If it is a requirement for you to use on certain platforms a special compile flag (or to not use it for other platforms), it must be tested, as you would do it with other requirements.

CMake is not a build system like makefiles. It is a meta-build system. The generated outcome of CMake is like an own application and the CMake files are the source code for this application. If you write something in the code, you would test it with unit-test, right? Then you should also test your CMake code accordantly to ensure, that the outcome of CMake is correct and consistent on all supported platforms.

Also the point with special target support like ARM and others: Cross compiling does never come without costs. Only for very simply projects you can simply take the same makefiles for cross compiling as you use for native compiling. Thus if you want to support cross-compiling, you have to spend effort to bring in this support into your makefiles. It the same with CMake: You have to spend effort to bring in cross-compile support.

The big advantage of CMake is, that you do not need to maintain many different build systems, but you do not get rid of thinking about the different needs of different compilers and target platforms and especially you do not get rid of testing the build process for all supported compilers and target platforms.

However: Maintaining CMake in addition to 3 other build-systems is of course completely nonsense and simply leads to much more effort and frustration as not using CMake at all. In this case you completely lose the main advantage of CMake and just get all the disadvantages like more effort, more complexity and more testing requirements (or more issues, if something is not tested).

[Marc--Olivier]

@noloader

I am really sorry to hear how much trouble you had to maintain the CMake files or find somebody wanting to maintain it.

However, moving the CMake files to the Wiki will make using it extremely difficult as they are not associated to a code revision any more. E.g., which CMake files should I use to compile cryptopp 5.6.42? Therefore, I would probably never use CMake files from a wiki whereas I would definitely be happy to use CMake files present in a repo and to try to make them work.

=> I would be very grateful if you could keep the CMake files in the repo. If you want to make clear that they are not supported, you could add a message FATAL_ERROR at the beginning of CMakeLists.txt explaining that CMake is not supported. People would have to uncomment this line to use it, either by modifying the file or applying a patch (the latter may be more convenient when using CI tools...). Moreover, you could also add information relative to CMake issues (how to fill them, nobody dedicated to fix them, ...) in the FATAL_ERROR message.

Anyway, thank you very much for your efforts and I perfectly understand that you would rather spend time on coding than maintaining a build system...