Conan: Support Intel compiler in CMake build helper and generators

[danimtb]

I am creating this issue in order to gather the information around the Intel compiler integration with CMake in Conan.

Stating with the work already done by @ohanar here: https://github.com/ohanar/conan/commit/7a62f3ac35573fdc6aa6a25d44111b87c726cb90

I have tried in Linux jusr sourcing _compilervars.sh_ and setting CC=iccandCXX=icpc
and it worked fine with GCC 8.

Then, I wanted to try the compatibility mode of binaries. So I tried to use the binary directly from a GCC 8 profile but got no success.

[100%] Linking CXX executable bin/example
/home/danimtb/.conan/data/pkg/1.0/us/ch/package/b80d46004713aa37d6a90b42e2a326a056a237b5/lib/libhello.a(hello.cpp.o): In function `std::char_traits<char>::length(char const*)':
hello.cpp:(.text._ZNSt11char_traitsIcE6lengthEPKc[_ZNSt11char_traitsIcE6lengthEPKc]+0x29): undefined reference to `__intel_sse2_strlen'
collect2: error: ld returned 1 exit status
CMakeFiles/example.dir/build.make:94: recipe for target 'bin/example' failed
make[2]: *** [bin/example] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/example.dir/all' failed
make[1]: *** [CMakeFiles/example.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

What am I doing wrong? Do I have to provide additional flags to the cmake gcc compilation in order to achieve linkage with the binary created with the intel compiler?

Related to #5590

[ohanar]

The intel compiler links against its own runtime libraries (on linux the main ones are libirc.so and libintlc.so). So, when linking against something built with the intel compiler you either need to explicitly include the intel libraries, or source compilervars.sh to have LD_LIBRARY_PATH set up. Alternatively, you can ask the intel compiler to statically link its runtime libraries with the compile flag -static-intel.

Internally, we have a runtime libraries package that is essentially everything depends upon. It is created by using CMake's InstallRequiredSystemLibraries generated with the intel compiler.

[memsharded]

Got further feedback from users, confirmed that in both Windows MSVC and Linux gcc, it is necessary to link with the Intel libraries to achieve compatibility.

We would need to review then our assumptions, as we thought that it was the other way round, and intel was using the gcc environment and runtime, and thus gcc would be fully agnostic when linking with intel compiled libraries. This means that binaries built with gcc, but linked with intel runtime will not be compatible with other gcc compiled packages.

[ohanar]

@memsharded So it depends. For instance, if I build a shared library libA.so with the intel compiler, and then build a binary my_executable with gcc that links against libA.so, that executable doesn't link against the intel runtime directly. However, to run said executable, you would have to have the intel runtime available in order to load libA.so. Moreover, you will occasionally see shared libraries link in the intel runtime statically, that way a consumer of the library can treat it just like they would treat any other library. It is true that if you had a static library instead, you would have to link my_executable with the intel runtime, however, generally speaking, users of the intel compiler ship the intel runtime alongside their application, so this is rarely a real issue.

To me, it is unfair to blame the binary compatibility of the intel compiler. Rather, as I see it, the fundamental issue is that conan doesn't do anything to track required system libraries and their versions. You can already encounter this with just gcc -- you can't use many binaries on conan-center/bintray if you are on an old linux distro with an old glibc, but conan doesn't know better, and just tries to use them anyway. For instance, trying to build the md5 example on CentOS 6 with gcc 8 using Poco as a shared library gives you the following:

$ conan install .. -o Poco:shared=True
Configuration:
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++
compiler.version=8
os=Linux
os_build=Linux
[options]
Poco:shared=True
[build_requires]
[env]

...
conanfile.txt: Installing package
Requirements
    OpenSSL/1.0.2o@conan/stable from 'conan-center' - Downloaded
    Poco/1.9.0@pocoproject/stable from 'conan-center' - Downloaded
    zlib/1.2.11@conan/stable from 'conan-center' - Downloaded
Packages
    OpenSSL/1.0.2o@conan/stable:35ea421ef4c1d32898da44c28e3cc900bc6459dc - Download
    Poco/1.9.0@pocoproject/stable:6f201f49982ef0e3564ce7801fc7b71a2bd3d4a9 - Download
    zlib/1.2.11@conan/stable:1d877a3df840030e6a8abb74c5ffb9088d08b47a - Download

...
conanfile.txt: Generator cmake created conanbuildinfo.cmake
conanfile.txt: Generator txt created conanbuildinfo.txt
conanfile.txt: Generated conaninfo.txt
conanfile.txt: Generated graphinfo
$ cmake .. -DCMAKE_BUILD_TYPE=Release
...
$ cmake --build .
Scanning dependencies of target md5
[ 50%] Building CXX object CMakeFiles/md5.dir/md5.cpp.o
[100%] Linking CXX executable bin/md5
/root/.conan/data/Poco/1.9.0/pocoproject/stable/package/6f201f49982ef0e3564ce7801fc7b71a2bd3d4a9/lib/libPocoNet.so: undefined reference to `std::__detail::_List_node_base::swap(std::__detail::_List_node_base&, std::__detail::_List_node_base&)@GLIBCXX_3.4.15'
/root/.conan/data/Poco/1.9.0/pocoproject/stable/package/6f201f49982ef0e3564ce7801fc7b71a2bd3d4a9/lib/libPocoFoundation.so: undefined reference to `__fdelt_chk@GLIBC_2.15'
/root/.conan/data/Poco/1.9.0/pocoproject/stable/package/6f201f49982ef0e3564ce7801fc7b71a2bd3d4a9/lib/libPocoMongoDB.so: undefined reference to `memcpy@GLIBC_2.14'
/root/.conan/data/Poco/1.9.0/pocoproject/stable/package/6f201f49982ef0e3564ce7801fc7b71a2bd3d4a9/lib/libPocoMongoDB.so: undefined reference to `std::__throw_out_of_range_fmt(char const*, ...)@GLIBCXX_3.4.20'
/root/.conan/data/Poco/1.9.0/pocoproject/stable/package/6f201f49982ef0e3564ce7801fc7b71a2bd3d4a9/lib/libPocoData.so: undefined reference to `std::__detail::_List_node_base::_M_transfer(std::__detail::_List_node_base*, std::__detail::_List_node_base*)@GLIBCXX_3.4.15'
/root/.conan/data/Poco/1.9.0/pocoproject/stable/package/6f201f49982ef0e3564ce7801fc7b71a2bd3d4a9/lib/libPocoFoundation.so: undefined reference to `std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&, unsigned long, std::allocator<char> const&)@GLIBCXX_3.4.23'
/root/.conan/data/Poco/1.9.0/pocoproject/stable/package/6f201f49982ef0e3564ce7801fc7b71a2bd3d4a9/lib/libPocoNet.so: undefined reference to `std::__detail::_List_node_base::_M_unhook()@GLIBCXX_3.4.15'
/root/.conan/data/Poco/1.9.0/pocoproject/stable/package/6f201f49982ef0e3564ce7801fc7b71a2bd3d4a9/lib/libPocoCrypto.so: undefined reference to `operator delete[](void*, unsigned long)@CXXABI_1.3.9'
/root/.conan/data/Poco/1.9.0/pocoproject/stable/package/6f201f49982ef0e3564ce7801fc7b71a2bd3d4a9/lib/libPocoMongoDB.so: undefined reference to `std::__detail::_List_node_base::_M_hook(std::__detail::_List_node_base*)@GLIBCXX_3.4.15'
/root/.conan/data/Poco/1.9.0/pocoproject/stable/package/6f201f49982ef0e3564ce7801fc7b71a2bd3d4a9/lib/libPocoMongoDB.so: undefined reference to `operator delete(void*, unsigned long)@CXXABI_1.3.9'
collect2: error: ld returned 1 exit status
gmake[2]: *** [bin/md5] Error 1
gmake[1]: *** [CMakeFiles/md5.dir/all] Error 2
gmake: *** [all] Error 2

Due to this issue, we only use internally built packages (even for header-only packages), as there is too much in the way of dependencies that is not tracked by conan for us to reliably use bintray or conan-center.

[lasote]

Added to 1.21 to look into and decide if the settings approach of https://github.com/conan-io/conan/pull/6052 is correct

[lasote]

Looks good. Basically, the steps are, call the compilevars.bat (instead of the vcvars) and pass the correct toolset. We will try to incorporate the changes from https://github.com/ohanar/conan/commit/7a62f3ac35573fdc6aa6a25d44111b87c726cb90 at 1.22

[memsharded]

I am assigning this to @SSE4

There is a ton of work https://github.com/ohanar/conan/commit/7a62f3ac35573fdc6aa6a25d44111b87c726cb90, but we still need to fully understand those contributions. So please @SSE4 I'd suggest the following:

• Start extracting from that either the most useful bits, and/or the lowest hanging fruits, things that are very low risk
• I'd prefer some small PRs, that explain a bit what the things are doing
• I would like to avoid very specific code like the proposed _env_diff in that commit. If some new functionality is needed, better treat it separately.
• Tests will be needed.

Please let me know if you can manage this, or are already at full capacity with ConanCenter. Thanks!

[SSE4]

hi @memsharded
I am going to allocate some time this week on this item. need to renew my Intel compiler licenses first, as they are already expired. will keep you informed.

[memsharded]

I am going to allocate some time this week on this item. need to renew my Intel compiler licenses first, as they are already expired. will keep you informed.

Excellent. But you can use the OSS intel compiler, right?

[SSE4]

yes, I will try this one

[ohanar]

The _env_diff function is basically a generalized version of vcvars_dict. It could be useful as a general utility for converting a shell command to an environment patch.

@SSE4 If you have any questions let me know.

[memsharded]

Some things started to be merged in 1.24, but most work still pending for 1.25.

[memsharded]

This has been implemented in latest PRs from @SSE4 , will be in 1.26.