Vcpkg: Building vcpkg on OSX using LLVM's clang 7

[mcraveiro]

Hi vcpkg developers,

thanks for an amazing tool. I have managed to successfully use it for my Linux and Windows builds and I am very happy with it :-) my last task is to update the build system on OSX to use vcpkg and here I am finding some difficulties. The version of XCode that came with the machine is fairly old and upgrading to a recent XCode (or even the XCode command line tools) requires an OS upgrade, which is not feasible at present. So I decided instead to use the LLVM supplied OSX binaries [1].

I unpacked the LLVM+Clang files locally, git cloned vcpkg and then bootstrapped as follows:

LDFLAGS="-L${PACKAGE}/clang+llvm-7.0.0-x86_64-apple-darwin/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names" CC=${PACKAGE}/clang+llvm-7.0.0-x86_64-apple-darwin/bin/clang CXX=${PACKAGE}/clang+llvm-7.0.0-x86_64-apple-darwin/bin/clang++ ./bootstrap-vcpkg.sh

Where ${PACKAGE} is a full path to the directory in which I expanded the downloaded LLVM + Clang. The build then fails to link:

Undefined symbols for architecture x86_64:
  "std::__1::__fs::filesystem::directory_iterator::__dereference() const", referenced from:
      vcpkg::Files::RealFilesystem::get_files_non_recursive(std::__1::__fs::filesystem::path const&) const in files.cpp.o
  "std::__1::__fs::filesystem::recursive_directory_iterator::__dereference() const", referenced from:
      vcpkg::Files::RealFilesystem::get_files_recursive(std::__1::__fs::filesystem::path const&) const in files.cpp.o
  "std::__1::__fs::filesystem::path::__filename() const", referenced from:
      vcpkg::Files::RealFilesystem::find_file_recursively_up(std::__1::__fs::filesystem::path const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const in files.cpp.o
<strip>

Apparently this is related to a move inside of clang, where FS has moved from libc++experimental to libc++fs [2]. Following the advice on that link, I managed to fix the issue by adding -lc++fs to my linker flags:

LDFLAGS="-L${PACKAGE}/clang+llvm-7.0.0-x86_64-apple-darwin/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lc++fs" CC=${PACKAGE}/clang+llvm-7.0.0-x86_64-apple-darwin/bin/clang CXX=${PACKAGE}/clang+llvm-7.0.0-x86_64-apple-darwin/bin/clang++ ./bootstrap-vcpkg.sh

I suspect vcpkg will need to have some internal knowledge of the FS library, or wait for CMake to do the right thing [3].

A second, not entirely related problem, but probably worth reporting is the handling of clang in the boostrap script. I see this error when I run it:

/LOCATION/vcpkg/scripts/bootstrap.sh: line 182: [: clang version 7: integer expression expected

Where LOCATION is the path to my vcpkg directory. The problem stems from the selectCXX() function in scripts/bootstrap.sh, which, if I understood it right, is not expecting clang to be the C++ compiler. This does not cause any problems though.

[1] http://releases.llvm.org/7.0.0/clang+llvm-7.0.0-x86_64-apple-darwin.tar.xz
[2] https://www.reddit.com/r/cpp/comments/92hmaf/libc_support_stdfilesystem_now/
[3] https://gitlab.kitware.com/cmake/cmake/issues/17834

[mcraveiro]

A second set of problems was a failure to link the vcpkg supplied libxml:

  "_iconv", referenced from:
      _xmlIconvWrapper in libxml2.a(encoding.c.o)
  "_iconv_close", referenced from:
      _xmlFindCharEncodingHandler in libxml2.a(encoding.c.o)
      _xmlCharEncCloseFunc in libxml2.a(encoding.c.o)
  "_iconv_open", referenced from:
      _xmlFindCharEncodingHandler in libxml2.a(encoding.c.o)
ld: symbol(s) not found for architecture x86_64

[mcraveiro]

I have fixed both of these problems as follows:

1. Do not build libxml2 from vcpkg for OSX. It appears it does something funny with iconv and the libconv that is supplied by vcpkg does not seem to be fully functional. To be totally honest, I didn't quite get to the bottom of it but there are several reports of similar linking errors [1], [2]. The solution for this problem was to rely on the OSX supplied libxml2 (includes in /usr/include for good measure).
2. It seems vcpkg (via bjam's defaulting mechanism) builds boost on c++ 14 for GCC and MSVC but, for some reason, not for clang [3]. This results in those weird and wonderful linking errors. I patched my ports as follows:

diff --git a/ports/boost-modular-build-helper/CMakeLists.txt b/ports/boost-modular-build-helper/CMakeLists.txt
index dc4d13dd..5654e4cd 100644
--- a/ports/boost-modular-build-helper/CMakeLists.txt
+++ b/ports/boost-modular-build-helper/CMakeLists.txt
@@ -126,6 +126,7 @@ add_custom_target(boost ALL
         threading=multi
         threadapi=pthread
         debug-symbols=on
+        cxxstd=14
     WORKING_DIRECTORY ${SOURCE_PATH}/build
 )

I now have a green build on OSX. However, for anyone trying to replicate this, there is one note of warning about my setup: I am using LLVM's Clang [4] to build my vcpkg dependencies but on Travis I build my code with XCode 10. It seems to work thus far but if you start to see some weird linking errors, I suggest you try to move to LLVM's Clang on travis as well or build vcpkg with XCode 10 - basically best not to mix and match.

[1] https://github.com/gibiansky/IHaskell/issues/563
[2] https://stackoverflow.com/questions/12619600/libiconv-and-macos
[3] https://github.com/Microsoft/vcpkg/issues/4437
[4] http://releases.llvm.org/7.0.0/clang+llvm-7.0.0-x86_64-apple-darwin.tar.xz

[ras0219-msft]

Thanks for using vcpkg and glad that your experience on Windows & Linux was so good! I'm sorry about the trouble on OSX though.

The version of XCode that came with the machine is fairly old and upgrading to a recent XCode (or even the XCode command line tools) requires an OS upgrade

Generally, we just recommend users to install a gcc-6 from homebrew to build the vcpkg tool (note that the libraries will still be built with $CXX by default, which is AppleClang). Could you go into a bit of detail about why you went with using the LLVM Clang distro instead of brew install gcc?

In the case of libiconv, I see that we're currently assuming that OSX works like Linux instead of Windows -- specifically that iconv will magically spring forth from the CRT (which certainly appears false here :)). So we just need to get a build for iconv working and that should resolve the issue. I've opened #4483 to track that.

[mcraveiro]

Hi @ras0219-msft,

Hey, I don't want to sound too negative :-) whilst it took me a little while to sort out OSX, I'm still extremely pleased with vcpkg! I finally have a single setup for all platforms that is very easy to maintain and even debug! :-D

With regards to LLVM Clang - to be honest, I just wanted to make my environments as close as possible as AppVeyor/Travis; I started by first trying XCode 10, which is what I use to build on Travis. When that failed I first tried Clang because that was the closest to XCode (i.e. same compiler, different versions + Apple patches). Had that failed I would have moved on to GCC :-) On the plus side, its nice not to have to install brew - I'm not really an OSX expert and my experiences in the past with Fink/Brew were not entirely positive :-)

In conclusion, with regards to my personal project, I think using LLVM Clang works well, modulus that silly problem of the move of std::filesystem. Of course, I do feel somewhat uneasy about having two versions of the compiler (XCode and LLVM Clang), but using LLVM Clang on Travis just seemed like too much work: its a 300 MB download, OSX does not support XZ natively (the LLVM package is compressed using xz) etc. Since it all linked and the binaries work, I just left it at that. I may revisit it in the future.

In terms of the vcpkg project, I personally recommend supporting XCode and LLVM Clang because it will make the development story more streamlined for the average developer. Soon XCode will have a recent version of Clang and then you can simply integrate vcpkg with it, rather than having to install brew and gcc. But those are my 2 Angolan Kwanzas :-)

Thanks for the iconv, I'm sure it will come in handy in the future.

[mcraveiro]

Actually, let me reply specifically to this point:

Generally, we just recommend users to install a gcc-6 from homebrew to build the vcpkg tool (note that the libraries will still be built with $CXX by default, which is AppleClang). Could you go into a bit of detail about why you went with using the LLVM Clang distro instead of brew install gcc?

I was worried of building the libraries with $CXX given it is an version of XCode/AppleClang, and then try to link using XCode 10 in Travis. I wanted the libraries to be as similar as possible to Travis. The building of vcpkg itself with LLVM Clang was mainly a byproduct of that, and since I got it working, I thought why not.

I still think not requiring brew is a plus though.

[ras0219-msft]

Yeah we are _strongly_ looking at removing the std::filesystem dependency (it also is a problem on linux!), but it's quite a bit of work since we manipulate a lot of files :D

Thanks for the explanation!