Or-tools: CMake build support future

[navidR]

I am trying to add or-tools to vcpkg [1]. But before that, I need to know what is future of cmake support in or-tools? I see it is experimental. Experimental means there is a good chance of removing it in future? Or experimental in this context means it is going to get more stable and eventually fully supported in or-tools? (In this case, I can rely on CMake support of or-tools team and add it to vcpkg).

1) https://github.com/Microsoft/vcpkg/issues/692

[lperron]

We would like to drop make in favor or cmake completely.

There are still a few issues, but this is the future of the build part of
or-tools.

Thanks
Laurent Perron | Operations Research | [email protected] | (33) 1 42 68 53
00

Le lun. 25 juin 2018 à 07:30, Navid Rahimi [e-mail] a
écrit :

I am trying to add or-tools to vcpkg [1]. But before that, I need to know
what is future of cmake support in or-tools? I see it is experimental.
Experimental means there is a good chance of removing it in future? Or
experimental in this context means it is going to get more stable and
eventually fully supported in or-tools? (In this case, I can rely on CMake
support of or-tools team and add it to vcpkg).

1. Microsoft/vcpkg#692 https://github.com/Microsoft/vcpkg/issues/692

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/google/or-tools/issues/732, or mute the thread
https://github.com/notifications/unsubscribe-auth/AKj17UrEs5fysT7GcZl1JYWbt7hGf9lLks5uAHWMgaJpZM4U1iI2
.

[Mizux]

Hi,

Yes it is intended to use CMake as default build (at least for C++) and to replace our current Makefile based system.
For vcpkg it's a long term goal but before this I want OR-Tools deps to be also available in vcpkg.

i.e. currently in CMakelists.txt I build zlib, gflags, glog, protobuf, mizux/CBC and merge everything.
I would like to depends on vcpkg packages instead but:

ZLIB: I want to use my CMake PR https://github.com/madler/zlib/pull/337
GFlag: better if we could use gflags::gflags target name https://github.com/gflags/gflags/pull/247
CBC:

• on github it is split in 5 repos (CoinUtils, Osi, CLP, CGL, CBC) we now use this layout for unix since most distro use this instead of the svn standalone CBC build
• Since COIN-Or doesn't want to provide support for CMake, I did one here https://github.com/Mizux/Cbc but i use the svn one repo version instead of the splited ones -> need to redo it

Then we will provide an OR-Tools vcpkg after.
note: If you want to work on our dependencies to vcpkg them all...

CMake is currently experimental since:

• dependencies management is not still clear (currently we add_subdirectory() all our deps using AddProject tricks but maybe we should use a compile/install/use approach cf these discussions
  https://github.com/abseil/abseil-cpp/issues/80
  https://github.com/abseil/abseil-cpp/issues/38
  https://github.com/abseil/abseil-cpp/issues/111
• Currently I only did C++ / python swig support. I need to figure out how to do the .Net and Java support since CMake is a C/C++ project generator at its core.
  e.g. CMake learn C# in CMake 3.8 https://cmake.org/cmake/help/v3.8/release/3.8.html ...
• We would like to hear from community (tester) if my CMake support is stable enough and work for them

[navidR]

It is good news to hear we are going to rely on CMake. And only C++ would suffice I think (for now).

I am fighting right now with https://github.com/google/or-tools/blob/master/CMakeLists.txt#L12

It seems you are redefining find_package. I think it is better to have something like ortools_find_library than redefining find_package. (LLVM does this with add_llvm_library). redefining find_package does not play well when I want to integrate vcpkg and use dependencies of or-tools from vcpkg.

For example :

find_package(gflags QUIET)
if(NOT TARGET gflags::gflags AND NOT gflags_FOUND)

This makes much more sense than trying to download-compile the dependencies all the time. With this I am able to do cmake -G "Ninja" .. -DCMAKE_TOOLCHAIN_FILE:STRING=$VCPKG_PATH/scripts/buildsystems/vcpkg.cmake -DBUILD_PYTHON=OFF -DBUILD_TESTING=OFF easily. (Although for now the find_library does have weird errors, since we are redefining it).

[Mizux]

Yup that the problem with CMake if you depends on Foo::Foo but include it using add_subdirectory() then find_package could fail because it doesn't check if the target is already present in your current "CMake context".

or maybe I should rewrite

option(ORTOOLS_STANDALONE ON)
...
if (ORTOOLS_STANDALONE)
  add_subdirectory(dependencies/bar)
  add_subdirectory(dependencies/foo)
endif()
...
if(NOT TARGET Foo::Foo)
 find_package(Foo required)

But if Foo depends on Bar and doesn't check that way, thus I need to overload find_package

BTW thanks for your feedback I think I will try to change this sooner...
In fact we want to keep two modes:

• one mode you find your dependencies installed in your system aka find_package
• one mode our CMake retrieve compile install all dependencies, let's call it standalone (it's better when creating OR-Tools python/java/c# standalone archive).

I know vcpkg is mainly for C++ developers but OR-Tools also provide swig wrapper in Python, Java and .Net and CMake is not really built for this...

[navidR]

Are you guys open to looking into a PR? I can restructure the project a little bit. I use clone vcpkg itself in our code repository. It makes management of dependencies way more easier. You should basically have a git submodule of vcpkg and then you can customize vcpkg to your need.

So basically the flow for compiling or-tools will be like this :

git clone $url --recursive
cd ortools/vcpkg
./bootstrap
./vcpkg install gflags glogs Cbc protobuf
cd ..;
cmake -G "Ninja" .. -DCMAKE_TOOLCHAIN_FILE:STRING=$(PWD)/vcpkg/scripts/buildsystems/vcpkg.cmake -DBUILD_PYTHON=OFF

That's about it. In all supported platform that is going to work. I am doing this in our private code repository. And you can have a private patch of vcpkg for your dependencies. As you wanted a customized version of zlib or anything else.

[Mizux]

While I'm agree with your flow, I don't think we need to have some vcpkg file here.

I mean vcpkg is a tools by Microsoft to fix their own package management issue (yes it's multi platform but we don't wait them to build some tools)...
I mean on GNU/linux distro you already have package manager and on MacOS you have homebrew
So objective is for OR-Tools C++ deliveries:

• Linux Distro Package for Linux
• Homebrew formula for MacOS (which should be in homebrew/homebrew-core)
• vcpkg for Windows
  Further more Microsoft provide a repo (i.e. Microsoft/vcpkg) for all package recipe (like homebrew)

So, while I'm open to have some change to CMake to be more usable, I think vcpkg stuff should live directly in https://github.com/Microsoft/vcpkg/tree/master/ports not in this repo.

i.e. here we should have an agnostic CMake that you can use/embed with your prefered tools (e.g. vcpkg, adment, conan, yocto recipe, android toolchain etc...).

note: Homebrew Makefile test https://github.com/Mizux/homebrew-or-tools/blob/master/or-tools.rb basically I want the same workflow for CMake, i.e. you can have installed all dependencies in the host system, then having a way to CMake/Make OR-Tools using them instead of compiling them.

[navidR]

Just as small fix, I think you have to reinvestigate vcpkg. The developers of vcpkg designed it to be integrated into every kind of project (not as a system-wide package manager, but as a customized dependency manager integrated into your codebase rather than being universal). It is basically dependency management tool. So having a one united dependency management tool for a C++ project is a huge sum. While the cost is very low and we have to only maintain a private branch of vcpkg (if we don't need anything special to the main branch, even that is not requirement).

The problem with using Linux distro package managers or homebrew (as I use right now) is mostly inconsistent version between libraries. Some are older, some are newer. vcpkg would give us one united and robust dependency management. Something like Rust's cargo. And as far as I know, some enterprises do use the approach I am explaining to you in their private repos.

But I agree with your other point. The integration of vcpkg would mean a headache for Android, embedded stuff, or custom targets. (Since it only supports Windows/Linux/OSX, right now. But I have heard from its developers they are going to add a way to have custom triplet).

Anyways, I hope we will see more cmake improvement to or-tools.

[ras0219-msft]

they are going to add a way to have custom triplet

Just to bump this up, we definitely have this capability[1]! Additionally, we accept CMake toolchain files[2], and have several users who've successfully build libraries for Android and iOS via that mechanism :)

[1] https://github.com/Microsoft/vcpkg/blob/master/docs/users/triplets.md
[2] https://github.com/Microsoft/vcpkg/blob/master/docs/users/triplets.md#vcpkg_chainload_toolchain_file

[lperron]

Currently no plan, sorry.
CMake support is still not finished after more than 1 year of work.
And no visible increase in user adoption of vcpkg.