Julia: switch from GNU Make to CMake?

[StefanKarpinski]

I'm creating this issue to discuss the possibility of standardizing our build system on CMake, which seems to be getting significant traction as a saner, cross-platform build system than Make. If we decide to use this to build Julia, we may want to standardize on it as a standard build system for Julia packages that need to compile things too.

[ViralBShah]

1519 is also related.

[ViralBShah]

Decision is that this is generally a good thing, but not important now since the Windows build works with the current build system.

[ViralBShah]

CMake would be useful if we wanted to build with MSVC and Xcode.

[stevengj]

It would be useful to be able to assume that cmake is available when building packages that wrap around existing source code.

[nolta]

Does Xcode provide cmake?

[ViralBShah]

Xcode does not provide cmake.

[ihnorton]

Small argument in support of CMake: build times, see timings below building llvm-3.2 (all clean source dirs) on core2 duo, 4 GB, Ubuntu 12.04.

./configure Makefile:

time make -j2
...
real    20m45.724s
user    35m4.664s
sys 2m13.084s

CMake Makefile:

time make -j2
...
real    13m31.478s
user    23m6.919s
sys 2m8.340s

CMake Ninja:

time ninja -j2
...
real    13m30.552s
user    23m16.759s
sys 2m1.472s

I used default configure and CMakeLists options (so maybe slightly different options...). Ninja doesn't make a big difference in cold build time on Linux, but small changes or no-op builds are much faster (no-op: ~0s vs 3.5s). And If Julia ever supports MSVC, the improvement is enormous: escape from msbuild.

Some relevant discussions of Ninja speed:
http://news.ycombinator.com/item?id=3896622
http://public.kitware.com/pipermail/cmake-developers/2012-February/003318.html
https://plus.google.com/108996039294665965197/posts/SfhrFAhRyyd

[stevengj]

Scratch my earlier comment, by the way. I've concluded that cmake is useless for packages because packages can't assume that Mac or Windows systems have compilers (hence you have to distribute binaries for those systems).

[vharavy]

I am not an advocate of CMake, but in my opinion current Julia build system demands serious attention. I spent about 4 hours trying to built 64-bit Julia on Windows. The only working method for now is using cross-compile.

Currently I am investigating the possibility of using Julia at work. I found Julia to be a way ahead of R and MATLAB in the matter of performance but there are some moments:

1. At the moment the only way to build Julia on Windows is using MinGW and MSYS. I find it a little bit annoying. It would be very useful to have ability to build Julia using other compilers, notable Microsoft Visual C++ and Intel C++ Composer. The latter includes math library and Intel Math Kernel Library which contains implementation of BLAS, LAPACK, SparseBLAS, Fourier Transforms, PARDISO, Vector Math Library etc. At the company I work we have an Intel C++ Composer and I believe it will be possibly for us to provide pre-built 64-bit binaries (and 32-bit if required) of Julia for Windows. Also, Intel C++ Composer for Linux is free for open-source projects so there is a possibility to get similar builds for Linux.
2. Currently installation of Julia includes by default a lot of functionality. It would be nice to have the ability to select whether to install it or not - i.e. move stuff like big integers, sparse matrices, Fourier transforms into packages and exclude them from the build. Thus Julia will build faster, start faster, consume less disk space and memory.

P.S. I would glad to participate in the development (especially in first clause).

[vtjnash]

.1a. There's already the option of downloading pre-built binaries from https://code.google.com/p/julialang/downloads/list
However, building with the intel compiler could give a nice speed boost.

.1b. The MKL can already be used with MinGW/MSYS/llvm/gcc compiled julia. I've made this change before locally. However, we don't have permission to distribute MKL so the default install uses OpenBLAS.

.2. While it would speed up the first build, it makes no difference to the speed of future builds to exclude these other libraries. Excluding them would save < 20 MB of disk space and have a negligible impact on startup speed (the current slow startup is unrelated to the number of libraries -- there's a different issue open for fixing that). Thus, there really isn't a particularly good reason to exclude these items.

PS. The windows build instructions are at https://github.com/JuliaLang/julia/blob/master/README.windows.md for both native and cross builds. Others have found that they are correct. However, Windows is very slow at some critical operations and thus will take much longer than 4 hours for the initial build. I believe the problem lies with how much better optimized linux file access is compared to Windows and since compiling inherently requires reading and writing a large number of files. Switching to cmake will probably help reduce compile time, but only somewhat.

[vharavy]

1a. There is no 64-bit pre-built binaries for Windows.,
1b. My company have the rights to build and distribute software with Intel C++ Composer.

P.S. I have read those Windows build instructions very carefully and still I have got some issues. First one is make produces several errors like the following:

make: *** No rule to make target `/d/Temporary/usr/bin', needed by `release'.  Stop.

It is cured by creating these folders manually. The second one appears when building some of the dependencies. It seems that somewhere on the way of compiling make messes with destination directory and produces output files in

/d/d/Temporary/usr/lib

while expecting them to be in

/d/Temporary/usr/lib

I can provide you with some details if you like. But my post was about build system in general. I believe it is too complicated for building Windows builds. E.g. necessity to install MinGW and MSYS (around 800Mb in total) seems strange. My opinion is build system of any software should have as least requirements (or dependencies) as possible.

[vtjnash]

1a. Not yet (v0.1 didn't build on windows 64), but there will be starting with v0.2 in a month or so. In the meantime, the 32-bit binary is identical in many respects.

1b. That would probably be helpful. We've talked about doing that for all of the builds, but nobody has had sufficient time or interest to make it happen so far. It is probably easiest to get the linux version working first. IIUC, the right to distribute programs compiled with icc is separate from the right to distribute MKL. Switching a pre-built 32-bit julia binary to MKL should be possible by editing the blas name in build_h.jl file and deleting the sys.ji cache file.

P.S. I think I found and fixed the cause of the first error. It is probably more appropriate to open a new issue for the second item (although it may be fixed by the first?).

[JeffBezanson]

At this time we believe we will not switch to cmake.

[twadleigh]

Is this an issue closed only for lack of urgency/volunteers? If so, I might be willing to do some work in toward this end.

[tkelman]

I think it's definitely worth revisiting this, see some recent discussion in https://github.com/JuliaLang/julia/issues/8937#issuecomment-63590284 (and this is the more correct place to discuss it, thanks for finding this issue!)

Especially now that we have the libgit2 dependency which requires cmake to be installed, and we have partially functional MSVC builds, I think this would be a good transition to investigate further. I've done a very small amount of work in this direction but would be interested in expanding it. Perhaps starting with something small like configuration probes for common build issues that we've encountered?

[ViralBShah]

Certainly worth revisiting. Time to start learning cmake.

[maleadt]

CMake also includes "External Project" functionality which should allow managing deps. Although that would be quite a big porting effort...

[stevengj]

Given that it's probably not practical to port all of the dependencies to cmake, what will we gain by switching Julia itself to cmake?

[tkelman]

A few things:
1) a separate "configure" step to detect system configuration and requirements ahead of time, rather than finding problems midway through a long build - doing this in cmake would be more modern and maintainable than autoconf or a custom script
2) sane non-hacky support for MSVC
3) other generators like ninja which are much better than make at incremental builds
4) out-of-tree builds (I _really_ want this) mostly works with makefiles now

[gitfoxi]

The problem with cmake is that cmake hard to build and has version issues. Like if you want to build cmake on an old system you need old cmake and hope what you actually wanted to build doesn't use new features. Or you can use old cmake to build incrementally newer cmake ...

Why not SCons? SCons is easy assuming you have Python 2.4 or newer.

If you're not happy with some of SCons tradeoffs they're nice to provide a comparison with lots of alternative build systems.

[tkelman]

How old of a system are you talking about? Yes cmake has grown new features over time, but it also makes a point of declaring your minimum version requirements right up front. We haven't had any complaints yet about cmake versions not being new enough for libgit2 on master, so I think all the versions on systems people are commonly building Julia on have been good enough so far. Modern cmake has a fairly simple bootstrap system to compile it from source if you're somehow on a distribution where you can't get new enough versions from your package manager or binary installers. It's not at all hard to build relative to the things we're using in Julia, and most people shouldn't need to build it anyway.

We already need cmake. Many of our dependencies have cmake build systems already written, so much of the fragile boilerplate repetition we currently have in deps/Makefile should be possible to simplify quite a bit using ExternalProject. Unless you want to convert all of the dependencies to scons there's no reason to be different.

[twadleigh]

@tkelman, I was thinking I might set some time aside to help work on moving this forward over the holiday. I'll send you an e-mail.

[vtjnash]

discussion can continue on the PR (#11754), if necessary. however, i don't think we really need to revisit this. the slow build times and msvc issues are as much a function of our dependencies (if not more), and switching julia to cmake wouldn't address that.

[tkelman]

msvc on gmake is a hack that doesn't work at all in msys2 (they broke that) so that would really be the main motivation for switching. agreed though that the deps are the messy part of this.

[habemus-papadum]

one small win for having a cmake build it that is some ide's have very good support for project configuration (clion and qt creator mainly, and but also in a slightly less integrated way, eclipse/xcode/visual studio (clion/qt creator make an effort at grokking cmake, and allowing users to make intelligent changes to both the source and cmake build files, the others are supported by cmake but there are limits to the scope of changes that are convenient to make in a cmake generated IDE project))

As an example, since llvm uses cmake, a great way to start grokking how it works it to clone it, and open the CMakelists.txt in clion and start from various examples and hop "backwards" into deeper parts of the core system. Not only do you have robust/template aware navigation within the entire code base, but you can easily run/debug examples, or just start hacking including adding new files, new libraries, etc. The net experience is a much lower barrier to entry for new developers who want to grok the code base. (nb. clion can be slow on llvm due to its size and project complexity though I find usable on a high end workstation)

Along the same lines, aside from cmake, a minimal but useful improvement would be to make sure that the current Makefile based system generates a compile_commands.json file (cmake builds provide this easily, but there are many other systems (e.g.Bear) that also achieve this for other scenarios like Makefile projects). This opens the possibility for using various clang based tools (e.g. rtags, FB's nuclide, maybe about 8 others) to help initially grokking the code base.

Basic point it that using a main stream build system opens the possibility for new developers to use existing tools accelerate the speed at which they can grok julia internals.

Should these comments be on the PR(#11754)?