godot 🚀 - Consider moving the buildsystem to Meson or another build system

For what I understand, the buildsystem tooling must provide:

Cross-compilation (compile to every possible target platform from the same host platform).
- By "possible" I mean that the host allows it (so macOS and iOS are only possible in a Mac machine, and from Linux with OSXCross).
A way to generate files (translation from PO, shader classes from GLSL, documentation data from XML).
- SCons knows that the generated files are dependencies, so changing them trigger a rebuild of relevant files. This should be considered too.
Multiple targets and options within targets (target=debug/release_debug/release, architecture as x86 or ARM, 32 or 64 bits).
Options to customize the build/enable extra targets (gdnative wrapper, no mono glue, enable/disable: modules, drivers, deprecated features, etc., set libs to compile statically using the built-in code or use the system libs, for each one individually).

I don't know Meson, but if it provides everything above it should be fine (if I didn't forget something). Of course, someone has to go the excruciating pain of rewriting the buildsystem in Meson and then show the tooling and build times are effectively better for Godot than SCons.

Note that a few third-party dependencies requires a patch to be compiled with Godot. I noticed Wrap has a patch support, but it looks like the patch must be available somewhere to download, while I think it's easier to manage if they are still part of the repository. Also, some of them require special compilation flags that should not be applied to the whole build.

vnen on 24 Jan 2018

I think such a proof-of-concept would be very valuable because

it gives us a better idea if the new build system handles everything we need and how well it does it
it gives us concrete instead of abstract data to discuss

For the second point, this means we can see the difference in build speed for Godot, we can see how much easier (or more complex) our build scripts become, and we can compare the differences more easily to make a choice instead of making educated guesses based on what whe think it could offer.

--

Now, personally, I have to say I'm neither for nor against this, but I think there are more low-hanging fruit to chase instead of (or before?) changing the entire build system. Especially at this critical point in time, where as many hands as possible are needed to make sure 3.0 and the first patches are in the best shape possible. But if we want to improve the build system, I'd look at the CI build times and the hassle of the Mono build situation. (I think these are on akien's TODO/list of aggrievances?)

Also, the bugged parallel builds on Windows suck, but that is more on a personal would-be-nice-if-it-was-fixed basis.

TL;DR:

I agree SCons documentation/resources are lacking, the build scripts being easy Python somewhat counteracts that, IMO
As a new-ish contributor, the Godot build system was one of the least annoying to setup, really low time to first successful compile
I'd rather see the effort spent on improving 3.0/3.1 as much as possible and maybe think about this once those are stable
Better CI build times & easier Mono builds could be more important

My opinions. :)

mhilbrunner on 24 Jan 2018

I absolutely do believe that getting 3.0 out there and focusing on the following releases (3.0.1, 3.1) should take priority over this, for the record.
I know why people don't like to work on buildsystems (since it's really only for the benefit of other developers/particularly tech-savvy end users - which isn't unimportant, it's worthwhile to make the devs' and contributors' lives easier, but ultimately the end users' happiness is the goal) and there is this whole "opportunity cost" thing, ie. when the time spent (or lost/wasted, depending on who you ask) on improving the build system could be spent on improving the project itself. (Unless you're like me and can't really work on the C++ side of things anyway.)

And ultimately, SCons really does "just work" which for most developers out there will probably be good enough, so while this is something that I hope does eventually happen, I'm not holding out my breath for it for 3.1 or even 3.2, if it even garners enough support to actually end up being worked on to begin with.

All this said, I might kickstart the POC later this week in my fork, for those who are willing to follow the progress on that (and likely help me out, as I don't have a huge amount of experience writing build systems), but I won't be opening a pull request for this for a while most likely.

RiverMesa on 24 Jan 2018

👍5

I dont really find SCons to be slow, and its flexibility (which we take a ton of advantage of) is probably unmatched by anything else. SCons is also extremely proven, i find it that it never screws up a build.

Honestly, unless someone volunteers to rewrite the whole build system (not just a proof of concept), and shows that it's either simpler (a lot less code) or much faster, I would not even consider anything else, not even a bit.

reduz on 24 Jan 2018

👍12 👎3

Now, personally, I have to say I'm neither for nor against this, but I think there are more low-hanging fruit to chase instead of (or before?) changing the entire build system.

That's not really detrimental for this change. I mean, most of Godot contributors work on free time and work on what they want. For @NullConstant's case it seems is either "do this or do nothing" (not saying you don't want to fix things, just that it's harder to work with the actual C++ code base and bug hunt on something unfamiliar), so doing this is a good thing in general.

Also, the "there are better things to do" argument is the way to have nothing done IMO. Of course nothing will be merged until we are confident that it won't break stuff and won't be done close to a release. I agree there are more important stuff, but it doesn't mean this can't be started.

The thing is that someone working on this will likely have to do this alone. Of course specific questions can be answered by the core devs, but anymore than that won't happen. That's why SCons was never considered to be replaced: no one so far wanted to get the hands dirty. Like I said about Bullet before: someone has to get it to work and prove it's possible and better, otherwise we will stay with the status quo.

vnen on 24 Jan 2018

I do think SCons is slow though. Not much slow, but things like the progress status and the cache make it slower for rebuilds (using disk I/O to read from the cache makes it slow), so I disable those when rebuilding. It does take a bit to start doing stuff, after that it's quite fast.

Don't know if Meson will be significantly faster, but if someone is willing to try I'm not against it.

vnen on 24 Jan 2018

Oh sure, if someone takes the time to do it and shows that its an improvement big enough to warrant switching, I'm all for it :)

mhilbrunner on 24 Jan 2018

Like I said, I'm not really equipped to deal with hunting C++ bugs or adding new features on that front, so I'm okay with doing this instead.

I suppose that this gives me (and/or anyone else working on this in the future) a pretty clean, two-fold goal to reach for:

get Godot to compile with Meson in any form to start with
make it as easy as SCons is at the moment, possibly even more so (by improving the speeds)

And I suppose that SCons' speed is not terrible-terrible (the bulk of the time of the build process is from the actual linking and compiling anyway, so the slowdown from the overhead of the build system is likely somewhat negligible), but it certainly could be improved, and offering Meson as an alternative might just be that improvement. We'll see.

RiverMesa on 24 Jan 2018

❤6

The biggest issue is probably replacing the python build scripts for autogenerating code like the shaders, it doesn't seem like meson has a feature like that.

Zephilinox on 24 Jan 2018

and offering Meson as an alternative might just be that improvement. We'll see.

It should not be an alternative, either we move to it if it's clearly better, or we don't even offer it. Having to support two build systems is crazy.

reduz on 24 Jan 2018

👍17

Right, that's fair.

I'm gonna start hacking away at it over the next few days/weeks, see if this goes anywhere and whether it's an effort truly worth pursuing (even if I end up being the only person working on it that whole time).

I'll report back if I get anything presentable (like a benchmark).

RiverMesa on 24 Jan 2018

👍6

Although, not meson, I'll make a test in the coming weeks to see what it takes to build godot under bazel for windows 64 bit under msvc.

Bazel is another system whose advantage is a shorter build description and faster builds. Also it's maintained by Google so it's not going anywhere.

fire on 24 Jan 2018

👍5

Anyway, I can't verify that, but it's very likely that 99% of the computational time spend to build Godot is spent in the compiler. So even if the overhead of the building software was none, this would lead to a 1% decrease of the build time. Even a 5% decrease would not be worth, unless it allows less code and more reliability (or if I'm proven wrong :) ).

groud on 24 Jan 2018

For raw build speed of a clean clone, I doubt it'll have any significant difference. But it can be significant if you consider rebuild speeds. SCons always rebuilds a lot of stuff that weren't changed when I pull a new commit (especially OpenSSL and Bullet, which are both big libraries and take a while to build).

As I said before, SCons is slow to start (takes a few seconds before really start compiling). If you're building the whole source that's negligible, but if you only changed a line and are compiling to test, it can significantly improve the workflow (which IMO is what most matter in the buildsystem: help the people who work with the code). So 5s decreased in a 10min build is irrelevant, but 5s decreased in a 10s rebuild is a major improvement.

But I agree that any change in this regard should be tested a lot to prove it works. As @reduz stated, SCons never messed up a build, and that's something to consider.

vnen on 24 Jan 2018

👍9

Bazel is another system whose advantage is a shorter build description and faster builds. Also it's maintained by Google so it's not going anywhere.

There is even a growing Wikipedia category of "things Google discontinued". :)

mhilbrunner on 25 Jan 2018

😄11 👍6

@mhilbrunner Nope it is still maintained https://github.com/bazelbuild/bazel

Geequlim on 5 Feb 2018

👍1

Their point was that just because google made it, doesn't mean it'll be around for a long time.

Zephilinox on 5 Feb 2018

👍3

Since we have embraced C# for game development, we might as well try something like Cake. I haven't tried it myself though, so I'm interested in what are your thoughts on that one. For one thing, at least, scripts wold run faster than with Python.

rraallvv on 24 Mar 2018

👎7

Since we have embraced C# for game development, we might as well try something like Cake.

I'm not sure if requiring a Mono installation to compile Godot is a good idea, considering that Godot is mostly a C++ project and can be compiled without C# support.

The level of performance offered by Meson or CMake (with the Ninja generator) should be more than enough for Godot's purposes.

Calinou on 24 Mar 2018

@Calinou you are probably right, although it seems everybody has jumped on the C# bandwagon, personally I'm using it for game scripting with Godot and productivity apps with Xamarin.Forms, also if I really want to squeeze performance I can try something like an IL2cpp converter or jump right into C++. I'm also thinking to use the mono/csharp REPL and ASP.net for things that I'd normally use JavaScript, Python or PHP for, like for instance web development related stuff or shell scripting. The thing is that I wouldn't mind to have a build system based on C# either, since I'd be already using it for pretty much everything. At least for my it would be the end of "dammit... another language I need to learn... why don't they stick with C++ and -insert decent dynamic/managed language here-"

Edit:

To clarify, requiring the Mono Framework to be installed in order to build Godot doesn't sound as bad to me, since the so called AAA game engines require way more disk space:

Godot <50MB
Godot with Mono support (<50MB) + Mono Framework (~350MB) = ~400MB
Unreal ~20GB
Unity ~10GB
CryEngine ~8GB

Besides, we could hardcode Mono support in the editor by default and have some logic to detect whether to support C# scripting or not at runtime, and ask for the Mono framework or the appropriate export templates.

rraallvv on 24 Mar 2018

😕7 👎1

I love CMake, and meson looks cool, but I tend to agree with this article: http://www.rojtberg.net/1481/do-not-use-meson.

isaachier on 26 Mar 2018

👎12 👍7

I'm very interested in doing the footwork to move Godot to Meson. The only thing currently stopping me from contributing to Godot (and using it) is Scons. I had to use it years ago, and I've never had a more painful incremental development experience in my entire life.

The article linked above has a lot of strawman arguments, arguing that CMake already exists and we should just use that. Meson's build files are extremely easy to read, and use a syntax close to that of Python. Conversion can be quick. What I'm posting in this thread for is, if the footwork was done, would a pull request even be considered for merging?

slurps-mad-rips on 28 Apr 2018

👍4

What I'm posting in this thread for is, if the footwork was done, would a pull request even be considered for merging?

It will be considered, but it has to be proven advantageous over SCons.

vnen on 28 Apr 2018

Once you consider CMake just another build system and not the de facto standard, you end up with too many options. Why meson over SCons? Or bazel/buck are decent tools as well? But the biggest argument I have in favor of CMake is the tooling built to support that ecosystem. Clang compilation databases, support for packaging to multiple operating systems, etc. The only valid downsides I have heard is bad syntax and docs, but that isn't really strong enough for me to change my position on this.

isaachier on 29 Apr 2018

Why meson over SCons?

Meson generates ninja files, which is much faster for incremental development. Additionally, on windows, you don't need to do any sort of visual studio path manipulation. Meson works out of the box. Using meson over scons would reduce a lot of the overhead of targeting windows, especially for those wanting to use visual studio (of which, I should note, meson can generate projects just as CMake)

Or bazel/buck are decent tools as well?

Given that bazel and buck are both written in Java (a dependency godot has only if you'll be targeting android, and even then one could technically use just the NDK), I would argue they're right out as a primary dependency for all users. Python is already in the dependency list, and you can easily install meson (and ninja) through a quick pip install meson

Clang compilation databases

Meson supports this by default, as this is a default feature built into ninja

support for packaging to multiple operating systems

Meson also has support for this, and relies heavily on pkg-config where appropriate

The only valid downsides I have heard is bad syntax and docs

Meson has a very python-like syntax, relying on variables, objects, and builtin functions. In my opinion, writing

my_meson_list = ['x', 'y', 'z']
message(my_meson_list)

vs

set(MY_CMAKE_LIST "x;y;z")
message(STATUS ${MY_CMAKE_LIST})

is a lot easier, especially given how cmake is sometimes case sensitive, and other times not. When you start to get into generator expressions, it can get extremely hairy.

# In one file
set(USE_STD_CXX11 "-std=c++11")
set(USE_STDLIB_LIBCXX "-stdlib=libc++")

# Possibly elsewhere
set(LIBCXX $<BOOL:${CAN_USE_STDLIB_LIBCXX}>,$<BOOL:${BUILD_WITH_LIBCXX}>)
set(NO_RTTI $<BOOL:${CAN_USE_NO_RTTI}>,$<BOOL:${DISABLE_RTTI}>)

# Later on...
target_compile_options(my-exe
  PUBLIC
  $<$<BOOL:${CAN_USE_STD_CXX11}>:${USE_STD_CXX11}>
  $<$<AND:${LIBCXX}>:${USE_STDLIB_LIBCXX}>
  $<$<AND:${NO_RTTI}>:${USE_NO_RTTI}>
  # Oh yeah, you're gonna want more of these, because you can't trust compiler interfaces
)

# Generator expressions mean you can't follow the flow of the build to see what is called and defined where. This is a completely valid use of CMake.
check_cxx_compiler_flag(${USE_STDLIB_LIBCXX} CAN_USE_STDLIB_LIBCXX)
check_cxx_compiler_flag(${USE_NO_RTTI} CAN_USE_NO_RTTI)

The equivalent meson looks like:

cxx = meson.get_compiler('cpp')
# This is an extremely simplified approach. One can do a different option when dealing with MSVC and gcc support.
args = compiler.get_supported_arguments('-std=c++11', '-stdlib=libc++')
add_project_arguments(args)

I think I should note, by the way, that of all the C++ build systems out there, Meson is the least terrible. Arguments against its use tend to be a picture of a web comic, some hand waving about how CMake is the most used, so you should just deal with it, and "hey at least CMake isn't autotools".

But above all (and I think this is a really important thing, especially for those using godot as for gdnative), Meson supports precompiled headers natively. CMake does not (and using cotire to hack the build can cause more problems than its worth).

I cannot tell you how much this could speed up the build for gdnative users, but it will be a significant boost on gcc and msvc (at the very least, a performance increase on msvc is inevitable). Add in the use of extern template for the very little amounts of templated code found in godot, and you could see decent build time improvements, which are important for iterative and incremental builds.

slurps-mad-rips on 29 Apr 2018

👍10

Meson generates ninja files, which is much faster for incremental development.

Note that CMake can also generate Ninja build files instead of traditional Makefiles; this is done by passing -G Ninja to its command line. It works quite well on most projects I've tried and is slightly faster overall (plus, it'll use all CPU threads by default).

Meson still wins in the raw performance comparisons I've seen, but this definitely helps close the gap between Meson and CMake performance.

Calinou on 29 Apr 2018

Yes, however, CMake does not automatically install ninja for you, and it is not the default generator. Most of CMake's time is spent configuring the project, in my experience, and while recent versions of CMake have vastly improved performance, not everyone will have the latest CMake.

slurps-mad-rips on 29 Apr 2018

OK that is quite thorough! I'm glad you have a good idea of the features you value in a build system. I still think there ought to be a standard build system in the C++ world that projects can rely on. I just wish meson were a CMake transpiler as opposed to its own thing.

isaachier on 29 Apr 2018

I still think there ought to be a standard build system in the C++ world that projects can rely on.

Trust me, no one wants this more than me. I've become sort of a build system trash goblin and even gave a talk on them at CppCon. I unfortunately (or fortunately?) care a lot about build systems, almost to the point of obsession. :)

slurps-mad-rips on 29 Apr 2018

I just want to say this, but every time I've did a long build of Godot on Windows, I've noted that Python consistently took >15% of my CPU. I doubt it's only slow to start.

PJB3005 on 29 Apr 2018

👍1

Who's going to make a small prototype godot engine built with meson? I never got to the point of creating a small test for bazel.

A 2d only build of godot should be sufficient to test with all the options disabled.

fire on 29 Apr 2018

The first roadblock I faced was core/make_binders.py. This is the autogenerated code that godot uses.

fire on 29 Apr 2018

I made an initial prototype in the build system Bazel but I'll discuss the setup on another issue. https://github.com/godotengine/godot/issues/18518

Don't have time over the week, but others can expand on the Bazel BUILD.

fire on 30 Apr 2018

Lol how did bazel get involved?

isaachier on 30 Apr 2018

Since there might be a one time opportunity to switch to a different build system, shouldn't we looking for the best possible options? I for one wold be interested in seeing a Cake vs Bazel vs Scons comparison chart, although I have to admit I have zero experience with Cake to do the footwork for those tests.

rraallvv on 30 Apr 2018

@rraallvv do you mean Cake or CMake?

isaachier on 30 Apr 2018

@isaachier My bad, I was talking about www.cakebuild.net, I should have been more specific, sorry about that.

rraallvv on 30 Apr 2018

Isn't that just for C#?

isaachier on 30 Apr 2018

@isaachier I really don't know, all this time I was assuming Cake was capable of that, just like SCons being written in Python and having support for C++. Nevertheless, I just asked at their gitter chat, I'll re-post here the answer.

rraallvv on 30 Apr 2018

@rraallvv for the record I think the idea of a side-by-side comparison of major build systems is a great idea. Here is what I found online that I think are fairly unbiased (not written by the build system developers): https://carlosvin.github.io/posts/choosing-modern-cpp-stack, https://www.reddit.com/r/cpp/comments/6euc7b/build_systems_bazel_buck/die6g1y/, https://stackoverflow.com/a/12022652/1930331.

My personal opinion:

CMake: Pretty ugly language, but ubiquitous and practically the industry standard.
SCons: Slow, old, and IDK status. Uses Python.
meson: New, cool, Python-based build system that is not incredibly slow. Main disadvantage is trajectory. Unclear if this will just be another flavor of the week build system.
Autotools: Ancient, no tooling support (i.e. compilation databases for Clang autocomplete support or static analysis). No Windows support. Avoid at all costs.
bazel: A Google package-manager/build-system rolled into one. Doesn't add much over CMake with a plugin such as Hunter. Also, Google software has a tendency to focus on what Google needs, not the public at large. Too new to gauge long-term trajectory.

I try pushing CMake whenever I get a chance because I want a de facto standard for C++ and CMake has the biggest "market-share" right now.

isaachier on 30 Apr 2018

@isaachier Thanks for sharing that information, I'm pushing C# since there isn't a build system that supports scripting in C++, it might sound silly but I don't know why someone hasn't done that yet, for instance this C++ REPL could be used for a build system if one wants to go to the next level and end the debate on which one is better, faster, etc.

rraallvv on 30 Apr 2018

I don't understand. C++ would probably be an awful build language. Slowness has more to do with the internal build system core AFAIK not with the details of the scripting language used.

isaachier on 30 Apr 2018

👍2

@isaachier My guess is that SCons is so slow because it was written in Python, but if someone were to take the task of porting the entire build system to C++ and also make it JIT compile C++ scripts to run those builds, it would be much faster. Maybe the same could be true for every build system out there.

rraallvv on 30 Apr 2018

It's a nice idea but meson is not considered slow by most accounts, yet it is written in Python.

isaachier on 30 Apr 2018

👍2

I doubt meson will be a flavor of the week as the GNOME project is currently abandoning autotools in favor of meson.

Additionally, Scons is slow not because it is written in python, but because it also executes the build. It is a direct build system, like make or ninja, but also uses massive amounts of XML for definitions and a large amount of its code is actually just XSLT conversions.

Meson, on the other hand, executes its build scripts to generate a dependency tree, and then dumps this out to a ninja file. Meson is fast, because ninja is fast. Additionally, not everything is a string in Meson, like in CMake.

Lastly, I should note, there are several in development build systems where you will be able to script in C++ (I"m actually writing one of them). However, asking godot to move to a build system that doesn't exist yet (and worse, that I myself am writing) is a bit arrogant, in my not so humble opinion ;)

I also want to reiterate, under the current development workflow python is a necessity. So, too, is meson. And that makes moving to meson as simple as a call to python's pip tool.

slurps-mad-rips on 30 Apr 2018

I completely agree @reduz that I dont really find SCons to be slow either.

As @slurps-mad-rips said:

Scons is slow not because it is written in python, but because it also executes the build

man scons:

scons supports building multiple targets in parallel via a -j option that takes, as its argument, the number of simultaneous tasks that may be spawned:
scons -j 4

I have a CPU with 8c16t, so use "scons p=x11 -j 18" is very fast for me.
It's 10x faster than use "scons p=x11" as the default setting "-j 1".
Do someone have a single core CPU at year 2018?
Let your CPU working fully.
please give it a try.

akanewf on 30 Apr 2018

Not everyone has these large CPUs, but more importantly, meson (well, ninja) is much faster at incremental recompilations. Scons, by default, performs an MD5 hash sum to see if a file has changed. Godot uses the MD5-timestamp setting, which means it will only hash a file if a timestamp has changed. It's still ridiculous.

slurps-mad-rips on 30 Apr 2018

I guess the best way to prove what is better is to write a new one and see :)

isaachier on 30 Apr 2018

👍1

Scons isn't fast for many people, especially windows users running into problem with anything above -j 1.

Zireael07 on 30 Apr 2018

👍5

@slurps-mad-rips it depends on the project. ninja seems to hype its speed a little too much. See this article: http://david.rothlis.net/ninja-benchmark/. Seems counting source files is your best bet here. If <20K won't make much of a difference.

isaachier on 30 Apr 2018

@isaachier that benchmark is between make and ninja, not ninja and scons, so you'll have to forgive me if I don't want to pay attention to it. ;)

slurps-mad-rips on 30 Apr 2018

I guess the best way to prove what is better is to write a new on and see :)

I agree, there is no point in bike shedding. If you have an opinion on a better build system, implement it and prove it's better. Otherwise this is a discussion of opinion and will lead nowhere. @fire has already done a partial test with bazel: https://github.com/godotengine/godot/issues/18518.

rminderhoud on 30 Apr 2018

@slurps-mad-rips ya totally different topic, not SCons. I just meant make vs. ninja isn't crazy difference.

isaachier on 30 Apr 2018

Slight update: I'm giving up on trying to move to meson. It's been piecemeal and while the build files are slightly easier to read, staying up to date with the current system is where most of my (now rare and in short supply) free time is spent, instead of moving scons to meson.

Right now, a lot of code and operations are placed directly in the scons scripts and this makes moving to another build system more difficult. If anything I'd argue time should be spent decoupling the existing python scripts that generate code from scons, as this would make a move to another build system like meson, cmake, or even bazel a hell of a lot easier.

slurps-mad-rips on 28 Jul 2018

Good thing a patch was just merged to move all the code generation out. https://github.com/godotengine/godot/pull/17595 Don't give up!

fire on 28 Jul 2018

👍5

SCons Project Co-Manager here. A couple notes:
1 - SCons is under active development and since our migration to Github we're seeing an increased rate of pull requests.
2 - We've identified a few specific parts of SCons which are responsible for about 50% of the null incremental build time and are actively working on addressing that. (The Subst functionality)
3 - We've started supporting Python 3.5+ (as well as 2.7.x) with SCons 3.0
4 - Let me suggest that instead of attempting to port to N different build systems, contributing some time to help improve SCons might be a more effective use of developer time
5 - We identified and addressed via profiling some of the low hanging fruit in improving performance which was released in 3.0.0 and 3.0.1. If you're using an earlier version the null incremental build can be 5-15% slower. (assuming Python 2.7.x)

As always the SCons project is ready and willing to help projects who use SCons with issues they run into.

bdbaddog on 28 Jul 2018

Some "stats" for SCons & godot.

I used (and still do) a Q6600 which is a 4 core cpu from 2008 or so with 2.4GHZ (and a lower instructions per cycle count) than the processors we are "all" using today (i7s).

On Q6600, a built took between 3 to 5 minutes (or more, but exact num not that important), out of which ~35-40 seconds was spent by SCons parsing the tree of dependencies and stuff (the prep work)... so, 80% of the time spent running cl.exe (MSVC compiler) process over and over again.

So, performance & optimisation wise, unless SCons (python really, not SCons) has a terrible process calling system that is taking more time than it should or some other tools is not compiling one file per process (like SCons currently is) and the 1 process per one file overhead is significant (I'm not an expert on cl.exe compliation internals & startup overheads), the only performance optimisation a faster build system would produce is probably in the building of dependency tree phase, and other prepwork for complication.

So, we are optimizing those 35-40 seconds out of a 5min build (on i7's now we are probably optimising a 20 second single core setup step out of a 3 min multi core build).

So, to recap, if i remember correctly, 3 to 5 min is actually a 4 core multicore build, while a single core build takes around 10? I need to rerun my tests...

So, "optimizing" SCons away means more or less optimizing those first 35 seconds of the builds start...

That's as far as "optimizations" are concerned when it comes to SCons vs other faster build system... It's possible that the actual compilation could be optimised a bit by avoiding calling a single cl.exe for every obj file, but that's only if the process overhead is significant (I fear HDD will always be the actual IO bottleneck there (and not the CPU), which is not avoidable/mitigatable without an SSD)

All of these paragraphs above consider a "full build". That said, SCons "cheats" if everything is already built and passes it's heavy handed consistency checks so if you change a single file it will actually compile a "single file" and then link everything back in, making a "full build" into somewhat of an incremental build). This however requires those 35-40 seconds of full dependency tree precomputation...

So, an "incremental build" is around 45 seconds on my old computer, and I thought when I was a noob 2 years ago that it would be CPU bound, but actually, it might be IO bound (checking several thousand files on the hdd) for a full dependency sweep/tree rebuild... So, if it IS IO bound, no "faster buidlsystem" is going to solve that one...

My hackish idea from 2 years ago was to try to attach a filesystem "watcher" to SCons, have SCons run & recompile a single changed file (and it's dependencies, that's automatic), and then just have it link everything back again... although I realise today that linking would still probably run a full dependency sweep... that said, the dependency tree sweep/rebuilding can be killed with an option and then SCons will use it's cached dep tree... which saves around 15-20 seconds of the 35 sec precomputation step (the first 15 seconds seem unavoidable) although that's might not guarantee a "perfect" build always like SCons does (though that might not be important for incremental builds in dev... if you decide that speed is a worthy tradeoff).

I now have enough knowledge that I could probably hack it through npm and npm watch... and or other system that I'm not currently familiar with... But that's for later, and when I have time (not currently).

This is getting a bit long so I'll end it here, but the point of these paragraphs was to give you some "statistical" information rather then to discourage anyone (go ahead if you like working on build systems), like I myself liked fixing some stuff in Godot's SCons. I hope some information was useful for you guys here.

If you wish to replicate/redo my statistics, just browse through SCons docs till you that turn on statistical info (timers stuff) and/or find a way to turn off dependency redolution (those first 20 to 30-35 seconds of "precomputation")...

Or for a slightly less manual solution, timer debugging info should already be present (edit: it's prob not, somebody did what looks like a sweet update to the vs project generation, this paragraph might be stale info) in the automated VS project generated by SCons (at the end of SConstruct file)... you can find info on how to generate a VS Project in the "Windows compilation tutorial" on the godot website. You can then run SCons from VS, I believe the generated project should still probably work, though I have not tested it in Godot 3... It's an option, but tailing scons to a > log.txt is also another option (etc)...

Here's hoping somebody found this info helpful.

Griefchief on 28 Jul 2018

@Griefchief Just to clear things up a bit, I think we aren't trying to optimize full-build times, since those are dominated by compiler time anyway. Instead, we want to optimize incremental builds where only a single source file is changed.

bojidar-bg on 28 Jul 2018

Indeed the "null incremental build" (nothing actually needs to be rebuilt, but scons will still need to process the whole dependency tree to ensure that) is the case we're working on.

Some profiling has pointed out some inefficiencies in processing the command lines (which have to be done for every target to ensure that they have not changed) have been identified. The bulk of that is the Subst() logic. This is some tricky code and thus far has been implemented in a simpler and safe, but not terrible efficiently way.

Another hot spot of time is the reading and writing of the sconsign files, currently implemented with cpickle and always read and written as a whole. We have some ideas on how to either speed up and/or make more incremental. Anyway watch future releases for improvements.

On a side note, you might try MSVC_BATCH to see if that speeds up your windows builds:

MSVC_BATCH
When set to any true value, specifies that SCons should batch compilation of object files when calling the Microsoft Visual C/C++ compiler. All compilations of source files from the same source directory that generate target files in a same output directory and were configured in SCons using the same construction environment will be built in a single call to the compiler. Only source files that have changed since their object files were built will be passed to each compiler invocation (via the $CHANGED_SOURCES construction variable). Any compilations where the object (target) file base name (minus the .obj) does not match the source file base name will be compiled separately.

bdbaddog on 29 Jul 2018

@bojidar-bg yes, yes, I was actually talking about incremental builds as well, thank you for the tip :D

@bdbaddog Hi baddog, just for your info, I actually wrote this big comment of mine before I saw your comment, so, it doesn't actually refer to anything in you comment specifically... but you did manage mention some of the "concerns" I had (information for other people really) while we were both writing our comments at the same time...

And I didn't want to ping you in it cause I didn't wanna "waste" your time :D, but I guess I will if spend some more time on godot build/scons in the future (I do plan to, i'm just very busy))

Thank you for your input here!

Griefchief on 29 Jul 2018

@bdbaddog BD, since you're providing support, if I may ask here:

If I attach npm watch to the filesystem (godot's source) and know which exact file has been changed, would that bring me any benefits with SCons?

Now I can run scons through npm watch and push the file to it...

scons [godot options] the/file/that/was/changed.

this will obviously recompile that file, probably avoiding Godot's whole dep tree? correct? So now dep tree traversal won't take 20 sec, but less then 1 prob, but I get 1 updated .obj file (or more, depends on what you change). It won't actually go through the whole godot dep tree for that one file? I do have an advantage in less dep tree traversal in this example (and possibly other advantages in the precompute phase that can not be gained through reusing the dep tree cache, like less files overall to go through?)?

Now, if I do have an advantage, and if I wanted to just link everything again, would I need a full tree traversal for a "simple" linking phase?

I understand this usecase might not be currently supported by Scons (just link everything again), but I'm asking both from a practical and theoretical standpoint? I realize that cached dep tree could be reused with that "don't do dep tree" option (I forgot it, it's been two years), and that would work for some usecases, and that usecase is no new files added in, correct? (npm watch could alert me to new file addition and I could then do a full dep tree refresh if I detect that... in other words, kind of push the "ensure dep tree consistency" back to a filewatcher like npm watch, automate typing in that "use dependency cache" option done manually by the user to a program like npm watch... let the program worry about consistency, not the user, and do that in "real time", the moment the user has saved a file, giving lots more time for scons to do it's thing)

Is there something in my idea that would not work as an optimisation(practically and theoretically), and do you have other suggestions (that do not have to guarantee a perfect build if something like npm watch can take care of stuff, it's fine as well)? Can you advise please? Thank you!

Griefchief on 29 Jul 2018

To recap, the usecase is similar to this:

1) run a filewatcher like npm watch on godot's source directory
2) user saves/modifies an old file
3) scons is immediately run for that file by npm.
4) if compilation is successful, npm instructs scons to run the linker and link the executable.

4.1) if dep tree is cashed (a full build was already run), npm can instruct the linker to just link at this point with a cached version of dep tree
4.1) if no cached dep tree is detected, npm watch runs a full godot build
4.2) if cache invalidation was detected, like adding a new file, perform a full dep tree traversal (npm won't use the "use cached dep tree" option with scons).

I hope this makes my idea a bit easier to understand, please let me know what's wrong with it at this point ;D

P.S. I believe this is the command that shaves of 15 seconds off of the 35 null incremental build, with it's "obvious" consequences... I basically want to automate what is described here (more or less, + let scons know which exact file was modified if that helps it in any way (or it could help it)):

https://www.scons.org/doc/latest/HTML/scons-user/ch06s04.html

Griefchief on 29 Jul 2018

So a few things here (in no particular order, but I'm using numbers because yolo

1) Incremental null builds are basically nothing under any tool that generates ninja files. Chrome has, AFAIK, a less than 2 second null incremental build. This is of important note, as a 35 second drop to 15 under an npm watch is still very large compared to a meson or cmake based Ninja build. Ninja also does the same thing that Scons does in that it does perform a rebuild in the event that the command line changes (make, of course, does not do this)

2) CMake recently implemented the ability to use CONFIGURE_DEPENDS as an argument to the glob system. This has always been recommended against because tools like xcodebuild and msbuild have historically (and presently) not supported directory change detection. This is, afaik, one thing that Scons could improve upon, but I have neither the time nor patience to deep dive into the project to implement this. Basically, every OS updates a directory when its contents change, be it an added file, modified file, or removed file. When making incremental changes, one can simply check which directories have changed and glob only those on a reconfigure check. This can reduce work done as you're not globbing the entire tree, but smaller bits. For small projects this is fine. Whether this would benefit godot or not would require some testing and that's a lot of work to check if it would even help.

3) While it's nice to see that Scons is receiving more attention, I personally think it is better for the community at large for godot to move to a more used build system. I cannot think outside of godot of any large project that uses Scons. GNOME/GTK recently moved to meson (skipping CMake), KDE has been on CMake for quite some time. While it's anecdotal evidence (and thus a basic confirmation bias), I know of several C++ developers who would like to try godot but would rather not have to touch scons ever again. Removing the nasty experiences and thus the sour taste left in developers mouths is difficult, and I do honestly wish the Scons project the best. But I do know that the C++ community at large already hates long compile time. Long incremental null builds (or even long incremental builds of 1 file) are a nightmare in the modern world.

I do intend to try again, but I might target CMake for this. Moving from CMake to meson is a hell of a lot easier thanks to meson having a conversion script that can do a lot of work. CMake might make handling dependencies a bit easier with the new FetchContent module. We'll see how it works out. I will say, I'm glad these codegen scripts were moved to separate files. Using CMake will be nice if only because moving to a different C++ standard is as simple as target_compile_features(<target> (PUBLIC|PRIVATE|INTERFACE) cxx_std_<number>)

Obviously any build system related stuff is going to be contentious, but trying can't hurt.

To be quite honest, (and this might come off as callous) I have no desire to deep dive into the Scons project to improve it. Maybe after 2020 when Python 2.7 is officially finally really dead and the project can move forward to Python 3.5+ only it'll be worth investigating or improving, possibly with async operations. Until then, however, I'd rather not have to touch it.

slurps-mad-rips on 31 Jul 2018

Maybe after 2020 when Python 2.7 is officially finally really dead and the project can move forward to Python 3.5+ only it'll be worth investigating or improving

Scons 3 does support Python 3.5+ : https://scons.org/tag/releases.html .

I don't really understand how one can dislike Scons so much and like the configuration mess that is CMake, where it's even hard to define your own build settings if you are a user, hard to understand what are the options and their types and what are internal defines, it only brought pain to me, and I was actually relieved that Godot didn't use it.

In any case, if you feel like it's going to be a better solution go on and try, I might end up changing my view on CMake.

Faless on 31 Jul 2018

where it's even hard to define your own build settings if you are a user,

If that is asked then CMake is being used wrong. All toolchain settings should be in a toolchain file and any project specific configurations are passed via flags (or gui or edit the cache file or whatever). Everything is documented and typed, even the project authors can well document and type their exposed configuration. You should read CGold and the modern CMake standards.

OvermindDL1 on 31 Jul 2018

❤1

@Faless no one denies the CMake language is terrible. But the implementation is best in class. I've been considering the possibility of writing a transpiler from a better language to alleviate the issue.

isaachier on 31 Jul 2018

@OvermindDL1 love the suggestion to read CGold.

isaachier on 31 Jul 2018

If that is asked then CMake is being used wrong

Well I guess that all the programs I encountered where doing it wrong then.

To be honest, I hate having to change flags via -DI_LOVE_CMAKE_AND_MAKE_DEFINES_LONG=on (really? on?). Not to mention the idea of editing the cache file O_o . Or that, if I want to change a flag, I basically end up having to delete the build folder and start from scratch (because, you know, cache differs!)... so much for incremental builds...

EDIT: Also, one thing I really liked about Scons (in Godot at least, and this might possible with other build system of course), is that I didn't actually read any standard or doc, I just run scons -h in the Godot folder

Faless on 31 Jul 2018

95% of build files I have seen on GitHub are wrong. People get lazy about build systems and follow cargo cult practices to cobble something together.

isaachier on 31 Jul 2018

@isaachier , just as a metric, could you provide 2 examples of a bad and good CMake file? How would you consider this: https://github.com/ARMmbed/mbedtls/blob/development/CMakeLists.txt ?

Faless on 31 Jul 2018

Better than most I've seen. Still, if you are not using CMake >= 3.0, then your experience with is not with modern CMake.

isaachier on 31 Jul 2018

Well I guess that all the programs I encountered where doing it wrong then.

I tend to see it done well nowadays.

To be honest, I hate having to change flags via -DI_LOVE_CMAKE_AND_MAKE_DEFINES_LONG=on (really? on?). Not to mention the idea of editing the cache file O_o . Or that, if I want to change a flag, I basically end up having to delete the build folder and start from scratch (because, you know, cache differs!)... so much for incremental builds...

You don't have to use on, you can use true, 1, and a variety of other things. It is to make it so you can choose an appropriate boolean name that flows well with flag name.

And you should not ever need to rebuild the build folder if you change options, cmake is exceptionally good at rebuilding when necessary.

Still though, everyone touching C/C++ build systems should read CGold as well as the official manual of CMake. It is really easy to use CMake right, it is often a great deal shorter as well. The ancient CMake2 methodologies should not be being used. Really though, anyone that depends on a CMake 2.x version is absolutely doing things wrong (not using targets and toolchains properly, not setting options properly, etc... etc...).

OvermindDL1 on 31 Jul 2018

👍1

Thanks for the explanations, while I remain sceptical, I'm looking forward to see how it would work for Godot :).

Faless on 31 Jul 2018

@slurps-mad-rips Your post with the supposed CMake example code for enabling C++11 mode is entirely non-standard CMake for a variety of reasons, including but not limited to:

Setting unexposed variables.
Toolchain arguments set in a non-toolchain file.
Absolutely not how you do something like enable C++11 mode.
Inefficient.

Given the shown self-described-as-simplified meson code in that post of:

cxx = meson.get_compiler('cpp')
# This is an extremely simplified approach. One can do a different option when dealing with MSVC and gcc support.
args = compiler.get_supported_arguments('-std=c++11', '-stdlib=libc++')
add_project_arguments(args)

The equivalant simplified (in that you can add more properties as well) CMake would be:

set_target_properties(godot PROPERTIES CXX_STANDARD 11)

And that will make sure that the toolchain used knows that it needs to set that mode (and you can specify how to handle failures if using a compiler that doesn't support it as well). Assuming special compiler-specific command line flags in the build file is extremely poor form as those should only exist in toolchain files and the fact the meson example shown in the prior post actually specifies specific arguments essentially means you are hardcoding the supported compilers.

Considering there is a pre-built CMake Toolchain file for just about any toolchain you can think of (including all the standards of MSVC of various varieties, GCC/Clang, android, iOS, emscripten, etc... etc...) you should not be re-inventing the wheel. If the user that is compiling something like godot wants to use their own custom compiler with unique commandline arguments (like take Intel's compiler or maybe to some dedicated chip or just a simple RPi) then the build script shouldn't need to care about it or need any changes at all (the code might of course, but optimally it should not ever either as CMake can tell it what all is supported via definitions) and the user can supply their own custom toolchain file for the build to use.

EDIT: And never discount the widespread support. I can't find anything about meson being able to generate project files for KDevelop (the IDE I use, which interestingly uses CMake as it's build format, I occasionally use CLion at work, it also uses CMake), where almost every IDE out (even Visual Studio!) can open CMake projects natively now (in addition to CMake being able to generate projects for most IDE's directly if you so wish).

OvermindDL1 on 31 Jul 2018

@OvermindDL1

It is not non-standard CMake in the absolute slightest. The code sample targets CMake 3.0, a version in which CXX_STANDARD does not exist. You can easily see from here that you use target_compile_features with cxx_std_ and the number of the standard. This is the new and modern cmake way to set the standard version. set_target_properties is an older per-target approach.
In fact, the exact code taken from CMake's documentation is

target_compile_features(mylib PUBLIC cxx_std_11)

However, CXX_STANDARD does not set libc++ support for clang as clang uses libstdc++ by default on linux unless configured (and it's never configured to use libc++ by default on most linux distributions, as this would cause linker errors and ABI issues). The only way to do it at this time is to check if the compiler supports -stdlib=libc++ and pass it to the compiler via a generator expression. There is nothing involving unexposed variables regarding this.

Furthermore, I do not appreciate you dropping a comment to make corrections from a comment I made months ago, especially given that I will be targeting CMake over Meson in my next attempt. It lacks tact, and in my opinion adds nothing to the conversation.

slurps-mad-rips on 31 Jul 2018

CMake 3.0 is old enough to no longer be supported (June 10th, 2014). No one should be running anything less than CMake 3.10 currently, though staying up to date with CMake 3.12 is preferred at the current moment.

And yes, the example you gave is entirely recommended against (I think even CGold mentions not doing that).

And yes, I did mean target_compile_features. ^.^;

However, CXX_STANDARD does not set libc++ support for clang as clang uses libstdc++ by default on linux unless configured (and it's never configured to use libc++ by default on most linux distributions, as this would cause linker errors and ABI issues). The only way to do it at this time is to check if the compiler supports -stdlib=libc++ and pass it to the compiler via a generator expression. There is nothing involving unexposed variables regarding this.

The current toolchains all do that for clang just fine, at least with C++14 as that is what I compile with (and a couple C++17's).

Furthermore, I do not appreciate you dropping a comment to make corrections from a comment I made months ago, especially given that I will be targeting CMake over Meson in my next attempt. It lacks tact, and in my opinion adds nothing to the conversation.

I apologize, did not notice the date as I was reading the backlog (and there is a lot of backlog here). I just noticed something that appeared woefully wrong for any kind of modern suggestions from what I've seen for a few years now and did not see any corrections in posts shortly after thus wanted the incorrect information to not propagate. :-)

The only way to do it at this time is to check if the compiler supports -stdlib=libc++ and pass it to the compiler via a generator expression.

However yes, that is the job of the toolchain file, that should absolutely not ever appear in a build file, ever.

There is nothing involving unexposed variables regarding this.

I was just referencing the variables that are set then used elsewhere in only single locations, that is recommended against in CGold among others for a variety of reasons.

OvermindDL1 on 31 Jul 2018

CMake 3.0 is old enough to no longer be supported (June 10th, 2014). No one should be running anything less than CMake 3.10 currently, though staying up to date with CMake 3.12 is preferred at the current moment.

They shouldn't, and yet some places are. For a game engine that tries to stay up to date and with the latest technology, staying on latest CMake is fine. However I am more than well aware of extremely active projects stuck on older versions such as 3.5 and in some cases 3.4. In fact, some well known C libraries still target CMake 2.8 (See: SDL2, libssh2, libgit2, etc.) It is a huge pain in the ass to be sure.

However yes, that is the job of the toolchain file, that should absolutely not ever appear in a build file, ever.

Making this assumption (that toolchain files are well written and cover all possible systems and all versions of CMake) is asking for trouble in the current general C++ ecosystem. When using clang, the option of compiling for libstdc++ or libc++ (especially when used via a FetchContent call) needs to be provided by the library. CheckIncludeCXX and CheckCXXCompileFlag are still necessary for even some flags that don't exist under GCC but do exist under Clang (and vice versa). The primary issue here is that compiler interfaces have diverged a large amount and instead of vendors spending time to facilitate compatibility options or even discussing a common interface, that behavior is thrust onto us, the developer when all we want to do is write code. Trust me, no one on this planet is more upset about the state of C and C++ build systems and dependency management than me, but pragmatism for existing libraries on CMake is needed. The nice thing about moving from a build system to CMake is that we can start out the gate with the latest and greatest features. The problem becomes stagnation of the build system because no one wants to touch it (except for that brave soul who kicked the code generation build scripts into separate scripts), but some work would be needed to make sure it doesn't get out of hand (which it will because... I mean it's a build system. They always do)

slurps-mad-rips on 1 Aug 2018

I must be the one guy who actually enjoys writing build scripts especially in CMake ;).

isaachier on 1 Aug 2018

Making this assumption (that toolchain files are well written and cover all possible systems and all versions of CMake) is asking for trouble in the current general C++ ecosystem. When using clang, the option of compiling for libstdc++ or libc++ (especially when used via a FetchContent call) needs to be provided by the library.

Such things should be properties passed to the toolchain that the toolchain can then act on.

As well as if you have very unique setup it is entirely reasonable to include toolchain files, like say for a very custom android build or so (though the extended polly set of cmake toolchains has covered almost everything I've needed thus far).

The problem becomes stagnation of the build system because no one wants to touch it (except for that brave soul who kicked the code generation build scripts into separate scripts), but some work would be needed to make sure it doesn't get out of hand (which it will because... I mean it's a build system. They always do)

This is actually a great point for choosing CMake now, it has a well defined set of standards to follow that work really well now after dealing with more than a decade of prior experience, both good and bad.

I must be the one guy who actually enjoys writing build scripts especially in CMake ;).

I enjoy it in comparison to alternatives for sure, though I've written in it long enough that it makes sense to me, though that definitely doesn't mean it would for everyone. ^.^;

OvermindDL1 on 1 Aug 2018

I've used a lot of different build systems over the years. The one that made most sense for me recently is CMake, but I can definately see the appeal of SCons and how flexible it is.

Even if SCons isn't perfect (what build system is?) it may be good enough.

Does it provide the required features?
How much overhead does it create?

If SCons scores "good enough" for both points then keep it.

spongeboburu on 3 Aug 2018

Has any of you proposing the replacement of SCons by something else actually taken the effort to learn how Godot build system works?

As a hint, it's incredibly complex and I don't think there are many (or any) projects using CMake building for this many amount of platforms and allowing to have several builds targets configured/compiled at the same time the way we do.

All efforts by those who said "I will port Godot build system to somehting else" failed miserably so far when realizing the complexity of what SCons is taking care of.

To me, it's hands down the best tool for the job by far. Nothing is even close to it. If you guys complain about base compilation time (which is just 4/5 seconds on a medium to high end system with SSD), you really should first try to understand everything Godot does when building and see how it would perform in CMake or anything else.

reduz on 3 Aug 2018

👎2

I did take a look. I am not convinced it is undoable in CMake or elsewhere. But I'm also lazy ;). We'll see what happens.

isaachier on 3 Aug 2018

As someone who did the bazel port. I was able to get to the point where the Godot editor started without icons.

I'm pretty sure CMake can reach that point.

PS. The closest competitor project with large amounts of platforms is Urho3d.

Github Urho3d

fire on 4 Aug 2018

I reproduced my bazel result on cmake.

https://github.com/fire/godot/tree/cmake

Assume visual studio 2017 is installed.

git clone https://github.com/fire/godot.git -b cmake
scons p=windows
Modify platform/register_platform_apis.gen.cpp
#include "register_platform_apis.h"

void register_platform_apis() {
}

void unregister_platform_apis() {
}

Install cmake

choco install cmake ninja -y
# Open visual studio command prompt amd 64 2017 native
# Go to godot source directory
cd ..
mkdir build
cd build
cmake ../godot -GNinja
ninja

Please play with it. The editor has the same problems as the bazel build (no icons), but you may find this sample helpful.

godot_2018-08-03_21-44-57

Note

Icons work as of 71175b45f819e7cc5e4368dbf3e42abdd19af542
Visual Script and GDscript work

fire on 4 Aug 2018

👍4

@fire Thanks for this. I see a lot of places for improvement that would make maintenance of these CMake files easier (and time spent adding new features or compiler flags). I might tackle this next weekend however, schedule pending.

slurps-mad-rips on 4 Aug 2018

Awesome work @fire. Glad to see someone here can produce something in more than just one build system :smile:.

isaachier on 5 Aug 2018

Slight update. I had some downtime today to tackle building off of @fire's work. Unfortunately, the code generation scripts are still a little too embedded with the current Scons system and some work is needed to fully extricate them (this is of course assuming I'm reading the code correctly). That said, it'll probably be a separate issue + pull request. I did identify a few areas where an external template file (or series of template files) could be used. Python has support for template strings and this might make life a bit easier, instead of having to process strings, add them to a list, and then process more strings, and add them to a list, etc. etc.

Ideally, the code generation scripts should be able to be run as if they were executables without knowledge of the build. As long as the proper files were passed to them, we'd be good to go for fully porting to any other build system.

slurps-mad-rips on 6 Aug 2018

@slurps-mad-rips Extricating that from your SCons system is a good idea even in SCons. You can hang up the GIL by having too much logic in process and hurt your build performance. Occasionally it's lead to some file close/open race conditions when building in parallel (though often resolved by opening the file with in a context).

bdbaddog on 6 Aug 2018

Hello all,

I'm going to use this space to keep people updated on the port to CMake. If requested, I'll make a separate issue and keep track of changes there, though this is a nice area given it's got a massive backlog of discussion and I think it'd be a waste to lose so much context.

I digress. I'm going to have to take a break from the CMake port for the next week while I attend a conference, however you can see the efforts of my initial porting attempt here. Please keep in mind that I'm going to be shooting to stay 1:1 with the current master branch, and as such I'll be force pushing changes as they come in so I can keep a rebase. As I approach stability, I'll file a pull request to keep rebases easier to work with, as well as to make it easier to comment on changes or ask questions. I think this port to CMake is not only doable, but at the very least it'll make one more person (me) well versed in how Godot is actually built and constructed and I might be able to put that into words so others can get a bird's eye view of the build system. This port also might possibly make everyone's workflow easier in the long run, especially with releases. I'm not gonna lie, having the CONFIGURE_DEPENDS globbing of files mixed with Ninja is actually quite nice.

One last thing, I highly suggest you all take a peak at the helper CMake Modules I've written just because if you are writing other CMake project's, these modules might come in handy.

slurps-mad-rips on 13 Aug 2018

👍4 ❤2 🎉1

Were you able to update to the latest 3.1 alpha? The alpha could be a good target. Currently trying the build.

fire on 6 Sep 2018

This issue's title is Consider moving the buildsystem to Meson. I think it is inappropriate to have CMake discussion here. Discussion about CMake should have it's own issue. Comparison of Meson vs anything else is fine.

zaniar on 6 Sep 2018

👍1

@zaniar There's been a lot of discussion in this thread regarding build systems in general. Just because the title hasn't been updated doesn't mean we're going to suddenly create a new issue and lose all that context for discussion and start all over again. I think asking us to suddenly throw away several months worth of discussions is inappropriate.

@fire I've been busy with work the last few weeks, I've got a large deadline for Friday, and then after that I might have some time to get my fork up to date. I've got a few changes locally I forgot to push up but I don't have the time right now. (Between this build system port, my fork of gnu m4, 15 proposals I'm writing for the C++ standards San Diego meeting and CppCon in the last week of the month, I'm absolutely swamped, and I have no one to blame but myself)

slurps-mad-rips on 6 Sep 2018

Yeah, Meson was the original discussion target, but I think it's kind of natural that other build systems have been brought up and discussed since, and it probably makes sense to keep the discussion within this issue here instead of spreading it out, like @slurps-mad-rips said.

RiverMesa on 8 Sep 2018

I agree with moving to CMake. It has a lot of support with its toolchains. Android builds for example become very easy compared to what Godot is doing in its SCsub file which looks 'unmaintainable' to me.

As an open source project, Godot should go with solutions that are easy to maintain and that contributors are more likely to know and be able to contribute to.

Also, I think this is important for GDNative adoption. Whichever build tool is used, there needs to be an example or standard project with easily modifiable build script that supports building for all platforms, including iOS, Android, etc.

Also normal modules should be able to be compiled as shared libraries. This is related to #19486 so that the engine can be separated into multiple downloadable components.

chanon on 16 Oct 2018

👍2

I'm actually tackling these. I haven't pushed my changes up in a while because I've been busy with CppCon, the upcoming C++ standards meeting, and we're doing a release next week at work. Suffice to say, I've not had much time to finish up the port, but most of my time is going to be spent ripping the python scripts that generate C++ code out going forward. I'm hoping to resume work on my port this weekend. I'm currently evaluating whether to automatically acquire third party dependencies and apply patches stored in the repo or not. One question I do have for the core developers is how open they would be to breaking all the third_party repos out of the main repo and into their own repositories on the organization. I'm relying fairly heavily on FetchContent in my own projects, and I'm currently wishing I could do that for these third party dependencies that have been slightly forked. This would also let me work on each one in turn to clean up their cmake files while also doing incremental changes to the core repo once each third party library is ported.

Additionally should note, is that I'm trying to also modularize the engine itself as well, so these components could technically be updated separate from each other if someone wanted it. This might knock out the issue that @chanon mentioned.

Lastly, I'm adding support for a few additional build tools for the core developers if they're found on the system. Things like clang-format, clang-tidy, ccache or sccache, distcc, clang-check, address sanitizer, ubsanitizer, thread sanitizer, etc. These are mostly linters (and might make some people's heads explode with the warnings and various errors they've been detecting) but they'll be optional to enable. At the very least using sccache or ccache will give some build improvement when switching between builds and vice versa. Waiting for CMake 3.13 to be released would be nice as this would solve some of the kludge-work I've had to rely on, but the important thing is that I'm setting it up so that the current "getting started" workflow is to still run pip install via Python 3.5 or later, however it'll be inside of a virtualenv so people don't have to mess with their system's environment. (This would also allow others to experiment with using additional python libraries as dependencies for various code generation scripts and not have to remember to uninstall them later)

Anyways, sorry for the update sudden info dump y'all. I'm dedicated to getting this port finished, but life has gotten in the way recently 😅

slurps-mad-rips on 16 Oct 2018

👍6 ❤2 🎉2

I'm relying fairly heavily on FetchContent in my own projects, and I'm currently wishing I could do that for these third party dependencies that have been slightly forked.

You could also use Hunter and make a godot-specific code repository (a special git repo, like on github for example) and access all the dependencies that way as it handles the building, cache'ing, etc... as necessary and properly (or submit all the dependencies to the 'main' Hunter package repo and just use it straight). Hunter is just a single cmake file (HunterGate.cmake) and a call to it to acquire the package repository information (whether the official one or a custom one).

OvermindDL1 on 16 Oct 2018

A humble request: this issue's comment thread has gotten huge - would someone who's been following it be able to summarize the general discussion-so-far in a comment (and maybe it could be linked to from @RiverMesa's initial post)?

voithos on 17 Oct 2018

Hello everyone. I am planning to use Godot as my next engine for making a small game.

I saw this conversation and as a person who has experience in both CMake and Meson (and autotools, tup, plain make, waf, SCons, trying full projects in all of these tools...) I wanted to give my feedback.

The systems I used more intensively have been Autotools, CMake and Meson. I have been using Meson and CMake for the past years, since I already discarded all the other build systems in production setups.

Pros of cmake

Mature project generators if you care about Visual Studio (I think Meson does support it well nowadays but did not try lately) and especially XCode (pretty sure it is more lacking here).
Wider adoption, more support in IDEs
Wider adoption -> easier to get contributions (though Meson is a breeze to learn I must say)

Pros of Meson

many useful targets by default: sanitizers, unity builds for free, precompiled headers for free, coverage for free...
the documentation kicks the ass out of CMake documentation: http://mesonbuild.com/
I found cross compilation to be much easier
Meson has an obvious way to do each thing

Which one would I choose? I would choose Meson if cross-compilation is going to be intensive and you care about fast compile-times (unity builds and precompiled headers are supported by default).
I would consider CMake since it is the standard and contributions maybe are important for the team. I would choose myself Meson personally, but that is just a personal opinion, since I find the tool just better and that saves my own time (CMake scripting, space handling, generator expressions, etc... I am looking at you).

Sorry for this shameless self-promotion below , but I think that the context is begging for it.

I have a small series of articles about more or less basic Meson here if anyone is curious (4 articles):

https://medium.com/@germandiagogomez/getting-started-with-meson-build-system-and-c-83270f444bee
https://medium.com/@germandiagogomez/getting-started-with-meson-in-c-part-2-58150354ff17
https://medium.com/@germandiagogomez/getting-started-with-meson-in-c-part-3-70b9bc419957
https://medium.com/@germandiagogomez/getting-started-with-meson-part-4-8bceec6149e1

I have a (bit outdated) reply on build systems in StackOverflow:

https://stackoverflow.com/questions/5837764/autotools-vs-cmake-for-both-windows-and-linux-compilation/24953691#24953691

Just my two cents :)

germandiago on 31 Oct 2018

❤8

@germandiago That is a great writeup -- I've been following this issue for a while and it would be wonderful if I could work with godot + CLion, which is sadly CMake only.

dorkbox on 31 Oct 2018

Yeah most modern IDE's are either CMake-only, and/or some Custom Build System, and/or 'raw calls' that lose a lot of features. CMake really has become standard, whether it is merited or not.

OvermindDL1 on 31 Oct 2018

@dorkbox if I can get these changes I'm working on in, you'd definitely be able to work in a CLion (cmake + ninja) workflow.

@germandiago

I'm actually friends now (more or less 😛), with the author of meson as of CppCon 2018. We both ended up on the same paper for the upcoming meeting regarding modules titled "Remember the Fortran", and I'll be presenting his paper at said meeting (It's next week, actually!), and we also were on the same build systems panel at CppCon.

I would love to use Meson. If you read further up in the thread, I had tried a meson port, but things were too integrated with Scons at the time. I'm still pulling out the current code generation scripts to be separate 'executable' scripts so that if, in the future, a move to Meson would not be out of the question. However it's slow work, and I have taken to writing a few jinja templates to help with the code generation so scons can be dropped entirely. That said, while my public fork is out of date, the one I have at home is a bit further ahead (and synced to a private fork). Work has unfortunately gotten in the way, and San Diego is unfortunately going to be 'stealing' more of my time. After that, however, I should be free to remove some of the local experiments I have on my private fork and finally get back on the godot cmake train. I am committed to making sure a full pull request can go in without also having a massive disruption in the current workflow process.

(As an aside: thanks everyone for the patience you've had while I work on this. I know it's been a while since I started it, but I represent my employer for the US National Body for the C++ standards committee now so my schedule has been a bit more full lately, especially as we hit important topics like modules, coroutines, and ranges)

slurps-mad-rips on 31 Oct 2018

Yeah most modern IDE's are either CMake-only, and/or some Custom Build System, and/or 'raw calls' that lose a lot of features. CMake really has become standard, whether it is merited or not.

A possible useful tool for using meson with IDEs:

https://github.com/prozum/meson-cmake-wrapper

It makes the IDE think it's using CMake, but is really using meson under the hood.

Some background: I was a huge proponent of CMake for many (10?) years, but fell in love with meson this last year or so. Meson seems to be written with the intent of having CMake's useful features while fixing/preventing many of CMake's annoyances. One roadblock I'm currently facing for adopting meson in a project is the IDE support. I just discovered the above tool, so maybe that will help.

wberrier on 3 Nov 2018

👍4

All and every build system have its major flaws, more than that, there is no reason to change that from technical point of vew, be it scons, autotools, cmake, meson, or whatever make vs ninja your religion is. I think it would be great to close all other build system change bugs as duplicate of this one.

slapin on 18 Nov 2018

<reduz> iFire: no, you did not, all you did was build it but did not port any of the dozens of custom scripts that generate code
<iFire> reduz: the other person worked on the little python script generation on her branch
<iFire> so they directly imported the python env
<iFire> it's workable
<reduz> iFire: that is what everyone said that attempted it, and failed :P
<reduz> there are too many things in there

Prove reduz wrong @slurps-mad-rips :).

fire on 27 Nov 2018

👍3

800lb gorilla in the room that hasn't been mentioned yet. One of Godot's claims to fame is cross platform compatibility. It should follow that whatever build system that is a potential for migrating to should be available and tested on all platforms that Godot targets. Whoever is behind such a migration should be inclined to test the migration on all supported platforms, or at least be part of a team that is doing so. Otherwise I would see such a move as contrary to Godot philosophy and generally just a bad idea.

lazybullfrog on 10 Dec 2018

@lazybullfrog I have access to about 5 separate machines for desktop, and several mobile devices. Part of my port to cmake is making sure there are cmake toolchains available for all the current platforms.

slurps-mad-rips on 10 Dec 2018

❤7

@slurps-mad-rips Is one of those Haiku? If not, I'd be willing to help test. Thanks.

lazybullfrog on 10 Dec 2018

Part of my port to cmake is making sure there are cmake toolchains available for all the current platforms.

Don't forget as well that if you need cmake toolchains for a platform then there is probably a prebuilt one at https://github.com/ruslo/polly and if there is not then you should PR it. :-)

OvermindDL1 on 10 Dec 2018

@slurps-mad-rips How has it been going? I noticed you updated the code on Jan 14th. I tried compiling on Windows but it didn't find pkgconfig.

fire on 13 Feb 2019

@fire life decided to throw a few curveballs my way and I had to take care of them. Personal stuff, unfortunately. I've been focusing on work and finishing the IXM library that I'm using to reduce the amount of work that would be copied into godot. Luckily I'm taking a vacation in a week (I'm heading to Kona for the WG21 ISO meeting for this quarter this weekend) so I'll have some spare time to work on this, and (hopefully!) I'll have a working build for macOS, Linux, and Windows soon™. Android and iOS, followed by FreeBSD and Haiku (sorry @lazybullfrog, though you're more than welcome to test early/often) will be my next priority after that.

The biggest chunk of work/blocker thus far has been extracting the python scripts into "installable" tools that are separate from Scons and executed like processes. The actual grabbing of dependencies, settings, etc. has been easier in comparison.

slurps-mad-rips on 13 Feb 2019

👍2

@slurps-mad-rips, life has thrown me a few curve balls too. Perhaps we should start a baseball team. 😂

Fortunately one of those curve balls since I last commented was a 8 core Xeon with 32gb ram with Haiku and FreeBSD installed. Perhaps soon I'll get some time to help test both.

I'm actually ecstatic that both of these platforms are getting attention. Look forward to testing soon.

lazybullfrog on 13 Feb 2019

I think it is reasonable to have multiple build-systems. Perhaps an official build-system that is well supported and the others provided on a "you figure it out" basis. Users will want to use their own build-system regardless, so why not pool their work into one place?

I would be willing to contribute a Bazel port if things go that way.

njlr on 4 Mar 2019

I think it is reasonable to have multiple build-systems. Perhaps an official build-system that is well supported and the others provided on a "you figure it out" basis. Users will want to use their own build-system regardless, so why not pool their work into one place?

As a Linux packager, I have seen hundreds of projects with various buildsystems. I've seen a few which did have several buildsystems available. I've never seen one where it was not a complete mess, especially when the alternative buildsystems were contributed by first-time "please add CMake it's so cool" contributors who then disappeared from the project, and maintainers didn't bother updating it.

There's no value at all in supporting several buildsystems, and I won't merge any PR going in that direction. I'm one of the main buildysytem maintainers, and I definitely don't want to have the burden of maintaining alternative ones whenever we need to change something.

We currently have a buildsystem that works for everyone and is well supported, and no intent to change. If someone really wants to investigate other options, feel free to, and if the final result also works for everyone, can be well supported and provides added value without big drawbacks, we'll consider it.

akien-mga on 4 Mar 2019

😕2 👍1

If someone _really_ wants to investigate other options, feel free to, and if the final result also works for everyone, can be well supported and provides added value without big drawbacks, we'll consider it.

That's what I've been doing with my CMake port. Currently a no-op or single file change under scons takes upwards of 10 seconds on my current machine (which has 64GB of RAM, comparable processor, etc.) With ninja + cmake the entire configure + generate process takes less time, and the ninja build finishes in under 0.2 seconds in a no-op build (on windows, where launching a process is expensive). I'm also able to use the sccache tool from mozilla if its installed, and that's been extremely useful for editing the codegen parts/running a regenerated build from scratch to make sure nothing is depending on my local environment.

most of my time has been spent on extracting the code generation scripts into python scripts that could run on their own. If that was the only thing merged into godot, my time spent on this wouldn't have been a total waste. It also solves the run_in_subprocess hack that currently exists to solve issues on windows with parallelization.

I just returned from my trip and was able to get a bit of work done. Right now I still need to copy a few files over from a manual scons run, but I've been chipping away at the generated sources as time goes on. The license.gen.h and method_bind.inc headers are currently my biggest target, but I'm taking a few days to deal with jetlag + work before I'll tackle them.

(also just gonna cc @fire since he's been interested in this progress for a bit)

slurps-mad-rips on 4 Mar 2019

👍6

From my experience Meson is totally f*ed up when building Android on Windows, I just cannot get it work to compile one of third party libraries that uses it (libepoxy). And I think it's too much tied to concrete compiler stuff.

Antonz0 on 19 Sep 2019

All I desire is a build system that can generate project files and database files separately from the build command.

Every time I check out a new branch with major changes to files, I have to run Scons from the command-line to regenerate my vsproj files, which also causes a full build to occur anyways. In Scons from what I have seen, there is no project generation stage of any sorts. This is also why it doesn't support generating compilation databases out of the box for usage in clang-tools.

Currently, the only viable solution to coding Godot on windows (IDE-wise) is with Visual Studio, otherwise be prepared to just use a text-editor.

Scons is wonderful and all, but when I have a 12 core threadripper and 128gb of ram and I have to wait 12+ seconds on windows to compile a single line change (most of that is Scons thinking), it gets a bit annoying.

But the real downside to Scons (and this could be my laziness in searching) is that I cannot compile individual cpp files. Being able to compile individual cpp files without building the entire project is a huge productivity booster and sanity checker.

Its not just line changes that take time, even figuring out if there is anything to build takes time...

2cs

Running from git-bash:

$ time scons -j24 num_jobs=24 vsproj=true platform=windows
scons: Reading SConscript files ...
Configuring for Windows: target=debug, bits=default
 Found MSVC version 14.2, arch amd64, bits=64
YASM is necessary for WebM SIMD optimizations.
WebM SIMD optimizations are disabled. Check if your CPU architecture, CPU bits or platform are supported!
Checking for C header file mntent.h... no
scons: done reading SConscript files.
scons: Building targets ...
[Initial build] progress_finish(["progress_finish"], [])
[Initial build] scons: done building targets.

real    0m16.082s
user    0m0.000s
sys     0m0.030s

marstaik on 3 Nov 2019

If you want to build only a single target file, then specify that on the command line

scons -j24 num_jobs=24 vsproj=true platform=windows <path  to targetfile>/targetfile.obj

bdbaddog on 4 Nov 2019

Just wanted to add that the latest version of CMake is adding some of the missing features people mentioned earlier as reasons to not use it.
Unity builds: https://cmake.org/cmake/help/latest/prop_tgt/UNITY_BUILD.html
Precompiled headers: https://cmake.org/cmake/help/latest/command/target_precompile_headers.html

Nosliwnayr on 15 Nov 2019

@Nosliwnayr Both of those have been supported by addons for many years, they are well tested and wanted enough now to be included. Those were never reasons not to use CMake as you've been able to do both in CMake for a very long time.

OvermindDL1 on 15 Nov 2019

@bdbaddog
i don't understand, do i have to specificy every single file manually? what does that .obj thing do?
how do i apply it to the whole project? does it automatically recompile individual files?

i hope the compile times can be optimized further, being able to compile c++ code near instantly with no tricks would be a god send - after seeing how fast godot compiles, i'm really struggling to go back to the unreal engine 4 (compiling it, took me 45 minutes, but now the engine somehow got more bloated, and it takes up to 60 minutes, which is fucking insane)

i mean i think godot has the potential to compile even faster, and i would replace the build system myself, but i have 0 experience, so i literally would have no idea as to what to do

@Nosliwnayr
can we please avoid pre compiled headers? i think they are complete garbage, just keep it simple

they also making sharing projects, way more annoying, and it's just never ending bloat, with too many downsides in my mind - but hey, if someone used them effectively, i'm cool, but to me they seem like a dirty trick, instead of a true optimization, and i don't like cheap tricks to be honest

after using the unreal engine 4, i'm tired of 20 GBs worth of pre compiled bullshit, i just don't think they are worth it, there has to be a less stupid system (the unreal engine 4, took up to 60 - 80 GBs for me, not cool)

if godot heads into that direction, i'll asume that the developers have lost their god damn minds

hell, fuck it, i'd rather have them stick to scons, if it avoids the unreal engine 4's ball of insanity, i think the smarter approach, would be to improve scons, instead of outright replacing it, with a whole new broken ass compiling system, when the current one works... more or less

i think the mental effort required, in order to shove in a brand new system, is way higher, than just improving on what we already have, and the new compiler, might end up being functionally broken for months

i sure as hell am not going to complain if it gets done though, but with such a critical engine component, you pretty much have to go all in, there's no turning back once this changes

ghost on 5 Dec 2019

@RaTcHeT302 - The command line specified is if you wanted to explicitly only compile a single target (in this case an object file) (and of course the files it depends on)

Indeed if all the folks complaining about issues with SCons would contribute to helping fix them rather than suggesting switching a complicated build system to another newer shinier one.. well many of such concerns would be fixed..

bdbaddog on 6 Dec 2019

@bdbaddog Scons does not generate intermediate stage build files. It does the "project dependency" step in the same step as it does compile, and they cannot be separated.

Its like telling CMake to generate the project files everytime before you hit your build tool, whether it be MSVC, ninja-build, make, llvm, pick your poison.

If I switch git branches, I cannot update visual studio without performing a scons command line build to regenerate the proper .vsproj and .sln files - and I almost always clean build branch changes since I have ran into issues in the past.

Those intermediate files arent just for building, they can be used by static analyzers and other projects as well.

While I agree that cmake isn't perfect (I don't like it as much myself), and I would prefer the flexibility of python, cmake has the best integration and support of all the build tools on the table.

marstaik on 6 Dec 2019

👍1

When converting my fork to CMake most of the complexity was in the code generation files. The idea that the build is complicated is a fallacy. The build doesn't need scons because its complex. It's complex because it uses scons.

With that said, the reason I halted work is because I met with the lead devs at GDC in 2019 (at the GitHub meetup) and they said they have absolutely no intention of switching to something more usable, citing that they'd prefer to do the build entirely on linux and indeed from what I've seen they're violating the EULA that Apple requires acceptance of to install build tools for cross compiling.

The mentality of those in charge is very much "ends justify the means" and there doesn't seem to be any care about treating the build as software, and instead godot treats it as a series of scripts that happens to produce a build.

I will not be contributing the changes I made to the code generation to godot (despite it being more readable and faster), and I've given up on this engine entirely and moved on with my life. I'm much happier if I'm being honest. The engine already has considerable tech debt and I see it being an issue in about 3 years, especially as the C++ standard committee is working on a tooling technical report (I'm involved in this and have been pushing for things like options files being standardized to a degree, but it's just a technical report so some folks will diverge from it).

It's a shame too. My CMake build was able to catch a bunch of static analysis errors that have led to tons of issues and are easily solvable, but I'm too burned out from having argued with the devs about this at GDC to file bugs or even attempt to fix them. My energy and my mental health are better spent elsewhere.

There is no world where scons will be able to handle the upcoming C++20 modules design, as much as I have shouted how it will break many tools. It can't handle the dependency tracking for that, and would require it to have a pre-scan step, which will slow the dependency tracking down even further.

Lastly, I'm going to suggest people not try to improve a tool that can only accept minor changes. Scons would need a complete rewrite from the ground up and a large number of breaking changes to get the improvements people here seek.

I suggest this issue be closed as the discussion of changing the build system was answered at GDC 2019, but the leads don't even think its worth wasting their time to comment on this issue, which I think shows how much they care about the ergonomics of development.

slurps-mad-rips on 6 Dec 2019

😕4 👍2 👀1

I feel as though if the existing build system were 100% converted in a plug and play fashion there would be no reason from the devs not to switch.

Building only on Linux is useless. Having a game engine that doesn't run on windows will deter any serious game developers.

I would like to be under the impression that the game developers care enough to change the build system to satisfy its users.

One of the reasons I left Unreal is that the engine is tailored primarily for Blueprint/front end game creation. Creating tooling in c++ modules is extremely painful and complex in Unreal.

So far, in Godot it has been decently pleasant. Making C++ builds faster would be a big plus for the community.

marstaik on 6 Dec 2019

👍2

IIRC they use Wine to run the MSVC build tools. Still doesn't change your point, and I agree.

slurps-mad-rips on 6 Dec 2019

Building only on Linux is useless. Having a game engine that doesn't run on windows will deter any serious game developers.

Godot has always been buildable on many platforms, not just Linux…

IIRC they use Wine to run the MSVC build tools. Still doesn't change your point, and I agree.

Official Windows builds are compiled using MinGW; we don't use WINE to run MSVC on Linux.

Calinou on 6 Dec 2019

👍1

@marstaik - if you switch builds and run scons just to generate the MSVS Project files, you should be able to specify those files as the target and not have to do a full build.

so try:

scons <path to created MSVS project file>/VSPROJECTFILENAME (replace with actual paths and file name which are generated)

SCons should be able to do this without doing a complete build and also it should do it correctly without you needing to do a clean first.
(BTW. I'm the SCons project maintainer)

bdbaddog on 6 Dec 2019

@bdbaddog I had to add vsproj=True to that command, but it seems to have worked

From the godot src:
scons -j24 godot.vxcproj num_jobs=x vsproj=true

Edit: For some reason you need to add the -j flag also...

marstaik on 7 Dec 2019

@bdbaddog I had to add vsproj=True to that command, but it seems to have worked

From the godot src:
scons -j24 godot.vxcproj num_jobs=x vsproj=true

Edit: For some reason you need to add the -j flag also...

A lot of such flags aren't vanilla SCons, but part of the way various packages (Godot in this case) implement their build system with SCons.
Glad you found a way!

bdbaddog on 8 Dec 2019

Why use SConstruct and insist on not switching to other build system???

tjysdsg on 20 Feb 2020

@tjysdsg The discussion above sums that up very well :slightly_smiling_face:

Calinou on 20 Feb 2020

I'll close this because we've made it clear over the years that we have no intention a priori to change the buildsystem. We're happy with SCons and any of its shortcomings can and should be assessed individually, both within our own SCons usage and upstream if it's due to the tooling itself.

Switching to a "cooler" buildsystem for the sake of it will not happen. The vast majority of core contributors is happy using SCons and it's the one we now know best. Switching to a new buildsystem has both an implementation and a learning cost, and it hasn't been proven so far that those would be worth the trouble.

All in all, a buildsystem change is not something that we'd accept from new contributors. It has to come from the inside, and all core contributors must see it as a positive change for it to make any sense. We won't change the buildsystem just to please new users who attempt to build Godot from source and somehow don't like that they have to type scons instead of cmake or meson which they are used to. We also will not accept attempts to introduce an optional new buildsystem - from experience, I know that this can only lead to inconsistency, bugs and output differences.

Of course we're open to changing our minds if clear gains can be listed (and shown with a working proof of concept) that outweigh the costs of changing the buildsystem, but as of today and after two years of discussion here, I personally don't see the incentive.

akien-mga on 20 Feb 2020

👍5 👎1

@slurps-mad-rips said it well.

It's complex because it uses scons.

and

My CMake build was able to catch a bunch of static analysis errors that have led to tons of issues and are easily solvable

Flat out disgusting that CMake isn't given serious consideration, I can think of many reasons to ditch SCons, but SPECIFICALLY that CMake finds errors Scons does not find makes it "worth it" to use something else.

Switching to a "cooler" buildsystem for the sake of it will not happen

CMake isn't "cooler", it's the standard. If you actually used it you would immediately understand why.

Of course we're open to changing our minds if clear gains can be listed (and shown with a working proof of concept) that outweigh the costs of changing the buildsystem

I'm not convinced that anyone is open to changing their minds. Again, @slurps-mad-rips showed clear gains, and she even did all the work... It's already done.

It is shameful, representing the official stance of the Godot team, to completely disregard her efforts improving the development process.

dorkbox on 21 Feb 2020

I quite understand your pain but I still agree with others - all the build
systems have their
pros and cons and generally the same. There is rarely any technical reason
to change it.
As for political reasons they will not do you anything good in the long
run. Everybody
prefers something different. Anyway nobody stops anyone from doing forks
for whatever
reasons, so I guess this one is as good as any.

On Fri, Feb 21, 2020 at 1:38 PM dorkbox notifications@github.com wrote:

@slurps-mad-rips https://github.com/slurps-mad-rips said it well.

It's complex because it uses scons.

and

My CMake build was able to catch a bunch of static analysis errors that
have led to tons of issues and are easily solvable

Flat out disgusting that CMake isn't given serious consideration, I can
think of many reasons to ditch SCons, but SPECIFICALLY that CMake finds
errors Scons does not find makes it's "worth it" to use something else.

Switching to a "cooler" buildsystem for the sake of it will not happen

CMake isn't "cooler", it's the standard. If you actually used it you would
immediately understand why.

Of course we're open to changing our minds if clear gains can be listed
(and shown with a working proof of concept) that outweigh the costs of
changing the buildsystem

I'm not convinced that anyone is open to changing their minds. Again,
@slurps-mad-rips https://github.com/slurps-mad-rips showed clear gains,
and she even did all the work... It's already done.

It is shameful, representing the official stance of the Godot team, to
completely disregard her efforts improving the development process.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/16014?email_source=notifications&email_token=AAABPU2PBIEK6PFKQ2LDTHLRD6VKVA5CNFSM4ENJ6NN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMSI4JY#issuecomment-589598247,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAABPU746R4YFHAI73Z57RTRD6VKVANCNFSM4ENJ6NNQ
.

slapin on 21 Feb 2020

@dorkbox - cmake found static analysis errors? or was it the compilation database output from cmake run through llvm's static analysis tools which found them?

bdbaddog on 22 Feb 2020

@bdbaddog, @slurps-mad-rips did the work on this, and it would be best to ask her. (I think it would be the output from llvm's static analysis, but I'm not the one who did the work to build via cmake)

dorkbox on 23 Feb 2020

@slapin

There is rarely any technical reason to change it.

How about static analysis tools and modern language features support?

dorkbox on 23 Feb 2020

@bdbaddog @dorkbox I simply turned on clang-tidy to look for things like "returning references to non-static locals" and "buffer overflows" via CMake. I also ran clang-check on a few things and got a few more results. I was able to get include-what-you-use to run as well, and it found quite a few unnecessary header inclusions.

That said, I didn't actually "finish" the cmake port. The part that never got ported was all the work I did with trying to change the current python scripts that simply call f.write on some file to generate a file from JSON. I was able to remove a lot of that logic and replace it with a generic jinja2 based template generator. This also solved a hack that the current scons build does on windows where it launches a script as an executable.

However I had hoped to keep parity with the scons build and at the very least get my changes to the code generation merged in. This didn't happen because the python scripts used to generate source files are constantly changing, and keeping those changes in sync with my jinja2 templatization was an exercise in frustration.

Given the very emphatic and hard "no" I got from the lead developers at the GDC GitHub meetup, I decided I no longer wanted to contribute to godot. My changes were unwanted, are not desired, and they are apathetic to anyone else's opinion on this matter. I won't be contributing to godot, nor to any forks of godot. I wish you all luck in trying to convince the lead devs in the matter but it's like talking to a brick wall. Unyielding, immovable, and apathetic to the plight of everyone in this thread.

slurps-mad-rips on 23 Feb 2020

So if SCons can spit out the compilation database which is input to the llvm tools then this would also be able to be done from SCons.

bdbaddog on 23 Feb 2020

So if SCons can spit out the compilation database which is input to the llvm tools then this would also be able to be done from SCons.

It looks like there is the sunk-cost fallacy going on with SCons. Why not use something that is easier, streamlined, and already part of existing build tools, like cmake?

From the scons website:

When compared to scons, CMake is :

Faster

Requires less code for common tasks

Arguably more stable

Supports outputting to projects like Code::Blocks, Xcode, etc. which scons does not

dorkbox on 23 Feb 2020

@dorkbox - I'm the SCons project co-manager. I'm just keeping an eye on desired functionality.

The last few releases and the forthcoming one have had performance improvements.

Additionally we have PR's for ninja file generation and some community members have compilation database tools. So some of the issues are likely to be addressed (at least partially, if not fully) in the next release.

I'm not aware of any stability issues affecting the godot project which we haven't addressed. If there are any outstanding please let us know.
(Reach us via: https://scons.org/contact.html)

I'm not sure cmake really requires less code for common tasks, I'd be interested in reasonable comparison.

bdbaddog on 23 Feb 2020

👍3

Well, what mostly annjoys me with scons is inability to generate plain old
GNU make makefile. Lack of ninja - I can live with it.
Also handling of cross-compile utility like CMake toolchains and autotools
cross-compiling capabilities is strong cmake weak point.
The rest is ok as I'm not IDE person. My IDE is UNIX environment.

On Sun, Feb 23, 2020 at 10:05 PM William Deegan notifications@github.com
wrote:

@dorkbox https://github.com/dorkbox - I'm the SCons project co-manager.
I'm just keeping an eye on desired functionality.

The last few releases and the forthcoming one have had performance
improvements.

Additionally we have PR's for ninja file generation and some community
members have compilation database tools. So some of the issues are likely
to be addressed (at least partially, if not fully) in the next release.

I'm not aware of any stability issues affecting the godot project which we
haven't addressed. If there are any outstanding please let us know.
(Reach us via: https://scons.org/contact.html)

I'm not sure cmake really requires less code for common tasks, I'd be
interested in reasonable comparison.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/16014?email_source=notifications&email_token=AAABPU5TQQFET5RS2JDRY7TRELCHDA5CNFSM4ENJ6NN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMWEOMI#issuecomment-590104369,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAABPU6UHPX3HJTWPAFTPATRELCHDANCNFSM4ENJ6NNQ
.

slapin on 25 Feb 2020

Godot: Consider moving the buildsystem to Meson or another build system

Most helpful comment

All 142 comments

Install cmake

Pros of cmake

Pros of Meson

Related issues