Darktable: Question about parallel building.

[ItaloCunhaBueno]

Hi all, I'm a Windows user, I often compile from source using the BUILD.txt step by step but this seems to take unnecessary long time. Speaking with other users in the darktable facebook page, they pointed me to the parallel building direction.

My question: Is it supported? Using "cmake --build . -j2" (or any other number except 1) crashes with Error 2

Again, I'm no expert in this, any help in speeding up the build process would be very welcome.

Thanks .

[ghost]

I made a note to look into setting CMAKE_BUILD_PARALLEL_LEVEL in the environment, which should work for CMake > 3.12. (My MSYS64 build system, and Appveyor, are on 3.17.)

I haven't tried it yet!

https://cmake.org/cmake/help/latest/envvar/CMAKE_BUILD_PARALLEL_LEVEL.html

[ItaloCunhaBueno]

How do I set this to test it?

[GrahamByrnes]

Let me know how you get on :-)

[ghost]

CMAKE_BUILD_PARALLEL_LEVEL=8 cmake --build . --target package (or whatever target you want)

I have just tried it and it definitely used more CPU, pegging it occasionally, but I think on my system it just went disk-bound instead (I haven't got round to upgrading to SSD). The highest number of cc1.exe processes I saw was 6, so I suspect the build would need some refactoring to get to 8 consistently.

It also reminded me of another tip for build speed - disable Windows Defender real-time checking temporarily!

[ItaloCunhaBueno]

So, I ran some tests, everything done with Windows Defender Real-Time Protection off:

CPU: Intel Core i5 (4 cores)

defaltu BUILD.tx: 00:56:14

cd ~
git clone git://github.com/darktable-org/darktable.git
cd darktable
git submodule init
git submodule update
mkdir build
cd build
cmake -G "MSYS Makefiles" -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/opt/darktable ../.
cmake --build .
cmake --build . --target package

with j5 arg: 00:26:55

cd ~
git clone git://github.com/darktable-org/darktable.git
cd darktable
git submodule init
git submodule update
mkdir build
cd build
cmake -j5 -G "MSYS Makefiles" -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/opt/darktable ../.
*cmake --build . -j5
cmake --build . --target package -j5

*only part of the process with cpu usage difference: 80~100% cpu usage

with CMAKE_BUILD_PARALLEL_LEVEL=8: 00:21:17

cd ~
git clone git://github.com/darktable-org/darktable.git
cd darktable
git submodule init
git submodule update
mkdir build
cd build
CMAKE_BUILD_PARALLEL_LEVEL=8 cmake -G "MSYS Makefiles" -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/opt/darktable ../.
CMAKE_BUILD_PARALLEL_LEVEL=8 cmake --build .
*CMAKE_BUILD_PARALLEL_LEVEL=8 cmake --build . --target package

part of the process with cpu usage difference: almost always 100% cpu usage
*part of the process with cpu usage difference: almost always 40~50% cpu usage

I did not install the resulting installer for each test.

[GrahamByrnes]

Without doing any timing... wow! Much faster :-)
Only step that still seems to take as long is the wait for the final install at the end of --build . --target install

Thank you!

[ItaloCunhaBueno]

I also got it done under 10min with these steps:

#!/bin/bash
cd ~
#git clone git://github.com/darktable-org/darktable.git
cd darktable
git fetch
#git submodule init
git submodule update
#rm -R ./build/
#mkdir build
cd build
CMAKE_BUILD_PARALLEL_LEVEL=8 cmake -G "MSYS Makefiles" -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/opt/darktable ../.
CMAKE_BUILD_PARALLEL_LEVEL=8 cmake --build .
CMAKE_BUILD_PARALLEL_LEVEL=8 cmake --build . --target package

My question is: How dangerous is to rebuild without deleting the build folder before?

[johnny-bit]

My question is: How dangerous is to rebuild without deleting the build folder before?

Unless you go back in time (eg check older version, bisect) or work on new features it's completely fine.

[ItaloCunhaBueno]

My question is: How dangerous is to rebuild without deleting the build folder before?

Unless you go back in time (eg check older version, bisect) or work on new features it's completely fine.

Is it normal to not update version tag?

[ghost]

The failure to update the version tag is a known issue. (I don't think there is a guthib issue open for it though?)

See comment in line 184 of https://github.com/darktable-org/darktable/blob/master/CMakeLists.txt

# FIXME: PROJECT_VERSION will not be updated automatically, until you rerun cmake

[ItaloCunhaBueno]

I see... but anyway, this arg of yours greatly reduced the build time, thank you very much. I hope it get inserted in the BUILD.txt :)

[ghost]

I have a Windows build improvements branch on the go, I will include it in there. ;-) (Not a priority though, I want to get the minimum product of Appveyor for both gcc and clang merged first.)

[ghost]

4m56.490s - used a ramdisk. :-)

[github-actions[bot]]

This issue did not get any activity in the past 30 days and will be closed in 365 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.