Packages: Enable parallel builds by default?

[diizzyy]

Since it mainly affects this tree I figured I'd start asking here.

It's 2018 and I think we can in ~95% of all cases assume that parallel builds will work just fine ootb.

Would it be such a bad idea to enable it globally?
Perhaps we can do a testrun on the buildsbots to see how much/what fails?

Also and possibly more important, does anyone have any numbers regarding performance if it's worth the effort at all?

@hnyman @dibdot @jow- @neheb @thess @cshoredaniel

[neheb]

Every package in the openwrt repo has it set to 1 or 0 explicitly. I think the same should be done here.

[diizzyy]

I'm suggesting that we enable it by default (for all packages) and disable by hand, not the other way around.

https://www.freebsd.org/doc/en/books/porters-handbook/building.html
6.5.1. Building Ports in Parallel
Like the above

[neheb]

Yes I know. I still think every package should specify it in the Makefile.

I have no numbers except to say that make V=s vs. -j 4 is painfully slow on WSL.

[diizzyy]

Wouldn't it be a better approach to avoid as much as possible in the Makefile?
In that regard, it would be like adding PKG_SOURCE_DIR in each package just for the sake of it.

[karlp]

I'm with @diizzyy we should be requiring as little as possible in makefiles. Especially when it comes to things that many package maintainers may have no real idea about. I'm all in favour of having it become on by default, and require disabling explicitly in packages that don't support it.

[diizzyy]

Hmm.... Does anyone know where we could get a good data source?
I'm thinking of Archlinux and/or Alpine Linux which is very similar to OpenWrt in terms of software (toolchain) however I can't figure out how to clone the aur repo?
https://wiki.archlinux.org/index.php/makepkg#Parallel_compilation
https://wiki.alpinelinux.org/wiki/Creating_an_Alpine_package#Make_options

[pprindeville]

I'm all in favour of having it become on by default, and require disabling explicitly in packages that don't support it.

Any packages using Cmake or autotools should have inherently parallel safe buildability.

[danielfdickinson]

I think you might want to ping @jow- . I seem to recall that at one time it was enabled by default and that got disabled due to too many random and hard to debug breakages in the packages feed. I think 2-3 years ago it was deemed more trouble than benefit.

[danielfdickinson]

@jow- is most relevant, and probably @hauke as they take care of most of the making sure OpenWrt is releasable which means dealing with fallout from broken things.

[danielfdickinson]

@diizzyy The other consideration is: for a build of bunches of packages at once how is it really going to speed things up? If you're running with -jX you're already using all your cores, just with different packages building at once instead of individual packages using multiple cores. I'll bet buildbots are not only using one core and building a single package at a time.

[neheb]

I recently found out on glib2 that building the documentation broke "make -j 4" for me despite it using autotools. That;s one of many considerations to have while fixing this stuff.

[diizzyy]

Most if it is going to be trial and error

[diizzyy]

https://github.com/openwrt/openwrt/pull/1508

[diizzyy]

Closing this as upstream showed little to no interest