Runtime: Redesign the subset feature

[ViktorHofer]

• Combine the -subset and -subsetcategory switches into just -subset. This probably involves making subset names distinct which IMO is a good idea anyway as we should remove the sub repositories (another case of that is removing the helper scripts for coreclr, libraries, mono, installer). Specifying -subset libraries should be sufficient to build all default subsets for libraries. No need for the extra -subsetcategory verb.
• Allow adding to the default set of subsets without the need to specify all of them, ie. -subset libs+libs.tests.
• Allow removing from the default set of subsets without needing to specify all of them...

_Moved proposal up_

Proposal

You can specify multiple subsets. clr, mono, libs and installer imply the child subsets.

 clr.runtime
 linuxdac
 clr.corelib
 clr.corelibnative
 clr.tools
 clr.packages
mono
 mono.runtime
 mono.corelib
 mono.packages
libs
 libs.native
 libs.ref
 libs.src
 libs.pretest
 libs.tests
 libs.packages
installer
 corehost
 installer.managed
 installer.depproj
 installer.packages
 bundles
 installers
 installer.tests

Examples

1. Build libraries and coreclr
   build.cmd/sh libs+clr

2. Build libraries tests:
   build.cmd/sh libs.tests

3. Build mono corelib and libraries
   build.cmd/sh mono.corelib+libs

4. Build libraries tests and installer tests
   build.cmd/sh installer.tests+libs.tests

5. Build coreclr, libraries and libraries tests
   build.cmd/sh clr libs+libs.tests

cc @dotnet/runtime-infrastructure @jkotas @danmosemsft @stephentoub

[msftbot[bot]]

Area Owners

@ViktorHofer, @jeffschwMSFT, @dleeapho

[jkotas]

It would be nice to drop the need for the -subset prefix. I would like to be able to just type build libraries.

[safern]

Something to consider while designing this is how are we going to identify in between different subsets when they clash. For example, if at some point in mono we could build only corelib, how will I be able to build mono corelib if I just say, build -subset corelib, that would be ambiguous if both mono and coreclr have a corelib subset, so we might need to append mono or coreclr to it. Something like, -subset coreclrruntime or coreclrcorelib, and if we want to build all coreclr, you would just say, subset coreclr.

[ViktorHofer]

Something to consider while designing this is how are we going to identify in between different subsets when they clash.

Having unique subset ids make most sense IMO. We shouldn't need to append the sub repo for all of them, ie only installers build installers (lol). What I'm doing in https://github.com/dotnet/runtime/pull/33553 is naming the libraries test subset, which we haven't had before, libtests.

It would be nice to drop the need for the -subset prefix. I would like to be able to just type build libraries.

For the subset I think that makes sense, for other arguments I would like to keep the current syntax which is -<option-name> <option-value> primarily to stay consistent with other repositories. One thing to keep in mind here is that we already overload the "non-option" value (which can occur at any position) to build individual libraries: https://github.com/dotnet/runtime/blob/master/eng/build.ps1#L125-L138. We could change this to require -projects option which already is an Arcade standardized one. As not many libraries devs us that feature I would be ok with changing that. Of course there are other ways to solve this (ie positional, hardcoded checks, ...).

Based on https://github.com/dotnet/runtime/issues/33821, the earliest we can get this in is start of May.

[safern]

Having unique subset ids make most sense IMO.

I agree, and that is why I brought that up. So how would we do it for the scenario I posted above, could you provide an example of what you're envisioning?

For the subset I think that makes sense, for other arguments I would like to keep the current syntax which is - primarily to stay consistent with other repositories.

https://github.com/dotnet/runtime/blob/master/eng/build.ps1#L125-L138. We could change this to require -projects option which already is an Arcade standardized one.

+1 on those two statements.

[BruceForstall]

Related: https://github.com/dotnet/runtime/issues/33167

[BruceForstall]

So we have top-level project subsets (coreclr, libraries, installer, mono) and each has a set of their own subsets (sub-subsets?); for the latter the names only need to be unique within their subset, right? So, e.g., "native" in "coreclr" and "native" in "libraries" mean something different, and don't conflict.

Keeping something similar to what we have today, maybe you should be able to:

build coreclr:runtime+corelib+nativecorelib libraries

where you specify "coreclr" subsets, but build whatever the "libraries" default is.

[danmosemsft]

I liked coreclr build.cmd where I could type eg build -windowmscorlib -linuxmscorlib

Rather than worrying about subsets, why not just have a list of workloads? It could be a long list, if it's logically organized. Then just list off the ones you want eg build coreclr coreclrcorelib or build monoruntime monocorelib libraries ?

[ViktorHofer]

Rather than worrying about subsets, why not just have a list of workloads? It could be a long list, if it's logically organized. Then just list off the ones you want eg build coreclr coreclrcorelib or build monoruntime monocorelib libraries ?

Workloads are the same thing as subsets. What you are proposing is the same that I was thinking of.

Keeping something similar to what we have today, maybe you should be able to:

I would like to get rid of the sub-repository boundaries in the subsets/workloads.

That's a draft proposal:

_Moved proposal into top post_

[trylek]

Looks great to me overall, I'm just wondering whether some of the longer combined words (like coreclrnativecorelib) aren't a bit hard to read/write (even though I guess you're well trained for that with your German language background ;-)) - any chance you'd be willing to consider using some word separator like coreclr-native-corelib, coreclr.native.corelib or coreclr/native/corelib?

[ViktorHofer]

That's a great idea, let me update the proposal.

[ViktorHofer]

Added an example above of how we would do an exclusion with the - sign.

[safern]

coreclr-corelib-native

Maybe that one should just be coreclr-corelibnative?

[ViktorHofer]

I'm fine with either :D

[jkoritzinsky]

If we’re going to allow exclusion with a - sign, I’d rather that we use a different delimiter than a dash in subset names. It would make it easier to read.

[ViktorHofer]

we could use a discard _.

[ViktorHofer]

Doesn't look well either. Other ideas for exclusions?

[trylek]

What about the slash in the subset names i.e. coreclr/corelibnative (according to Santi) - the tree-ish feeling of such syntax would seem pretty natural to me even though I'm not sure if it doesn't clash with any other parser characteristics.

[ViktorHofer]

Hmm I prefer the - as the delimiter but that's just my personal opinion.

[jkoritzinsky]

Using periods as delimiters would be a good option as well.

[ViktorHofer]

[ViktorHofer]

If we come to an agreement by EOD I could try implementing this so that we can still deploy it on 4/6 (https://github.com/dotnet/runtime/issues/33821). I think combining the root helper script removal PR with this one would make sense as it provides a suitable replacement.

[safern]

I think combining the root helper script removal PR with this one would make sense as it provides a suitable replacement.

I agree.

If we’re going to allow exclusion with a - sign, I’d rather that we use a different delimiter than a dash in subset names. It would make it easier to read.

What about a no prefix? Something like, no.libraries.tests?

[ViktorHofer]

At the moment we don't really need the exclusion feature. What about deciding on - or . as the delimiter and iterate on the exclusion afterwards?

What about a no prefix? Something like, no.libraries.tests?

That looks SUPER weird 😀

[safern]

At the moment we don't really need the exclusion feature. What about deciding on - or . as the delimiter and iterate on the exclusion afterwards?

I vote for .

That looks SUPER weird 😀

Yeah I know 🤦‍♂ , but at least is clear that you're saying, don't build this.

[akoeplinger]

I prefer . to - as the delimiter too.

[danmosemsft]

If you need exclusion, how about build coreclr libraries --not libraries-tests libraries-packages

[BruceForstall]

With the current system, the guidance for "most" coreclr developers is to do a build like the following after sync'ing:

build.cmd -subsetcategory coreclr-libraries -subset runtime-corelib-nativecorelib-all -c Release -runtimeconfiguration Debug

with the new proposal, would that be:

build.cmd coreclr.runtime coreclr.corelib coreclr.corelib.native libraries -c Release -runtimeconfiguration Debug

?

While it makes more sense than before, it's still looking really verbose.

(As a nit: could we use "clr" as an alias for "coreclr", and "lib" for "libraries", to shorten things?)

[safern]

(As a nit: could we use "clr" as an alias for "coreclr", and "lib" for "libraries", to shorten things?)

+1 on that.

While it makes more sense than before, it's still looking really verbose.
That's why we want to introduce a way to exclude things, that way you could do:

build.cmd coreclr libraries --not coreclr.tools coreclr.packages -c Release -runtimeConfiguration Release

That would be the equivalent of what you wrote up there.

[ViktorHofer]

@BruceForstall couldn't you instead just do

build.cmd clr lib -c Release -runtimeConfiguration Debug

which admittedly also creates the linuxdac, tools and packages but shouldn't take more than a minute or two?

+1 for lib and clr

[BruceForstall]

which admittedly also creates the linuxdac, tools and packages but shouldn't take more than a minute or two?

correct. when I'm doing a "sync and rebuild the world" that's normally not too bad, but I've had problems where those parts of the build fail locally (esp. nuget?) and that's just annoying if I don't care about those results. If I want to do "smallest possible incremental build to rebuild the JIT" I can do build.cmd clr.runtime -c Debug.

Only somewhat related: we have clr.runtime. What if I just want to rebuild the JIT part of the runtime components. Would it be possible to have a clr.jit option that only rebuilds that part of the CLR native build?

[jkoritzinsky]

We would have to update the native build scripts to be able to do that, but it should be possible in CMake to just say “build and install the jit”, even if it’s not the cleanest.

[ViktorHofer]

Updated the proposal. I'll prepare a PR later today.

[jkotas]

ref
src

Is the lib prefix intentionally missing on these, or is it just a typo?

[ViktorHofer]

Is the lib prefix intentionally missing on these, or is it just a typo?

Subsets / Workloads which are unique don't need to prefixed. A part of the dotnet/runtime repository is to build the ref and src/impl assemblies for the shared framework. Isn't that distinct enough?

[jkotas]

src building libraries implementation only is not intuitive. You have to read fine-print in the manual to figure it out.

I think following the convention for it and naming it libs-src would make more sense.

[ViktorHofer]

Works for me

[ViktorHofer]

Updated the proposal in the top post. I'll start with the implementation today.

[ViktorHofer]

Updated the proposal as I wasn't able to make specifying multiple subsets space separated work on both powershell and bash. Instead you want to combine them with the + sign, i.e. clr.corelib+libs.

[danmosemsft]

Your spec at the top says that libs includes all its children. But you also have an example of specifying libs+libs.tests

[safern]

libs.tests is not included. I think that is the only subset not included by default on libraries, we should update that.

[danmosemsft]

Yeah, probably in -subset help this should make it clearer:

  - Libs
      The libraries native part, refs and source assemblies, test infra and packages.

Would this help? https://github.com/danmosemsft/runtime/pull/new/subsethelp