Runtime: build-native.[cmd|sh] should not depend on the "run" tool

[ellismg]

We should remove build-native's dependency on run.exe. The rationale is that run requires managed code but there are lot of scinerios where we need to build the native components without being able to run managed code (platform bring-up is one case).

In general, we need to be able to build native code without invoking managed code and the use of the run tool in build-native is an impediment to that.

This does imply a bit more cost in maintenance as we have to ensure that options to build-native remain somewhat consistent across platforms (and we need to do a better job of documenting them) but I think this cost is worth the flexibility.

[ellismg]

/cc @weshaggard @chcosta

A possible workaround today is to invoke src/Native/build-native.sh directly. I would like to get all repositories that produce native code in a world where build-native.sh from the root does the right thing and we have consistent options for things like Configuration and Architecture.

[weshaggard]

This is definitely an interesting issue which I agree we need to figure out. I'm a little torn about whether or not we have two modes one that uses run and one that doesn't. I still think the usage documentation being shared with run is interesting. What if root script still used run for most cases but has a mode where if you run it on another machine with "-whatif" it will dump out the raw non-run based commandline you can execute. Does that seem like a reasonable approach?

[jkotas]

I think we should be working towards getting rid of the run tool completely. It is unnecessary complexity that makes the system hard to understand and debug.

[weshaggard]

The run tool is supposed to serve two purposes. One is to provide a common way to parse command line options so we don't have to duplicate it in all the scripts to try and maintain it. The other is to provide a way to document all the various command line options. We can debate whether or not it provides enough value for the complexity, I could live with docs for the options but I really hate seeing a bunch of crazy argument parsing duplicated in all the scripts.

cc @markwilkie @mattgal

[jkotas]

I have been avoiding the run tool as much as I can. To give you one example, here is the first hand experience:

C:\corefx>run
Tools are already initialized.
Error: No command given.
Our dev workflow has changed! Use -? for help in the new options we have and how to pass parameters now.

I do not understand why do I care that the dev workflow has change, but lets see what the command line options are:

C:\corefx>run -?
Tools are already initialized.
Error: Object reference not set to an instance of an object.    at Microsoft.DotNet.Execute.Setup.FormatSetting(String option, String value, String type, String toolName)
...

Ok, we are on Windows. Maybe the /? will help?

C:\corefx>run /?
Tools are already initialized.
Calls one batch program from another.

CALL [drive:][path]filename [batch-parameters]
...

Even once you figure out how it sort of works, it does not get better. I never does what I want.

[weshaggard]

Those are bugs which should be fixed, assuming it gets better do you feel it can add enough value or do you feel the potential for value is still too limited?

[jkotas]

I think the value is still too limited - not enough to compensate for the extra complexity that comes with it.

[karelz]

Command line parsing can be done in scripts - yes, we will need 2 scripts (Windows & Linux), but if they are written nicely in a readable form, they are also easy to debug, understand, comprehend and improve -- way much better than some magic in a tool.
Anytime we have a black-box tool, we highly raise a bar for improvements and contributions - only folks familiar with it will be likely to fix it / change it (because you need to know where the source is, how to update the binary, etc., etc.).

I wrote my share of batch files with extensive command line parsing and will be happy to help / provide guidance. I expect it will be even easier on Linux.

[weshaggard]

I think the value is still too limited - not enough to compensate for the extra complexity that comes with it.

I think that is fair and we should consider our options. @MattGal @markwilkie can you guys take that feedback and see what our best options are?

Command line parsing can be done in scripts - yes, we will need 2 scripts (Windows & Linux),

I would completely agree with you if it were only 2 scripts however it is many more than 2. Every script that we had duplicated a bunch of logic that was constantly outdated and redundant and needed to be updated in many files.

I expect it will be even easier on Linux.

The duplication on all the linux scripts are actually the worst part of this problem because of all the various versions of linux that we need to deal with in computing the arguments. Take a look at https://github.com/dotnet/corefx/commit/7e2bd07936179c192e682d979b2938b4a7e32030 for some idea of the script duplication we eliminated when we switched to the run tool.

[jkotas]

The duplication on all the linux scripts are actually the worst part of this problem

You can solve this problem by moving the common part into a helper script. Check how it is done in CoreRT: https://github.com/dotnet/corert/blob/master/buildscripts/buildvars-setup.sh

[markwilkie]

There's wide agreement that we can do better. To reiterate the goal - it's to have a common interface (contract if you will) across all of our repos regardless of platform. (Obviously there will be bits that are specific to each repo, and that's ok of course)

There is a tracking issue to learn from what we've done so far, and get to a better for functional (and simpler) place. https://github.com/dotnet/core-eng/issues/327

[ViktorHofer]

done with the arcade move.