Sdk: Need a way to turn BundledDotnetTools off for source-build

[eerhardt]

Try to source-build up through the CLI.

There are a bunch of BundledDotnetTools that aren't source-buildable (pretty much all of them).

We need to be able to turn these off during source-build. We could possibly use the same build variables we are currently using to not package the rest of ASP.NET in the SDK during source-build.

/cc @dseefeld @omajid @livarcocc @wli3

[omajid]

We need to be able to turn these off during source-build.

What is the behaviour/user-experience if these are turned off? Will this be treated as a missing nuget package (and downloaded the first time it is used)?

cc @dleeapho

[wli3]

I don't understand source build well. However, if you simply remove BundledDotnetTools, verbs like dotnet-watch will not be available. It is also no longer distributed independently anymore. So the user will have no access to these verb/tools

[wli3]

The way it is inserted to CLI is very similar to template engine. How templates get source build?

[omajid]

I don't understand source build well.

A lot of technicality aside, it's about only building things from sources and never using prebuilt artifacts - such as nuget packages - during build. The goal is to get close to a reproducible build: sources go in on one side and the binaries come out of the other. Nothing is downloaded from NuGet or other package feeds during build. No binaries are allowed.

How templates get source build?

Templates are built by source-build and the resulting nuget packages from building the templates repo are provided to the cli repo to use when cli builds.

However, if you simply remove BundledDotnetTools, verbs like dotnet-watch will not be available. It is also no longer distributed independently anymore. So the user will have no access to these verb/tools

It sounds like we may want to add dotnet-ef (and other tools) to source-build then. I take it there is no chance the decision to bundle these tools can be re-evaluated?

[livarcocc]

@DamianEdwards @natemcmaster @KathleenDollard for the bundle commands in the CLI question.

I guess @natemcmaster is also a good person to talk about enabling source build for these tools.

[KathleenDollard]

@DamianEdwards Can we reconsider having a single command the user runs (which would also initiate first run and any restores)? This would need access to NuGet to download based on this gesture (I have in mind a first run script)

@livarcocc Is there a route that will not cause engineering chaos that will allow us to source build Bundled Tools?

@leecow I know you are also involved with source build issues so wanted you to be aware. Also, am I correct that we have a source build goal for 2.1.300?

Other ideas?

[natemcmaster]

dotnet-watch, along with the rest of the ASP.NET Core runtime, bundled tools, and templates, does not yet support source-build. We are not currently scheduled to finish the work for that in the 2.1.0 timeframe.

cc @muratg

[dleeapho]

We need to be really careful about what gets pulled into source-build as a result of bundling, the effort is considerable on the part of the bundled component/repo. I think the bar we held with aspnetcore makes sense for source-built distributions: the verb is available but using it causes packages to be downloaded from nuget on first time use.

[livarcocc]

There is not such mechanism today (or planned) for the CLI to go fetch nuget packages once a particular command is invoked.

If you don't have dotnet-watch bundled, typing dotnet watch will simply fail with a command not found. Also, as @natemcmaster stated above, there will be no dotnet-watch NuGet package available for us to go download and install.

This is a much bigger ask of the CLI: on demand extension of the CLI commands based on NuGet packages. It is not a feature we have designed and that I am concerned about designing for 2.1.

What could be an alternative, is for us to publish these packages to NuGet.org and tell users of build from source builds to run dotnet install to obtain these tools, but that feels like a horrible experience.

What would be the work involved in enabling build from source just for asp.net's global tools?

[natemcmaster]

What would be the work involved in enabling build from source just for asp.net's global tools?

Oh, just a few items. For starters, I know that we would need to enable build from source on the following

• SqlClient
• Azure's identity model packages
• Roslyn (they only partially support source-build)
• Newtonsoft.Json 10/11
• Most of the 48 repos referenced via aspnet/Universe
• Aspnet's build tools/scripts (or #ifdef out all the stuff we don't need in source-build)

[leecow]

I can't stress enough that any functionality which we consider to be in-box behavior needs to be included in source-build where 'source-build' means building from source with the bare minimum of bootstrap binaries needed to build and nothing required from the interwebs during the process.

[livarcocc]

@natemcmaster are you telling me that a tool like dotnet watch depends on the entirety of asp.net to build? That seems like a horrible layering problem to me.

[DamianEdwards]

Can we simply not include those bundled tools in the source build version?

[DamianEdwards]

Until such time that the rest of ASP.NET Core & EF is brought in

[leecow]

Depends, what will be busted without 'bundled' tools?

[omajid]

are you telling me that a tool like dotnet watch depends on the entirety of asp.net to build?

Not sure about dotnet-watch, but https://github.com/dotnet/cli/pull/8844 is bringing in dotnet-ef. I wouldn't be surprised if that legitimately (by design) needs most of ASP.NET/EF to build.

[livarcocc]

dotnet watch, dotnet dev-certs, dotnet ef, dotnet dev-certs and dotnet user-secrets.

I dont think we can ship without that. The question is... can we ship with these only as global tools? As in, you need to run dotnet install to acquire them?

[DamianEdwards]

I think we can require that on source build folks need to install them themselves from nuget.

[KathleenDollard]

OK, but it opens another can of worms.

Our position had been that we would have a single delivery mechanism for any particular piece of the infrastructure. I have been telling people they could be in box, or on NuGet, but not both. Either serviced via the SDK or self-serviced on NuGet.

So, @livarcocc how do we manage that? Do we have to give the up and have two delivery mechanisms for these things?

[livarcocc]

I think the statement here is that if you have the command as a native verb of the CLI, the CLI will always win, no matter what you install as a global tool.

This works here, because this version of the SDK would not have these commands native.

I think it is extremely messy and suspect that are issues we are not thinking about. We should think more about it, but right now, I can't think of anything else we can do.

[natemcmaster]

We could possibly use the same build variables we are currently using to not package the rest of ASP.NET in the SDK during source-build.

How will a user get the ASP.NET Core runtime? IIRC in 2.0, RedHat changed the default in the SDK / templates to disable the ASP.NET Core package runtime store using PublishWithAspNetCoreTargetManifest=false. Users acquired ASP.NET Core runtime from nuget.org as packages. In 2.1, this won't work. We no long distribute ASP.NET Core as a package runtime store...it's a shared framework. I'm not sure it's possible to opt-out of this. How are we expecting RedHat users to acquire the ASP.NET Core shared framework? The sharedfx isn't installed as a nuget package. Can we re-use this installer to make dotnet-watch et. al. available?

are you telling me that a tool like dotnet watch depends on the entirety of asp.net to build?

@livarcocc no, but the tools depend on enough of asp.net that you might as well just solve source-build for the entirety of Microsoft.AspNetCore.App. We don't currently have ways to build a subset of aspnetcore. It's all or nothing build system.

By the way, dotnet-watch is not the tool I'm worried about here; it does not depend on much from aspnetcore. The real problems come from bundling dotnet-ef and dotnet-aspnet-codegeneration (scaffolding). These depend on Entity Framework Core and MVC, which in turn depend on all the things I listed above. From what I was most recently told by @DamianEdwards, these are shipping in-box in 2.1.300.

Re: shipping on NuGet.org and shipping in the SDK --- I agree with @livarcocc and @KathleenDollard. Let's do one or the other. Doing both is messy.

[DamianEdwards]

So it seems we have a somewhat time critical investigation and decision to make here. I suggest we meet.

[livarcocc]

Agreed. Sent an invite.

[tmds]

How will a user get the ASP.NET Core runtime?

@natemcmaster I asked the same question here: https://github.com/dotnet/source-build/issues/83#issuecomment-363023840. I guess the plan is Microsoft.AspNetCore.App will be in a source-buildable state for 2.1? Is your question/comment about the Microsoft.AspNetCore.All shared framework?

Why does Microsoft continue to make the Microsoft.AspNetCore.All for 2.1+?

For 2.0 runtime store, by using PublishWithAspNetCoreTargetManifest, many users did not notice the runtime store was not in the source-built package.
Since now we are talking about tools the user is executing, it will directly impact the UX.

There is not such mechanism today (or planned) for the CLI to go fetch nuget packages once a particular command is invoked.

There is a request for tool command to 'fetch and execute': https://github.com/dotnet/cli/issues/5147#issuecomment-371394361

Can global tools also be used as per-project tools (for example in DotNetCliToolReference)?

[wli3]

Can global tools also be used as per-project tools (for example in DotNetCliToolReference)?

No, they are not compatible

[tmds]

@livarcocc @DamianEdwards @natemcmaster Can you please give an update on the plan for global tools? and the plan for shared frameworks (Microsoft.AspNetCore.App, Microsoft.AspNetCore.All)?

[natemcmaster]

We don't have specific plans at the moment. We're still discussing how to prioritize this against the other work we have to get done for 2.1.0 rtm. I wrote up what we would need to implement to support the shared frameworks in source build here: https://github.com/dotnet/source-build/issues/375. As I mentioned there, source-build requires refactoring almost all of our build infrastructure, plus we need to work with partner teams to get their builds to support source-build, too. We may be able to start with something smaller that just supports the global tools packages first, but we have to figure out how to slice the work.

[omajid]

https://github.com/dotnet/cli/pull/8977 is probably related to this

[livarcocc]

Our plan is to have a variable to turn off including bundled tools into the CLI. This change went in in the PR @omajid mentioned above.

Other than that, we will add special error messages when you try to dotnet and we don't find the tool indicating the user to run dotnet install -g (which will only happen in source build tools).

Issue dotnet/cli#8861 is tracking adding that error message.

Given that this issue was tracking the OFF switch for the bundle tools and that went in, I am going to go ahead and close it.