Sdk: Tool dependecies versioning issues

[moozzyk]

When you build a tool that uses NetCore.App version that you don't have installed the tool does not work at all. Example:
I compiled the tool against NetCore.App 23911 and 23923 (
"Microsoft.NETCore.App": {
"type": "platform",
"version": "1.0.0-rc2-23911"
}
}
) and it did not work because:
C:Program FilesdotnetsharedMicrosoft.NETCore.App1.0.0-rc2-23924

When I compiled against 23924 the tool worked.

Since projects using tools don't control what versions of dependency a tool is using it defeats the purpose of a shared framework since in the end you will need a version of a framework that corresponds to the dependency of the tool.

Also the message is quite confusing as well:

Could not load host policy library []

[moozzyk]

/cc @NTaylorMullen

[moozzyk]

/cc @natemcmaster

[TheRealPiotrP]

@gkhanna79 @schellap what are your thoughts here?

I discussed this with @schellap last week. The problem goes away for a while once we ship 1.0.0, but until then it means that using a tool requires installing the shared framework the tool was built for. We can make this a bit better by allowing roll-forward for pre-release versions. However, this is in a fairly sensitive portion of the host.

Is this something we can pursue once the host is stabilized?

[moozzyk]

@piotrpMSFT - can you elaborate a bit more on how this is fixed with 1.0.0? What happens once 1.0.1 ships? Won't the 1.0.0 tools stop working if only 1.0.1 sdk installed?
Given that dotnet follows semver I think it is reasonable to roll tools forward (and possibly backward?) within the minor or maybe even major version.
Also, not sure if this is reasonable but maybe it should also be possible to create self contained tools that would bring the runtime with themselves if the one they need is not already installed.

[gkhanna79]

For pre-release scenarios like this, there is no auto-roll forward by design as there are no compat guarantees between even daily builds. This issue is specific to such intermediary scenarios where the underlying platform is still being developed.

For actual production scenarios, the tool would be built against a shipping version of the framework and would just work. It should be kept in mind that any tool, targeting .NET Core, is a .NET Core app and like any such app, would need to have its dependencies present for it to work. If the dependency is on .NET Core 1.0.0 for the tool and the onyl FX present is 1.2.0, there is not going to be an auto-roll forward for the same as well.

The question here would be how .NET Core apps get installed and declare their (production scenario) dependencies. I think this is a dev-scenario issue and I will have @blackdwarf think about enabling the intended developer experiences.

[NTaylorMullen]

Looping in a few others. /cc @DamianEdwards @Eilon

[Eilon]

We need to try our best to make sure that NuGet packages such as Razor Tooling don't break when a newer .NET CLI is installed, even though it could have work.

Is there a reason to be restrictive on which versions are compatible, when there's probably a decent change it'll just work?

[Eilon]

It's worth noting that ASP.NET went to great lengths to ensure that N+1 runtimes worked with N tooling as long as possible, even with pre-releases (it often _eventually_ breaks, but only when there's no reasonable alternative).

[gkhanna79]

Is there a reason to be restrictive on which versions are compatible, when there's probably a decent change it'll just work?

This constraint is only applicable to pre-release versions for the reason called out above.

It's worth noting that ASP.NET went to great lengths to ensure that N+1 runtimes worked with N tooling as long as possible, even with pre-releases

This is less of a host issue but rather overall policy/scenario question - in the past discussions, the concern was always about breaking across pre-releases.

CC @weshaggard

[moozzyk]

It seems that this rule makes it also virtually impossible for early adopters to use the tools. They typically install the latest cli. To use a tool they would have to know the version of the Microsoft.NETCore.App version the tool depends on (probably means they need to crack open the package) and install the shared runtime that corresponds to this version (and it's not easy to install not the latest).

[Eilon]

I just wonder if such a policy is a bit _too_ restrictive for pre-releases. It certainly might break, but normally we leave that to people on the bleeding edge to experience, and it's part of what they expect (that's the "blood" part of "bleeding edge").

[muratg]

What's the plan for RC2? How do the tools work in RC2?

[gkhanna79]

@davidfowl mentioned on Friday that he is working on a tool to address this. Have you chatted with him on this?

[blackdwarf]

@gkhanna79 what kind of a tool would this be? This seems to me to be an issue that would also require guidance for people writing the tools, no?

[brthor]

This issue is larger than tools, it is true for portable apps in general.

In production we will have some form of roll forward in place (as I understand it) but with prerelease bits, portable apps can only work against the sharedfx that they target. The fact that project tools are now portable apps is the cause here.

I'd like to get thoughts on whether we be looking at solutions for tools in specific, or portable apps in general?

[blackdwarf]

Hm, that's true. But with portable apps, isn't the statement that you've made a very deliberate call on what you depend on?

[muratg]

We will either need a guidance doc for tool writers, or enable roll-forward now. Otherwise experience will be really bad. RC2 users will need to have matching CLI versions everywhere (command line, VS, CI, ...)

[gkhanna79]

But with portable apps, isn't the statement that you've made a very deliberate call on what you depend on?

This is correct.

@davidfowl Can you please share the details of the tool you were referring to on Friday?

RC2 users will need to have matching CLI versions everywhere (command line, VS, CI, ...)

@muratg Can you please elaborate? Are you suggesting that developers using RC2 final bits will update their dependencies to depend upon a pre-release build? Is this going to happen in a production usage of the tool?

[blackdwarf]

@muratg for the guidance, I will be writing that regardless. But @gkhanna79 has a good point above: for RC2, you should only depend on the version of the NETCore.App that is RC2, right? Why would you depend on before or after if you want to run on RC2?

[muratg]

@blackdwarf We'll be rolling newer versions between RC2 and RTM. Don't we want to keep RC2 users be able to update the CLI so that they can use newer bits?

I feel like the current design is too restrictive. Ideally risk of breaking due to roll-forward is more acceptable than forcing folks to use exact versions.

[gkhanna79]

Don't we want to keep RC2 users be able to update the CLI so that they can use newer bits?

Are you referring to releasing new builds of SharedFX and targeting them? If so, then yes, they will be published but the developer would need to take an explicit action to update their component to target it. Rolling forward to from a production (RC2 final) to pre-release may not be a good idea as pre-release builds could have bugs/breaks, which violate invariants associated with production scenario.

@muratg This issue has been marked as blocking. Can you please elaborate what production scenario it blocks?

[muratg]

@gkhanna79 Right now it's really hard for us to have a matching set of ASP.NET Core tools and VS tooling (WTE). They have to have the exact build of CLI. Which means folks will have a lot of trouble when they run tools with newer runtime (till we ship RC2) or newer tooling (as we get daily WTE builds.) It's not technically "blocking" the release, but it's making day to day work really hard for a lot of engineers.

Re risk on roll-forward from RC2-final to RTM... I disagree. See Eilon's comment above (https://github.com/dotnet/cli/issues/2071#issuecomment-202511888) which I agree with.

[TheRealPiotrP]

@muratg

Right now it's really hard for us to have a matching set of ASP.NET Core tools and VS tooling (WTE).

If this is true then we have an implementation bug. Installing a newer CLI should not remove an older shared framework. This means that as folks upgrade their CLI to keep up with tooling they are getting more versions but scenarios that worked yesterday should continue working today. The only case in which someone becomes broken is when they have not yet installed the Shared Framework to which the tool is coupled. Are you seeing a different behavior?

[muratg]

@piotrpMSFT We build our tools (dotnet-publish-iis, dotnet-watch, ...) in our CI. To be able to run those tools locally, VS/WTE on the local box needs to have a matching version of the CLI with the one on the CI.

[TheRealPiotrP]

Only a matching version of the Shared Framework. If that's broken, please let us know ASAP.

[Eilon]

The problem is that a machine will have a _tool_ that was _built_ with "yesterday's" CLI/shared framework, but try to run against "today's" CLI/shared framework.

And that's the problem here: tools are built only rarely (e.g. only when there's a bug fix), but apps likely want to use the most recent CLI/shared framework. The machine won't have the "old" CLI/shared framework installed.

How can an app use a slightly older tool with a newer CLI/shared framework?

[TheRealPiotrP]

These are disjoint scenarios, right? The tool is restored and executed independently of the app. If the right shared framework is on the box the tool will run against its version while the app will run against its version. The CLI itself can run against a third version, for completeness sake.

Tools are just portable applications so this is actually exercising the mainline scenario of enabling disparate apps to run against the frameworks they were built for.

Since tools are built rarely, this is even less of an issue since the shared frameworks they depend on will change slowly.

[Eilon]

I think the problem is that "the right shared framework" isn't on the box, and there's also no way to know which version to install, even if that was the right fix.

E.g. my project uses the tool dotnet-frob, which has its most recent build that was built against CLI/shared framework 1234. But my project happens to use CLI/shared framework 6789. So I need to have CLI 1234 _and_ 6789 installed. And now imagine my project is using 5 different tools, each built against a different shared framework. I now need 5 + 1 = 6 CLIs and shared frameworks installed.

I think some simple rules could fix this:

• A prerelease of x.y.z should roll forward to any newer pre-release of x.y.z.
• A release of x.y.z should roll forward to any newer patch release of x.y.

If a command got rolled forward, there should be a warning indicating that a roll-forward happened, which would alert a user that there might be some "fun" behavior.

[NTaylorMullen]

This means that as folks upgrade their CLI to keep up with tooling they are getting more versions but scenarios that worked yesterday should continue working today.

This is assuming folks continually do in-place upgrades and more importantly have been along for the ride from the start. Without auto-rollforward support here I'd imagine you'd need to have every CLI install every shared framework that was previously released to ensure compatibility for first-time users. What about CI's? Do they install the whole shared framework world every time?

The only case in which someone becomes broken is when they have not yet installed the Shared Framework to which the tool is coupled.

Having a single line entry in project.json potentially explode your Visual Studio experience seems troublesome. Users will often follow tutorials online, do every one of these tutorials now need to say "users must use Visual Studio X" or "users must install dotnet shared framework version Y" when in reality their current shared framework is compatible? I'd understand if a tool said it required 1.2.3 and a user only had 1.2.2 on their box but not vice versa.

---

When this issue was created the shared framework seemed to be new for every dotnet build. I just compared 002200 and 002202 and they both seem to share the same shared framework. How frequently do you intend to rev the shared framework?

---

Do we really expect to make breaking shared framework changes in non-major version changes? Why not support auto-rollforward for non-major version changes? It's been painful with it's current limitation.

[muratg]

@gkhanna79 any updates on this? Let us know when this is ready. :)

[schellap]

Will be ready before 1 PM.

[moozzyk]

@piotrpMSFT

If this is true then we have an implementation bug. Installing a newer CLI should not remove an older shared framework. This means that as folks upgrade their CLI to keep up with tooling they are getting more versions but scenarios that worked yesterday should continue working today. The only case in which someone becomes broken is when they have not yet installed the Shared Framework to which the tool is coupled.

I think I pointed it out in one of the other issues I opened but if I have a tool that was built against an older version of the NetCore.App how do I even a) find and b) donwnload the version of the runtime I need?
With regards to:

a) the version of dotnet cli does not match the version of the CoreApp package (e.g. the version is 1.0.0-beta-002252 how should I know if this installs the version of shared runtime that matches my tool built against 1.0.0-rc2-23928)

b) if I somehow know that I need 1.0.0-beta-002237 how - as a user - do I install it? because the links typically point to the .latest. and without reverse engineering build/obtain scripts it's hard to guess what the link to a specific version should look like.

[TheRealPiotrP]

@moozzyk this is somewhat moot since @schellap is implementing rollforward on pre-release at the moment. Your points are well taken and are being addressed as part of getting ahead of this issue for 1.0->1.1, etc. More to come.

[moozzyk]

@piotrpMSFT - Agreed. Rollforward makes it irrelevant for now.

[gkhanna79]

This got fixed with https://github.com/dotnet/cli/pull/2316. Closing the issue.