Sdk: Local Tools Early Preview Documentation

[wli3]

Local Tools Early Preview Documentation

The feature is shipped in 3.0.100 SDK. We are working to convert this documentation to an official one.

Motivation

Enable high cohesion between tools and corresponding code repositories. The repository maintainer could check-in a list of dotnet tools with certain version. And other contributor can use these tools within the repository with certain gestures. Local tools use the same package as global tools.

Walk through

Install the latest dotnet SDK "checkin build"

1. You can find the installer at here under the _Master (3.0.x Runtime)_ column.
2. Most of the existing dotnet tools package(global tools) are targeting runtime netcoreapp2.1. You need to install it in order to run the existing command. You can find it at here under _Run apps - Runtime_ column.

Create tool manifest file

Run dotnet new tool-manifest command. It will create a tool manifest file _dotnet-tools.json_ under the directory _.config_.

Restore command

In order to use the command specified in tool manifest file. You need to run command dotnet tool restore and you should see the following success message:

Tool 'dotnetsay' (version '2.1.4') was restored. Available commands: dotnetsay


Restore was successful.

Run the command

Run local tool would require prefix dotnet. So it will not be ambiguous most of the time. And when there is a global tools with the pattern "dotnet-*" CLI will find the tool with the following rule

| global tools command invoke | local tools command invoke |
| ------ | ------ |
| dotnetsay | dotnet dotnetsay |
| dotnet say (aka dotnet-say) | dotnet say |
| dotnetsay and dotnet say both exist | dotnet dotnetsay and dotnet dotnet-say |

You could also call it by dotnet tool run dotnetsay hi. "dotnet tool run" is not ambiguous.

If you see the following error, please install netcoreapp2.1 runtime. At https://www.microsoft.com/net/download/dotnet-core/2.1. Since _dotnetsay_ and most of the global tools are targeting 2.1 runtime while preview dotnet SDK only contains preview netcoreapp3.0 runtime.

It was not possible to find any compatible framework version
The specified framework 'Microsoft.NETCore.App', version '2.1.0' was not found.
  - Check application dependencies and target a framework version installed at:
      C:\Users\name\Downloads\dotnet-sdk-latest-win-x64-localtools\
  - Installing .NET Core prerequisites might help resolve this problem:
      https://go.microsoft.com/fwlink/?LinkID=798306&clcid=0x409
  - The .NET Core framework and SDK can be installed from:
      https://aka.ms/dotnet-download
  - The following versions are installed:
      3.0.0-preview1-27017-04 at [C:\Users\name\dotnet-sdk-latest-win-x64\shared\Microsoft.NETCore.App]

Manage local tools

Install new local tool

Run dotnet tool install <PACKAGE_ID> (dotnet tool install dotnetsay) to install new command. It will add the tool entry to your manifest file. And at the same time it will restore the tool so you can invoke it right away.

The message will show which manifest file it added to

You can invoke the tool from this directory using the following command: dotnet tool run dotnetsay
Tool 'dotnetsay' (version '2.1.4') was successfully installed. Entry is added to the manifest file /Users/name/tools/sub/dotnet-tools.json

Uninstall local tool

Run dotnet tool uninstall <PACKAGE_ID>

Update local tool

Run dotnet tool update <PACKAGE_ID>. If the new package version is higher. SDK will find the first manifest file that contains the package id and update it. If there is no such package id in any manifest file (that is in the _scope_), SDK will add a new entry to the closest manifest file.

List local tool

Run dotnet tool list to list all available tools in the current directory. It will also list the origin manifest of the tool. It can be used to diagnose unexpected tools in the current directory

$ dotnet tool list
Package Id            Version      Commands             Manifest
----------------------------------------------------------------------------------------------------------------------
dotnet-dbinfo         1.3.1        dotnet-dbinfo        /Users/name/tools/sub/dotnet-tools.json
dotnet-depends        0.2.0        dotnet-depends       /Users/name/tools/sub/dotnet-tools.json
amazon.ecs.tools      3.0.0        dotnet-ecs           /Users/name/tools/sub/dotnet-tools.json
dotnet-encrypto       1.0.5        dotnet-encrypto      /Users/name/tools/sub/dotnet-tools.json
t-rex                 1.0.53       t-rex                /Users/name/tools/.config/dotnet-tools.json

Feature Detail

The _scope_ of the tool manifest file

If there is a tool manifest file or a tool manifest file under folder _.config_ in the parent/current directory of the current directory. The current directory has the access to the tools specified in the tool manifest file. For example. If there is a tool manifest with a tool _dotnetsay_ in the root of the repository, the user could run dotnet tool run dotnetsay hi in any sub directory of the repository.

isRoot flag

The flag in the manifest file means "stop looking up parent directories for manifest file when the first one is found". This entry in manifest file is required.

Invoke local tools

Global tools use $PATH and shell to resolve command via shim. However we could not use the same mechanism due to the limitation of the operation system shell.

Example of multiple tools

{
    "version": "1",
    "isRoot": true,
    "tools": {
        "t-rex": {
            "version": "1.0.53",
            "commands": ["t-rex"],
        },
        "dotnetsay": {
            "version": "2.1.4",
            "commands": ["dotnetsay"]
        }
    }
}

Requirement of commands

We require listing out the command in the tool package. Note many dotnet tool packages do not have matching package id and the command.

Term

Local tool scope: the working directory that has the access to the tool specified in the dotnet-tools.json stored in current or parent directory.

Tool manifest file: dotnet-tools.json

Repository maintainer: A dotnet tool consumer. Programmers have the write access to the repository and decide which dotnet tools should be used in this repository.

Repository contributor: A dotnet tool consumer. Programmers have the read access to the repository and use dotnet tools decided by repository maintainer.

Local tools restore: Restore all tools that is possible to be accessed by current directory and generate cache.

Resolver cache: Technical, not customer facing. Cache to speed up the local tool resolution.

[wli3]

I will close this issue after preview is done

[jmezach]

Am I correct in assuming that this would allow different versions of a tool within different repositories? For example, in the npm world we can have different versions of say typescript within different repositories. What happens if a global tool is also available? Does the local one basically override the globally installed one, provided that you'd run it by using dotnet ?

[wli3]

Am I correct in assuming that this would allow different versions of a tool within different repositories? For example, in the npm world we can have different versions of say typescript within different repositories.

Yes. That is one of the problem we want to solve.

What happens if a global tool is also available? Does the local one basically override the globally installed one, provided that you'd run it by using dotnet ?

_Outdated_

Run local tool would require prefix dotnet. So it will not be ambiguous most of the time. And when there is a global tools with the pattern "dotnet-*" CLI will find the tool with the following rule

| global tools command invoke | local tools command invoke |
| ------ | ------ |
| dotnetsay | dotnet dotnetsay |
| dotnet say (aka dotnet-say) | dotnet say |
| dotnetsay and dotnet say both exist | dotnet dotnetsay and dotnet dotnet-say |

[wli3]

Update:
To invoke local tools you need to use dotnet tool run dotnetsay hi.

[wli3]

Added _Manage local tools_ session

[amis92]

After latest change (a move back from dotnet tool run fulltoolname to dotnet fulltoolname), shouldn't that:

global tools command invoke | local tools command invoke
-- | --
dotnetsay | dotnet dotnetsay
dotnet say (aka dotnet-say) | dotnet say
dotnetsay and dotnet say both exist | dotnet dotnetsay and dotnet dotnet-say

Be rather

global tools command invoke | local tools command invoke
-- | --
dotnetsay | dotnet dotnetsay
dotnet say (aka dotnet-say) | dotnet dotnet-say
dotnetsay and dotnet say both exist | dotnet dotnetsay and dotnet dotnet-say

For some reason this single case "infers" the dotnet part. I think it shouldn't as well?

[wli3]

@amis92 sorry that was a mistake in the document. We still use dotnet tool run dotnetsay hi today. And we are still discussing if we need a short hand version or not. Corrected in the issue

[GerjanOnline]

I am not liking the verbosity and unintuitiveness of dotnet tool run dotnetsay

Would it be possible to "auto detect" which tool should be run? So I think checking from working directory?

For example when I am inside my repository (which has dotnet-tools.json) it should pick the one listed. When I am outside the repository, it should pick the global one.

D:dotnetsay (global one, ie version 1.1)
D:\myrepodotnetsay (local one, ie version 1.0)

But probably you need to do some weird magic/tricks, right? Because you only can have one in your PATH, so you need to create shims and all that kind of stuff.

[Thraka]

What about compromising on that with dotnet tool dotnetsay? Basically, any command after dotnet tool that isn't recognized is parsed as if dotnet tool run was used.

[kzu]

having local + global automatic fallbacks with shorthand names would be awesome.

[wli3]

bring back dotnet <Toolname> to run a local tool after https://github.com/dotnet/cli/pull/10980

[amis92]

Seeing as there is https://github.com/dotnet/announcements/issues/107, and the "documentation" is currently this issue, can we have some info about how to author these local tools? I'm quite interested in testing them in preview.

[wli3]

@amis92 authoring is the same as global tools which is available already https://docs.microsoft.com/en-us/dotnet/core/tools/global-tools-how-to-create

[poke]

Just out of curiosity: Where are local tools installed to? Are they installed into a local folder relative to the dotnet-tools.json? Or are they still installed into a global location so projects that depend on the same version of the same tool version can reuse the binary? What is the uninstall story in that case? Is there a way to list all local tool installations and uninstall them properly?

[wli3]

added "Update local tool"

[wli3]

Note the following are current implementation detail, that is subject to change.

Where are local tools installed to?

The asset are installed to nuget global package folder.

uninstall story

It will be removed from the manifest file. No change in nuget global package folder since it is maintained by a different set of command(detail in previous link).

Is there a way to list all local tool installations and uninstall them properly?

If you mean "install" by downloading asset. There is no easy way to do so, however, you could browse your nuget global package folder if there is no override of that folder. At the same time, you could clean that folder using nuget command. However, that will also clean your other nuget packages like newtonsoft.json. (and cause a longer review next time a project uses it)

[tebeco]

@wli3
Update is a great idea and would be really effective with a full update option :

dotnet tool update --local --all

asking developper to manually list + update tool one by one is kinda harsh

I'm super lazy and if would add a nightly build on my CI just for that one

so every night the tooling is updated

[wli3]

@tebeco there is an existing issue https://github.com/dotnet/cli/issues/10860 and I'd like to add this feature. However, this is unlikely to make in 3.0.100 release due to this is not a trivial feature.

[wli3]

@tebeco and @restrellasosa I moved the other issue to https://github.com/dotnet/cli/issues/11386

I hope this thread only focus on bigger design of local tools

[hknielsen]

I like that its part of the dotnet tool, and local tools. But why not part of the csproj? We got away with the nuget file, but now adding a new file for this feels weird? 🙂

[jnm2]

@hknielsen To me it's a relief that it's not part of a csproj. I make use of this in a PowerShell build script where there is no one csproj that is more particularly connected to the build script than any other csproj.

[tebeco]

@hknielsen
dotnet tool should not have any relation with csproj.
dotnet tool are CLI adding it to a csproj would force you to restore all the dependency tree for your business code.
This means :

• the restore time will blow up
• that transitive of the CLI will impact your code (or fail the restore)
• transitive of your code can make the resolution fail

there are scenarios where you want to use a CLI on a repo without restoring the nuget from any

a great example is powershell core
if you take a look at the Dockerfile for the SDK 3.0-previewX you will see that it's now doing a
dotnet tool install to install powershell

there are also scenario where you need the tooling in order to be able to restore.
this is how fake and paket works for fsharp and this is pretty neat

The good news about the file you want to get rid of is that

• the version of the tools are under source control
• if you have multiple repo / multi version it is clear
• if a new teamate arrives, all info are together
• you don't need to use powershell to remember the custom --tool-path (this was what you had to use since 2.1 because global tool is insane when working with a team or multi repo

[JeremyMorton]

Does this hook into dotnet restore or dotnet build, at all?

We have a DotnetCliTool that is a code generator that we run before compiling C#, dotnet restore gets the package and the DotnetCliTool, so someone building a project that uses the generator just needs to run dotnet build and it gets what is needed.

If I understand this, switching to a local tool means that every developer needs to run dotnet tool restore every time they don't have the version of our generator that's checked into the .configdotnet-tools.json file in our repo. They can't just have build do the right thing, correct?

[wli3]

If it is a generator that is part of and always part of the build, writing a Task and deliver it via nuget would be more appropriate.

Local tools operate on a directory level while "dotnet restore" and "dotnet build" operate on project level. Although you could add "dotnet tool restore" as part of your build by configuring your csproj.

[JeremyMorton]

We also have a Task in an assembly in a NuGet package for some build number logic. The big drawback of that is that the dll is held open by VS, so restoring when the project is opened in the IDE fails because it can't overwrite the assembly with a new version.

We get around this by having an init1.ps in the package (which only runs nowadays if you manually open the Package Manager Console in VS ☹️) to rename the in-use assembly. This is obviously a bit ugly, and only works if someone manually opens the Package Manager Console in VS.

# Copy task binary file
$taskBinaryFileName = "CreateLocalPackages.dll"
$oldTaskBinaryPath = $taskBinaryFileName + ".old"
$olderTaskBinaryPath = $taskBinaryFileName + ".older"
$destTaskBinaryPath = Join-Paths $solutionDir.Directory $taskBinaryFileName
$packageTaskBinaryPath = Join-Paths $installPath "lib" "netstandard2.0" $taskBinaryFileName

if (Test-Path $olderTaskBinaryPath)
{
  Write-Host "Deleting $olderTaskBinaryPath"
  Remove-Item $olderTaskBinaryPath -ErrorAction:SilentlyContinue
  if (Test-Path $olderTaskBinaryPath)
  {
    Write-Host "Can't delete $olderTaskBinaryPath as it is in-use"
    Write-Host "Deleting $oldTaskBinaryPath"
    Remove-Item $oldTaskBinaryPath -ErrorAction:SilentlyContinue
  }
  else
  {
    Write-Host "Successfully deleted $olderTaskBinaryPath"
    if (Test-Path $oldTaskBinaryPath)
    {
      Write-Host "Moving $oldTaskBinaryPath to $olderTaskBinaryPath"
      Move-Item $oldTaskBinaryPath -Destination $olderTaskBinaryPath
    }
  }
}
else
{
  if (Test-Path $oldTaskBinaryPath)
  {
    Write-Host "Moving $oldTaskBinaryPath to $olderTaskBinaryPath"
    Move-Item $oldTaskBinaryPath -Destination $olderTaskBinaryPath
  }
}

if (Test-Path $destTaskBinaryPath)
{
  Write-Host "Moving $destTaskBinaryPath to $oldTaskBinaryPath"
  Move-Item $destTaskBinaryPath -Destination $oldTaskBinaryPath
}

Write-Host "Copying $packageTaskBinaryPath to $destTaskBinaryPath"
Copy-Item $packageTaskBinaryPath -Destination $destTaskBinaryPath

[AndersMalmgren]

Hi, we are migrating our enterprise application from 2.2 to 3.0. I found this post telling us to move to local tools if we use dot net CLI

https://github.com/dotnet/announcements/issues/107

So I found my way here. This does not seem like an replacement for donetcli tool at all? With dotnet cli we made a nuget package. And could reference it from project like

<DotNetCliToolReference Include="dotnet.cqsgen" Version="1.0.22" />

And run it like

  <Target Name="BuildContracts" AfterTargets="Build">
    <Exec Command="dotnet cqsgen $(OutputPath)IC.Eko.Core.Contracts.dll cqs.contracts.ts IC.Eko.Core.Contracts.Commands.Command;IC.Eko.Core.Contracts.Queries.Query" />
  </Target>

When a developer pulled latest version of our project from git and build the tool would invoke and do its code generation magic. I dont see how this can replace that?

Thanks

[wli3]

@AndersMalmgren as the previous comments stated. If the tool is only used during build, could you try to make it a msbuild task?

At the same time, you could restore a local tool and run it with the similar fashion.

[wli3]

We used local tools for our tests. Here is a sample. Although you can start from "dotnet tool restore" instead of "dotnet new tool-manifest" https://github.com/dotnet/sdk/blob/062d9a036ea335833e472d3b3c17174ba62c5c19/src/Tests/Directory.Build.targets#L37

[AndersMalmgren]

@wli3 I wasnt aware you could restore msbuild tasks using nuget?
I solved it like this for now,

  <Target Name="BuildContracts" AfterTargets="Build">
    <Exec Command="dotnet tool restore" />
    <Exec Command="dotnet SignalR.EventAggregatorProxy.AspNetCore.GlobalTool $(OutputPath)SignalR.EventAggregatorProxy.Demo.AspNetCore.dll $(MSBuildProjectDirectory)\wwwroot\js\events.js SignalR.EventAggregatorProxy.Demo.AspNetCore.EventTypeFinder" /> 
  </Target>

though it feels a bit like a step back from CLI tool that you are forced todo a restore even though its fast becasue of nuget cache

[wli3]

@AndersMalmgren we acknowledge this short coming. And we do want to remove this step once some nuget feature is ready

[AndersMalmgren]

@wli3 Good to hear. I have found a showstopper sadly. We have private feeds. These interfere with dotnet tool restore. Even though the tool is published to nuget.org it tries to use the private feeds and fails

[wli3]

@AndersMalmgren --ignore-failed-sources should solve it

[AndersMalmgren]

@wli3 Thanks, that works. But I cant help to think the old CLI solution was more straight forward :D

[NickDarvey]

If the tool is only used during build, could you try to make it a msbuild task

@wli3, if I understand correctly, an msbuild task would be a fine replacement for simple things but as soon as you need to use a dependency the advice I've read is _don't build an msbuild task_.

I don't quite understand the separation that's trying to be achieved here, (which may just be because I haven't been in the space long enough) but it seems to me using local tools as part of a build would be a common use case that should be designed for (with something like DotNetCliToolReference) rather than requiring a workaround.

[stijnherreman]

Is this feature still in preview? Or is it the documentation that's in preview?

[wli3]

@stijnherreman only the documentation is in preview. It is shipped in 3.0.100 SDK.

We are working to convert this documentation to an official one

[wli3]

The official doc is here https://docs.microsoft.com/en-us/dotnet/core/tools/global-tools#install-a-local-tool