Sdk: An item with the same key has already been added. Key: TMP

[binki]

Steps to reproduce

mkdir hwapp
cd hwapp
dotnet new
dotnet restore
dotnet run

Expected behavior

I expecdted to see the “Hello World!” output.

Actual behavior

C:\Users\ohnob\OneDrive\Documents\Visual Studio 2015\Projects\hwapp>dotnet run
Project hwapp (.NETCoreApp,Version=v1.0) will be compiled because expected outputs are missing
Compiling hwapp for .NETCoreApp,Version=v1.0
An item with the same key has already been added. Key: TMP
C:\Program Files\dotnet\dotnet.exe compile-csc @C:\Users\ohnob\OneDrive\Documents\Visual Studio 2015\Projects\hwapp\obj\Debug\netcoreapp1.0\dotnet-compile.rsp returned Exit Code 1

Compilation failed.
    0 Warning(s)
    0 Error(s)

Time elapsed 00:00:00.1135244

Environment data

dotnet --info output:

C:\Users\ohnob\OneDrive\Documents\Visual Studio 2015\Projects\hwapp>dotnet --info
.NET Command Line Tools (1.0.0-preview2-003121)

Product Information:
 Version:            1.0.0-preview2-003121
 Commit SHA-1 hash:  1e9d529bc5

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.14393
 OS Platform: Windows
 RID:         win10-x64

C:\Users\ohnob\OneDrive\Documents\Visual Studio 2015\Projects\hwapp>set | grep -ie TMP
TEMP=C:\msys64\tmp
TMP=C:\msys64\tmp
tmp=C:\Users\ohnob\AppData\Local\Temp

[binki]

Current workaround:

C:\Users\ohnob\OneDrive\Documents\Visual Studio 2015\Projects\hwapp>SET TMP=

C:\Users\ohnob\OneDrive\Documents\Visual Studio 2015\Projects\hwapp>SET TEMP=

C:\Users\ohnob\OneDrive\Documents\Visual Studio 2015\Projects\hwapp>dotnet run
Project hwapp (.NETCoreApp,Version=v1.0) will be compiled because expected outputs are missing
Compiling hwapp for .NETCoreApp,Version=v1.0

Compilation succeeded.
    0 Warning(s)
    0 Error(s)

Time elapsed 00:00:01.3746388


Hello World!

Looks like the CLI needs to deal with merging environment variables like the real .net framework and every other Windows program. This is the first time I’ve seen this sort of behavior in a program.

[ghost]

:+1: always a good idea to namespace/prefix all your environment variables with app name, e.g. DOTNETCLI_TMP.

[binki]

@jasonwilliams2000K I haven't looked at the source yet but I figured this might just be CoreFX's System.Environment implementation failing when reading in the environment from the OS.

[TheRealPiotrP]

Can you try this out with preview3 bits? I've not been able to repro with PJ CLI nor the new bits...

[binki]

@piotrpMSFT I don’t see preview3 here. Where do I get it?

[TheRealPiotrP]

Official bits will be available soon. Until then, grab the latest bits from https://github.com/dotnet/cli

[binki]

When I try to download the last Windows x64 build, I cannot get past the login page.

[binki]

OK, I tried with 1.0.0-preview4-004079 and it still happens. But I can only figure out how to have both tmp and TMP show up in SET if I’m using MSYS2’s bash shell. But, as I’ve said before, I have never seen it cause an issue with another program and Windows’s CreateProcess() apparently lets you do this even though most of the tools that ship with Windows (e.g., cmd) don’t let you.

ohnob@DESKTOP-A4G2C0J MSYS ~/AppData/Local/Temp/hwapp
$ /c/Program\ Files/dotnet/dotnet.exe --version
1.0.0-preview4-004079

ohnob@DESKTOP-A4G2C0J MSYS ~/AppData/Local/Temp/hwapp
$ /c/Program\ Files/dotnet/dotnet.exe run

Welcome to .NET Core!
---------------------
Learn more about .NET Core @ https://aka.ms/dotnet-docs. Use dotnet --help to see available commands or go to https://aka.ms/dotnet-cli-docs.

Telemetry
--------------
The .NET Core tools collect usage data in order to improve your experience. The data is anonymous and does not include commandline arguments. The data is collected by Microsoft and shared with the community.
You can opt out of telemetry by setting a DOTNET_CLI_TELEMETRY_OPTOUT environment variable to 1 using your favorite shell.
You can read more about .NET Core tools telemetry @ https://aka.ms/dotnet-cli-telemetry.

Configuring...
-------------------
A command is running to initially populate your local package cache, to improve restore speed and enable offline access. This command will take up to a minute to complete and will only happen once.
Decompressing 100% 3726 ms
Expanding 100% 35901 ms
An item with the same key has already been added. Key: TMP

ohnob@DESKTOP-A4G2C0J MSYS ~/AppData/Local/Temp/hwapp
$ echo $?
1

[TheRealPiotrP]

dotnet/sdk#6464 is a similar issue.

Use this workaround:
set the env var DOTNET_SKIP_FIRST_TIME_EXPERIENCE=true. This will skip the first-run experience.

When fixing dotnet/sdk#6464 we will make a first-run failure non-blocking.

[binki]

Even when setting that envvar I still get the error:

ohnob@DESKTOP-A4G2C0J MSYS ~/AppData/Local/Temp/hwapp2
$ DOTNET_SKIP_FIRST_TIME_EXPERIENCE=true /c/Program\ Files/dotnet/dotnet.exe new
Created new C# project in C:\Users\ohnob\AppData\Local\Temp\hwapp2.

ohnob@DESKTOP-A4G2C0J MSYS ~/AppData/Local/Temp/hwapp2
$ DOTNET_SKIP_FIRST_TIME_EXPERIENCE=true /c/Program\ Files/dotnet/dotnet.exe restore
An item with the same key has already been added. Key: TEMP

ohnob@DESKTOP-A4G2C0J MSYS ~/AppData/Local/Temp/hwapp2
$ DOTNET_SKIP_FIRST_TIME_EXPERIENCE=true /c/Program\ Files/dotnet/dotnet.exe run
An item with the same key has already been added. Key: tmp

ohnob@DESKTOP-A4G2C0J MSYS ~/AppData/Local/Temp/hwapp2
$ DOTNET_SKIP_FIRST_TIME_EXPERIENCE=true /c/Program\ Files/dotnet/dotnet.exe --info
.NET Command Line Tools (1.0.0-preview4-004079)

Product Information:
 Version:            1.0.0-preview4-004079
 Commit SHA-1 hash:  43dfa6b8ba

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.14965
 OS Platform: Windows
 RID:         win10-x64

[yupferris]

I'm also seeing the original error, and interestingly enough I'm also using msys2. It works fine with straight cmd, but ideally that shouldn't make a difference.

[yupferris]

Setting DOTNET_SKIP_FIRST_TIME_EXPERIENCE=true does not make a difference.

[MirKml]

Yes, same here. With cmd.exe shell it works, with msys2 shell it doesn't. Error message is: An item with the same key has already been added. Key: tmp
and content of tmp env variable is

$ echo $tmp
C:\Users\<MyUserName>\AppData\Local\Temp

[TheRealPiotrP]

potentially related to dotnet/sdk#4479

[inosik]

Still happens with the latest SDK from the website. I'm on Windows 7 and use Bash which comes with Git. Here is the stack trace:

Unhandled Exception: System.ArgumentException: An item with the same key has already been added. Key: HOME
   at System.ThrowHelper.ThrowAddingDuplicateWithKeyArgumentException(Object key)
   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   at System.Diagnostics.ProcessStartInfo.get_Environment()
   at Microsoft.DotNet.Cli.ForwardingApp.Execute()
   at Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingApp.Execute()
   at Microsoft.DotNet.Tools.Restore.RestoreCommand.<>c__DisplayClass0_0.<Run>b__0()
   at Microsoft.DotNet.Cli.CommandLine.CommandLineApplication.Execute(String[] args)
   at Microsoft.DotNet.Tools.Restore.RestoreCommand.Run(String[] args)
   at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, ITelemetry telemetryClient)
   at Microsoft.DotNet.Cli.Program.Main(String[] args)

This makes dotnet.exe pretty much unusable from Bash. Note that this happens even when I start cmd.exe from Bash and then run dotnet restore.

[dsplaisted]

@inosik I wasn't able to repro this using the Git Bash. I tried setting the TMP and TEMP environment variables and then running dotnet restore. Can you provide repro steps from scratch for this?

[binki]

@dsplaisted The error is that .net core assumes that the environment will contain no variables that differ only in case. I’m guessing the developers did this because that’s how things work on Windows. And somehow it only cares about when that happens to TMP and/or TEMP:

ohnob@DESKTOP-A4G2C0J MINGW64 /projfun
$ TMP=a tmp=b dotnet restore

Unhandled Exception: System.ArgumentException: An item with the same key has already been added. Key: tmp
   at System.ThrowHelper.ThrowAddingDuplicateWithKeyArgumentException(Object key)
   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   at System.Diagnostics.ProcessStartInfo.get_Environment()
   at Microsoft.DotNet.Cli.ForwardingApp.Execute()
   at Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingApp.Execute()
   at Microsoft.DotNet.Tools.Restore.RestoreCommand.<>c__DisplayClass0_0.<Run>b__0()
   at Microsoft.DotNet.Cli.CommandLine.CommandLineApplication.Execute(String[] args)
   at Microsoft.DotNet.Tools.Restore.RestoreCommand.Run(String[] args)
   at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, ITelemetry telemetryClient)
   at Microsoft.DotNet.Cli.Program.Main(String[] args)

I’m using the build I downloaded from this project’s README:

ohnob@DESKTOP-A4G2C0J MINGW64 /projfun
$ ls -l "$(where dotnet)"
-rwxr-xr-x 1 ohnob 197609 127488 Mar 23 15:52 'C:\Program Files\dotnet\dotnet.exe'*

ohnob@DESKTOP-A4G2C0J MINGW64 /projfun
$ dotnet --version
1.0.0

I would say that it is a very good thing that it is crashing right now as if calling IDictionary.Add() with the same key twice instead of IDictionary.Item.set() with the same key twice. If it did the latter, you would get subtle bugs. Consider the following environment:

ohnob@DESKTOP-A4G2C0J MINGW64 /projfun
$ set | grep -ie ^TMP=
TMP=/c/tmp1
tmp=/c/tmp2/but/nobody/names/envvars/they/try/to/pass/to/programs/with/lowercase

Now when the user runs dotnet restore, which value should it use? If the authors of dotnet CLI had used IDictionary.Item.set() rather than IDictionary.Add(), the process wouldn’t crash and, maybe, it would use tmp instead of TMP.

So, which variable should the program respect?

• A Windows user might say “How in the world do you have two variables? That’s illegal!”
• A *nix user might say: “TMP, of course. tmp is completely irrelevant and probably a local in the shell script that was only “leaked” to my program’s environment unintentionally. Why did you pass -i to grep? Is this supposed to be a trick question?”

I’m not sure how Git Bash and MSYS2 process invocation interacts with Windows processes. But it appears that the environ passed through is unfiltered enough that Windows actually does permit environment variables only differing by case through. The best fix is probably to detect if you’re running on *nix and require TMP. If you detect you’re running on Windows, do something like this:

1. find all case insensitive matches to TMP.
2. If the count is 1, just use the one found (to support Windows users who miscapitalize their environment variables).
3. If there is an exact case match for TMP, use that and ignore others.
4. Throw an exception with semantics similar to AmbiguousMatchException (except that’s for assembly binding so I don’t think that’d be the right one for this exact case).

[binki]

Oh, never mind, it doesn’t care if it’s TMP or some other variable:

ohnob@DESKTOP-A4G2C0J MINGW64 /projfun
$ A=1 a=1 dotnet restore

Unhandled Exception: System.ArgumentException: An item with the same key has already been added. Key: a
   at System.ThrowHelper.ThrowAddingDuplicateWithKeyArgumentException(Object key)
   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   at System.Diagnostics.ProcessStartInfo.get_Environment()
   at Microsoft.DotNet.Cli.ForwardingApp.Execute()
   at Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingApp.Execute()
   at Microsoft.DotNet.Tools.Restore.RestoreCommand.<>c__DisplayClass0_0.<Run>b__0()
   at Microsoft.DotNet.Cli.CommandLine.CommandLineApplication.Execute(String[] args)
   at Microsoft.DotNet.Tools.Restore.RestoreCommand.Run(String[] args)
   at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, ITelemetry telemetryClient)
   at Microsoft.DotNet.Cli.Program.Main(String[] args)

[inosik]

@dsplaisted you need to add two environment variables which differ in case, like FOO and foo, then run dotnet restore, or any other command, which shells out to another program. This should trigger the exception:

$ FOO=foo foo=foo dotnet restore

The problem is the new implementation of ProcessStartInfo, which behaves differently to the one in the .NET Framework. This is actually already being tracked at dotnet/corefx#13146.

[inosik]

It's probably due to some enterprisey settings on my machine, but I happen to have HOME and home when I run in Git Bash. HOME being set to /h/, and home to H:\.

I assume @binki has a similar setup, judging by his machine name DESKTOP-A4G2C0J :smile:

[binki]

It looks like for me the lower-case variant of tmp is coming from https://github.com/Alexpux/MSYS2-packages/blob/045bd64f5199f91c3143075448c7eb0f84d7f0e5/filesystem/profile#L72

It looks like cygwin, which MSYS2 forked from, already encountered this sort of issue but with .NET Framework 2/3.5 (apparently 4 no longer crashes) and Cygwin worked around the issue by avoiding introducing case insensitively equal environment variables anymore. So MSYS2 is probably doing a nonstandard thing by introducing the same variable with alternate casing. But it is a common enough thing in shell scripting when using a POSIX sh, so coreFX should be fixed to not crash regardless.

[berdon]

@dsplaisted you need to add two environment variables which differ in case, like FOO and foo, then run dotnet restore, or any other command, which shells out to another program. This should trigger the exception:

$ FOO=foo foo=foo dotnet restore
The problem is the new implementation of ProcessStartInfo, which behaves differently to the one in the .NET Framework. This is actually already being tracked at dotnet/corefx#13146.

I see the exact same exception given this circumstance: "An item with the same key has already been added: tmp"

[dluc]

Issue still open with latest version. Any update?

# dotnet

Microsoft .NET Core Shared Framework Host

  Version  : 1.1.0
  Build    : 928f77c4bc3f49d892459992fb6e1d5542cb5e86

# dotnet --info
.NET Command Line Tools (1.0.4)

Product Information:
 Version:            1.0.4
 Commit SHA-1 hash:  af1e6684fd

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.15063
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\1.0.4

[DaveOMac]

Having the Unhandled Exception: System.ArgumentException: An item with the same key has already been added. as well with the latest NPM (6.1.0) as a task runner.

For us, we run dotnet run via an NPM script. NPM 6.1.0 for whatever reason adds three path environment variables; [path, Path, PATH].

When we call npm run dotnet which is:

  "scripts": {
    "dotnet": "SET ASPNETCORE_ENVIRONMENT=Development&& dotnet watch run"
  },

dotnet run fails with:

Unhandled Exception: System.ArgumentException: An item with the same key has already been added. Key: Path
   at System.ThrowHelper.ThrowAddingDuplicateWithKeyArgumentException(Object key)
   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   at System.Diagnostics.ProcessStartInfo.get_Environment()
   at Microsoft.DotNet.Cli.Utils.Command..ctor(CommandSpec commandSpec)
   at Microsoft.DotNet.Cli.Utils.Command.Create(ICommandResolverPolicy commandResolverPolicy, String commandName, IEnumerable`1 args, NuGetFramework framework, String configuration, String outputPath, String applicationName)
   at Microsoft.DotNet.Cli.Utils.Command.Create(String commandName, IEnumerable`1 args, NuGetFramework framework, String configuration, String outputPath, String applicationName)
   at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, ITelemetry telemetryClient)
   at Microsoft.DotNet.Cli.Program.Main(String[] args)

We're running Windows 10 and dotnet --info:

.NET Command Line Tools (1.1.9)

Product Information:
 Version:            1.1.9
 Commit SHA-1 hash:  77f0268904

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.17134
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\1.1.9

Microsoft .NET Core Shared Framework Host

  Version  : 2.0.7
  Build    : 2d61d0b043915bc948ebf98836fefe9ba942be11

Would be nice if it handled merging the environment variables together.

[dsplaisted]

@DaveOMac It looks like the issue was fixed in .NET Core 2.0. Can you try this with the latest .NET Core SDK and see if you still hit the issue?

[DaveOMac]

@dsplaisted Yup using SDK 2.1.301 we don't see this error, boots up normally. Thanks for the info!

As 1.1 is in LTS do you think the same fix could be applied?

[dsplaisted]

@DaveOMac Generally I would recommend using the latest version of the .NET Core SDK (currently 2.1.301) even if you your app targets an LTS version of .NET Core (ie 1.1). Would that work for you?

[DaveOMac]

@dsplaisted Yup that could work. We were under the assumption that the versions should be aligned (i.e. we're targeting LTS, so until 2.1 is adopted, runtime framework version 1.1.8 and I had assumed we should fix the SDK to 1.1.9). If there's no consequences of using the latest SDK then we can certainly switch to it.

Thanks again for the info!

[dsplaisted]

Thanks for confirming that this is fixed in the latest SDK, @DaveOMac. I'm going to go ahead and close this issue.