Sdk: Build error "An item with the same key has already been added."

[cwe1ss]

"dotnet build" throws an error when project.lock.json contains assemblies with the same name in different packages. While this seems logical on one side, it is quite hard to debug these issues.

My biggest concern is that this is a common problem in Microsoft packages as well. Packages like "Microsoft.Net.Http" and "System.Net.Http" contain assemblies with the same name. AFAIK people should move to the newer "System.Net.Http" package but I'm afraid that's not always under our control. Some 3rd party libraries might still use "Microsoft.Net.Http" and others might already use "System.Net.Http".

Steps to reproduce

These are two sample projects that fail:

{
  "version": "1.0.0-*",
  "compilationOptions": {
    "warningsAsErrors": true
  },

  "dependencies": {
    "Microsoft.Bcl.Immutable": "1.0.34",
    "Microsoft.AspNet.Mvc": "6.0.0-rc1-final"
  },

  "frameworks": {
    "net46": { }
  }
}

Error: An item with the same key has already been added. Key: (System.Collections.Immutable, neutral, b03f5f7f11d50a3a)

{
  "version": "1.0.0-*",
  "compilationOptions": {
    "warningsAsErrors": true
  },

  "dependencies": {
    "Hyak.Common": "1.1.0",
    "System.Net.Http.Formatting.Extension": "5.2.3"
  },

  "frameworks": {
    "net46": { }
  }
}

Error: An item with the same key has already been added. Key: (System.Net.Http.Extensions, neutral, b03f5f7f11d50a3a)

Expected behavior

I'm not sure if these are exactly the same libraries and a binding redirect would solve the problem. I've seen that there are already some other issues (eg dotnet/sdk#4805 ) on that topic.

If it can't be solved with a binding redirect, maybe it can be resolved by renaming the libraries?

If this can't be fixed, how should we proceed in such cases? If the vendor of the library doesn't change his dependencies we might end up in a blocking situation.

Environment data

dotnet --version output:

.NET Command Line Tools (1.0.0-beta-001718)

Product Information:
 Version:     1.0.0-beta-001718
 Commit Sha:  5b0afe7880

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.10586
 OS Platform: Windows
 Runtime Id:  win10-x86

[xatabhk]

Same issue. I have not been able to get dotnet.exe working since 001718 I think.

[davidfowl]

@xatabhk This is a nuget bug https://github.com/NuGet/NuGet.Client/commit/f489a21dbb40706c4b8ac436e5462828feb9d1d5. We'll pull a new version ASAP.

That's not related to the original issue though it is the same exception.

[flagbug]

@davidfowl Is there a workaround for now?

[MariovanZeist]

@davidfowl Besides having the same issue @xatabhk has, I also can't build using visual studio.
I am able to build some libraries using dotnet build, but building the same library from inside visual studio gives a plethora of errors mostly of the format

The type 'IList<>' is defined in an assembly that is not referenced. You must add a reference to assembly 'System.Runtime, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.

Where IList<> can be object or other basic classes. Might this be the same issue?

I have added an issue report to the aspnet/mvc repository

[fearthecowboy]

Any progress on this? This is blocking me in a rather bad way at the moment.

[flagbug]

Just downloaded the latest version of the CLI and this seems to be fixed!

[pakrym]

Should be fixed by https://github.com/dotnet/cli/pull/2137

[hovsepm]

I start hitting this on VS Update 3.

D:\GitHub\azure-sdk-for-net\tools>dotnet --info
.NET Command Line Tools (1.0.0-preview1-002702)

Product Information:
 Version:     1.0.0-preview1-002702
 Commit Sha:  6cde21225e

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.10586
 OS Platform: Windows
 RID:         win10-x64

D:\GitHub\azure-sdk-for-net\tools>NuGet.exe update -self
Checking for updates from https://www.nuget.org/api/v2/.
Currently running NuGet.exe 3.5.0-beta2-1456.
NuGet.exe is up to date.

How to repro:

1. clone https://github.com/hovsepm/azure-sdk-for-net/
2. switch to AutoRest branch
3. open src\ResourceManagement\Authorization\Authorization.sln
4. build

Actual:
Error An item with the same key has already been added. Key: Microsoft.Azure.Test.HttpRecorder TestFramework C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\DotNet\Microsoft.DotNet.Common.Targets 241

Expected:
Build successful.

[dragonfly91]

Is this fixed? I'm hitting the same issue as @hovsepm mentioned in the above post.

[witchdrash]

Not fixed

[andyjansson]

I'm having this issue as well. My current workaround has been to delete project.lock.json and rebuild the project.

[Serjster]

The only solution that has worked for me to remove these annoying errors is to remove solution project references from the project in the error, as in go to Solution Explorer, right click on References in the project specified in the error, click on Add Reference, then uncheck all checkboxes in the Projects/Solution. Then 'dotnet restore' in the 'Package Manager Console'. After that's done, go back and readd those references the same way you removed them. Rebuild the solution. That seems to fix the errors, but is super annoying.

[derFunk]

I'm seeing the same still.

dotnet --info
.NET Command Line Tools (1.0.0-preview2-003131)

Product Information:
 Version:            1.0.0-preview2-003131
 Commit SHA-1 hash:  635cf40e58

Runtime Environment:
 OS Name:     Windows
 OS Version:  6.3.9600
 OS Platform: Windows
 RID:         win81-x64

Repro:

• Clone https://github.com/aws/aws-sdk-net.git locally
• Open AWSSDK.CoreCLR.sln in VS Pro 2015 Update 3
• Let VS/NuGet fetch all dependencies / rebuild
• Find these build errors:

Code    Description Project File    Line    Suppression State
An item with the same key has already been added. Key: Core AWSSDK.AWSMarketplaceCommerceAnalytics.CoreCLR  C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\DotNet\Microsoft.DotNet.Common.Targets  262 
An item with the same key has already been added. Key: Core AWSSDK.AWSMarketplaceMetering.CoreCLR   C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\DotNet\Microsoft.DotNet.Common.Targets  262 
An item with the same key has already been added. Key: Core AWSSDK.AWSSupport.CoreCLR   C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\DotNet\Microsoft.DotNet.Common.Targets  262 
An item with the same key has already been added. Key: Core AWSSDK.ApplicationDiscoveryService.CoreCLR  C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\DotNet\Microsoft.DotNet.Common.Targets  262 
An item with the same key has already been added. Key: Core AWSSDK.CertificateManager.CoreCLR   C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\DotNet\Microsoft.DotNet.Common.Targets  262 
An item with the same key has already been added. Key: APIGateway   IntegrationTests    C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\DotNet\Microsoft.DotNet.Common.Targets  262 

[Serjster]

I found another solution that seems to fix this problem, do a:

dotnet restore --no-cache

[derFunk]

@Serjster Exactly! I was just about to write the same.

• The project.lock.json generated with dotnet restore works. This lets the project compile in VS, emitting no "same key has already been added" errors.
• The project.lock.json generated with Visual Studio 2015 Update 3 (e.g. after deleting the project.lock.json file manually) does not work. This will emit the "same key has already been added." errors.

For the AWS .NET SDK the projects.lock.json generated by dotnet restore is 140kB smaller than the one generated by Visual Studio.

I've attached both files here, maybe somebody can quickly spot the reason why this happens.

• project.lock_generated_by_dotnet_restore-works.json.txt
• project.lock_generated_by_vs2015upd3-doesntwork.json.txt

Anybody knows how VS is creating the lock file differently than the cli? Shouldn't it use the same cli command?

/Edit: Visual Studio is issuing dotnet restore someRestoreDgFile.dg.
In my case, it's "C:\Program Files\dotnet\dotnet.exe" restore "e:\aws-sdk-net\sdk\.vs\restore.dg.
Exactly this last file parameter is the reason for dotnet restore creating the bigger project.lock.json file.
Without the dg file parameter, the project.lock.json is smaller and presumably without duplicates.

The .dg file is a reference mapping file for project-to-project references and it gets created by Visual Studio.

[CodingSamurai]

Having the same exact issue but in my case the duplicate reference is to an Core Class Library project in the solution used by the Core Web API project (both projects are targeting the full framework).

[louislewis2]

I am also having the same issue, also have all the latest updates installed.

[gimlichael]

Why is this bug closed? Like louislewis2, I still experience this bug. Only workaround working was the dotnet restore --no-cache suggested by Serjster. But that is a workaround .. please fix this annoying bug!

[Serjster]

one thing you can try doing is to clear out your nuget cache... this seems to have solved my problem... at least for now.

1. Download nuget.exe: https://github.com/NuGet/Home/releases
2. nuget.exe locals all -clear from the folder where you downloaded above (I put mine in C:\Program Files (x86)\NuGet\Visual Studio 2015 )
3. Run dotnet restore in your VS Package Manager Console

[ByronAP]

I have this problem regularly, my solution seems to be opening the root folder of the solution and doing a find for *.lock.json and deleting them all then opening the solution.

[osblue13]

I have started to encounter this bug numerous times everyday. My workround is same as @ByronAP , delete the *.lock.json file and re-open the solution.

[TheRealPiotrP]

.lock.json means you're on some pre-preview3 version of CLI. the lock file was renamed to project.assets.json starting in preview3. Please move forward to a newer build.

[ByronAP]

why is M$ not promoting or pushing preview3 anywhere? REF: https://www.microsoft.com/net/core#windowsvs2015 which specs to preview2 and VS update3 which specs to preview2 as well.

Even the latest release is Preview 2-1-003177 I only see 1 ref to a 3.* build in releases

[andreimcristof]

have the exact same problems described here in detail with VS 2015 community edition. I did the dotnet restore --no-cache multiple times, I did the nuget locals clear command as well, I still have the error:

An item with the same key has already been added. Key: Microsoft.NETCore.App

Also tried as suggested above to delete the lock.json file, still the error persists. This is very frustrating. Any other suggestions please?

[ByronAP]

You must have not have deleted all of the lock files (as it is the root src of the issue, deleting the files is the solution).

Make sure Visual Studio is closed then browse to the solution directory and do a search for *.lock.json and delete ALL the files that come up (may take a minute depending on sln size). Now when you open the project Visual Studio will rebuild all those files.

[andreimcristof]

@ByronAP thank you very much!

[andrewslavin]

Same issue here. Whenever I add a nuget package to the solution, or add a reference from one project to another, multiple projects stop building with this error. The fix is to delete project.lock.json in those projects, but it's annoying to have to keep doing it all the time. Using VS 2015 update3 and tools 1.0.0-preview2-final.