Docfx: Error extracting metadata for project : An item with the same key has already been added

[darylrobbins]

DocFX Version Used: 2.33

Template used: default

Steps to Reproduce:

Paths have been anonymized to protect the innocent.

1. Execute docfx metadata C:\Users\someuser\Code\SomeProject\Module\SomeLibrary.Shared\SomeLibrary.Shared.vbproj -f

Expected Behavior:
Successfully generates metadata

Actual Behavior:
Metadata generation fails with the following error:

[18-03-19 07:58:56.263]Info:[ExtractMetadata]Generating metadata for each project... 
[18-03-19 07:59:04.842]Error:Error extracting metadata for C:\Users\someuser\Code\SomeProject\Module\SomeLibrary.Shared\SomeLibrary.Shared.vbproj: Error extracting metadata for project "C:\Users\someuser\Code\SomeProject\Module\SomeLibrary.Shared\SomeLibrary.Shared.vbproj": An item with the same key has already been added.

Notes:
This is the first time trying to run docfx on our project. I saw the same error on other VB or C# projects. I tried earlier versions and saw similar/additional errors. I also see the same error when using a solution that contains the projects.

[vicancy]

Hi @darylrobbins Could you share a minimum project that can repro the issue to us?

[michaelp]

I have the same issue with version 2.35.4
_Is it possible to increase verbosity and get the docfx log ?_

[vicancy]

Hi @michaelp Yes good suggestion. Meanwhile, could you provide some minimum project for us to repro the issue? Is there duplicate references in the project?

[JulienM28]

Hello,
I just got the same issue with 2.3.5 version.

Impossible to build the documentation.

Projects are using .Net Framework 4.5.2.

[michaelp]

The following snippet of code is the relevant entry point to the source of the issue. Please see my additional comments for the details. (docfx\src\Microsoft.DocAsCode.Metadata.ManagedReference\ExtractMetadataWorker.cs)

private static async Task<ConcurrentDictionary<string, AbstractCompilation>> GetProjectCompilationAsync(ConcurrentDictionary<string, AbstractProject> projectCache)
        {
            var compilations = new ConcurrentDictionary<string, AbstractCompilation>();
            var sb = new StringBuilder();
            foreach (var project in projectCache)
            {
                try
                {
                    var compilation = await project.Value.GetCompilationAsync();
                    compilations.TryAdd(project.Key, compilation);
                }
                catch (Exception e)
                {
                    if (sb.Length > 0)
                    {
                        sb.AppendLine();
                    }
                   // 
                   // MP _ WTF! You are loosing you call trace
                    //
                    sb.Append($"Error extracting metadata for project \"{project.Key}\": {e.Message}");
                }
            }
            if (sb.Length > 0)
            {
                throw new ExtractMetadataException(sb.ToString());
            }
            return compilations;
        }
// -----------------
// which goes into 
//
//(docfx\src\Microsoft.DocAsCode.Metadata.ManagedReference.Roslyn\ExtractMetadata\Project\RoslynProject.cs)
public override async Task<AbstractCompilation> GetCompilationAsync()
        {
            var c = await _project.GetCompilationAsync();  // <----- exception is here
            return new RoslynCompilation(c);
        }

[michaelp]

FYI @vicancy

   at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   at Microsoft.CodeAnalysis.SolutionState.CompilationTracker.<FinalizeCompilationAsync>d__30.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.CodeAnalysis.SolutionState.CompilationTracker.<BuildCompilationInfoFromScratchAsync>d__24.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.CodeAnalysis.SolutionState.CompilationTracker.<GetOrBuildCompilationInfoAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.CodeAnalysis.SolutionState.CompilationTracker.<GetCompilationSlowAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.CodeAnalysis.SolutionState.CompilationTracker.<GetMetadataReferenceAsync>d__32.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.CodeAnalysis.SolutionState.CompilationTracker.<FinalizeCompilationAsync>d__30.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.CodeAnalysis.SolutionState.CompilationTracker.<BuildCompilationInfoFromScratchAsync>d__24.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.CodeAnalysis.SolutionState.CompilationTracker.<GetOrBuildCompilationInfoAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.CodeAnalysis.SolutionState.CompilationTracker.<GetCompilationSlowAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.CodeAnalysis.SolutionState.CompilationTracker.<GetMetadataReferenceAsync>d__32.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.CodeAnalysis.SolutionState.CompilationTracker.<FinalizeCompilationAsync>d__30.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.CodeAnalysis.SolutionState.CompilationTracker.<BuildCompilationInfoFromScratchAsync>d__24.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.CodeAnalysis.SolutionState.CompilationTracker.<GetOrBuildCompilationInfoAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.CodeAnalysis.SolutionState.CompilationTracker.<GetCompilationSlowAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.DocAsCode.Metadata.ManagedReference.RoslynProject.<GetCompilationAsync>d__12.MoveNext() in C:\Users\michael.pavlovsky\Projects\docfx\src\Microsoft.DocAsCode.Metadata.ManagedReference.Roslyn\ExtractMetadata\Project\RoslynProject.cs:line 48
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.DocAsCode.Metadata.ManagedReference.ExtractMetadataWorker.<GetProjectCompilationAsync>d__15.MoveNext() in C:\Users\michael.pavlovsky\Projects\docfx\src\Microsoft.DocAsCode.Metadata.ManagedReference\ExtractMetadataWorker.cs:line 432

[michaelp]

 private async Task<SolutionState.CompilationTracker.CompilationInfo> FinalizeCompilationAsync(SolutionState solution, Compilation compilation, CancellationToken cancellationToken)
      {
        SolutionState.CompilationTracker.CompilationInfo compilationInfo;
        try
        {
          bool hasSuccessfullyLoaded = this.ProjectState.HasAllInformation;
          List<MetadataReference> newReferences = new List<MetadataReference>();
          Dictionary<MetadataReference, ProjectId> metadataReferenceToProjectId = new Dictionary<MetadataReference, ProjectId>();
          newReferences.AddRange((IEnumerable<MetadataReference>) this.ProjectState.MetadataReferences);
          foreach (ProjectReference projectReference1 in (IEnumerable<ProjectReference>) this.ProjectState.ProjectReferences)
          {
            ProjectReference projectReference = projectReference1;
            ProjectState projectState = solution.GetProjectState(projectReference.ProjectId);
            if (projectState != null)
            {
              if (projectState.IsSubmission)
              {
                compilation = compilation.WithScriptCompilationInfo(compilation.ScriptCompilationInfo.WithPreviousScriptCompilation(await solution.GetCompilationAsync(projectReference.ProjectId, cancellationToken).ConfigureAwait(false)));
              }
              else
              {
                MetadataReference key = await solution.GetMetadataReferenceAsync(projectReference, this.ProjectState, cancellationToken).ConfigureAwait(false);
                if (key != null)
                {
                  newReferences.Add(key);
                  metadataReferenceToProjectId.Add(key, projectReference.ProjectId);
                }
                else
                  hasSuccessfullyLoaded = false;
              }
            }
            projectReference = (ProjectReference) null;
          }
          compilation = this.UpdateCompilationWithNewReferencesAndRecordAssemblySymbols(compilation, newReferences, metadataReferenceToProjectId);
          this.WriteState((SolutionState.CompilationTracker.State) new SolutionState.CompilationTracker.FinalState(SolutionState.CompilationTracker.State.CreateValueSource(compilation, solution.Services), hasSuccessfullyLoaded), solution);
          compilationInfo = new SolutionState.CompilationTracker.CompilationInfo(compilation, hasSuccessfullyLoaded);
        }
        catch (Exception ex) when (Microsoft.CodeAnalysis.ErrorReporting.FatalError.ReportUnlessCanceled(ex))
        {
          throw ExceptionUtilities.Unreachable;
        }
        return compilationInfo;
      }

[rbbb]

To reproduce the issue, add a subtle double link to a reference project.

For example, go into a .csproj and copy paste a ProjectReference three times.

Or go into a .csproj and add a reference to the project and the dll.

Or go into a .csproj and add a second reference to a project through an equivalent but different path, with an incorrect name and guid.

Roslyn needs clearer error reporting of those cases.

Hth, raph (sgcib/far)

[vicancy]

@superyyrrzz Looks like a duplicate ProjectReference issue

[ElanHasson]

A fix for this was to use this script to dedupe the entries in project files.

https://github.com/rodrigoff/powershell/blob/a341380b857b882b0d01ec3601445967244ababa/csproj-utils/RemoveCsProjDuplicates.ps1

[skini82]

Didn't worked for me :( Any other idea?
I'm on docfx 2.42.3.0

[superyyrrzz]

@skini82 We have a test case to generate document for a sample csproj with duplicate project reference, and it works fine. So now it is hard to reproduce this issue. Could you share your project?