Roslyn: System.AggregateException with innerException "A task was cancelled" trying to add reference to certain projects of a solution

[happyhebaby]

Version Used: Latest version 1.3.2

Steps to Reproduce:

1. Load a sample standalone solution file that has some of our real production solution's projects, enumerate each project trying to add a code coverage dll, most will add but a small few will fail with the exception.

I have a standalone that repro the issue. But I couldn't share the actual solution file as it's part of our production source code. I have copy the whole standalone repro code below for diagnosis.

Expected Behavior: TryApplyChanges call succeed for all projects

Actual Behavior: It will fail on a small subset of projects .

Repro code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using Microsoft.CodeAnalysis.CSharp.Syntax;
using Microsoft.CodeAnalysis.MSBuild;
using System.Reflection;

namespace ConsoleApplication4
{
class Program
{
static void Main(string[] args)
{

        string projName = args[0];
        MSBuildWorkspace workspace=null;
        try
        {
            workspace = MSBuildWorkspace.Create();
        }
        catch (Exception ex)
        {
            if (ex is System.Reflection.ReflectionTypeLoadException)
            {
                var typeLoadException = ex as ReflectionTypeLoadException;
                var loaderExceptions = typeLoadException.LoaderExceptions;
            }
        }
        // In this sample solution file that I try to load, it has 2 projects that are our real projects belong to one of our production solution file.
        // TryApplyChanges call will succeed on one project and will fail on the other.
        // When we try to run the code against the real production solution file(95 projects on it), it will fail a small few of projects while the
        // majority will succeed.
        string solutionPath = @"C:\work\ConsoleApplication5\ConsoleApplication5.sln";

        string dllPath1 = @"C:\docusign_source\SecTools\Code\CodeCoverage\CodeCoverageConfig\bin\Debug\Docusign.CodeCoverage.CodeCoverageConfig.dll";
        string dllPath2 = @"C:\docusign_source\Core\Platform\Monitoring\Monitoring.System\bin\Debug\Monitoring.System.dll";
        string dllName1 = "Docusign.CodeCoverage.CodeCoverageConfig.dll";
        string dllName2 = "Monitoring.System.dll";
        string name;
        bool containRef1, containRef2;
        bool changeApplied;
        Project project;
        Solution solution = workspace.OpenSolutionAsync(solutionPath).Result;
        Solution newSolution = solution;
        foreach (ProjectId projectId in solution.ProjectIds)
        {
            project = newSolution.GetProject(projectId);
            name = project.Name;
            if (!name.Equals(projName))
                continue;
            containRef1 = false;
            containRef2 = false;
            foreach (MetadataReference reference in project.MetadataReferences)
            {
                if (reference.Display.EndsWith(dllName1))
                {
                    containRef1 = true;
                }
                else if (reference.Display.EndsWith(dllName2))
                {
                    containRef2 = true;
                }

                if (containRef1 && containRef2)
                {
                    break;
                }
            }
            if (!containRef1)
            {
                project = project.AddMetadataReference(MetadataReference.CreateFromFile(dllPath1));
            }
            if (!containRef2)
            {
                project = project.AddMetadataReference(MetadataReference.CreateFromFile(dllPath2));
            }

            if (!(containRef1 && containRef2))
            {
                newSolution = project.Solution;
                //This call will fail for a small few projects of our production solution.
                changeApplied = workspace.TryApplyChanges(newSolution);
                if (!changeApplied)
                {
                    throw new ApplicationException(string.Format("TryApplyChanges on reference failed on {0}", name));
                }
            }
            Console.WriteLine("Project {0} done.", name);
        }
    }

}

}

[happyhebaby]

We are blocked on this for an important project. If anyone could point out a work-around code that works, that will be greatly appreciated. I tried a few hours trying alternative roslyn call to make it work but couldn't.

[happyhebaby]

I play around the code further. Correction: It doesn't just fail a few projects, instead, once it failed at a first project(It will pass a few before failing on a first project), the work space seems to be in a corrupted state and will continue to fail on every subsequent project.

[Pilchie]

@CyrusNajmabadi Was investigating a similar report on another issue, but was never able to reproduce it.

[CyrusNajmabadi]

Can you supply those dlls? I can give you my work email address if you need it. Thanks!

[happyhebaby]

@CyrusNajmabadi , are you referring to dll that I tried to add? Once you clarify I will check with company. I guess they will ok share one dll. I got 2 dlls, include any one will repro.
Here is the call stack, which I forgot to include in the initial post.
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.MSBuild.MSBuildWorkspace.GetAssemblyIdentity(ProjectId projectId, MetadataReference metadataReference)
at Microsoft.CodeAnalysis.MSBuild.MSBuildWorkspace.ApplyMetadataReferenceAdded(ProjectId projectId, MetadataReference metadataReference)
at Microsoft.CodeAnalysis.Workspace.ApplyProjectChanges(ProjectChanges projectChanges)
at Microsoft.CodeAnalysis.MSBuild.MSBuildWorkspace.ApplyProjectChanges(ProjectChanges projectChanges)
at Microsoft.CodeAnalysis.Workspace.TryApplyChanges(Solution newSolution, IProgressTracker progressTracker)
at Microsoft.CodeAnalysis.MSBuild.MSBuildWorkspace.TryApplyChanges(Solution newSolution, IProgressTracker progressTracker)
at Microsoft.CodeAnalysis.MSBuild.MSBuildWorkspace.TryApplyChanges(Solution newSolution)
at ConsoleApplication4.Program.Main(String[] args) in C:\work\ConsoleApplication4\Program.cs:line 91
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()

[CyrusNajmabadi]

Yes, i'm referring to:
string dllPath1 = @"C:\docusign_source\SecTools\Code\CodeCoverage\CodeCoverageConfig\bin\Debug\Docusign.CodeCoverage.CodeCoverageConfig.dll";
string dllPath2 = @"C:\docusign_source\Core\Platform\Monitoring\Monitoring.System\bin\Debug\Monitoring.System.dll";

I want to be able to actually repro what's going on here to track down the root issue. Thanks!

[happyhebaby]

Hi Still wait for the ok from company. Will update.

[happyhebaby]

Finally get the ok. If you could share with me your msft email? thanks. BTW I find a work around that is works around the issue except one project with performance cost.

[CyrusNajmabadi]

Yup. It's [email protected]. Thanks!

[CyrusNajmabadi]

If you can zip this all up and put it on dropbox/onedrive/etc. that would be great. Thanks!

[happyhebaby]

Hi Cyrus, send you a google drive link with one of the dlls. I can always repro with either one of the dlls. So you can remove the code for dllPath2 and just use dllPath1 to repro.I also hit a separate issue with Formatter.Format(newRoot, Formatter.Annotation, workspace) call that it doesn't do correct formatting. I will file a new bug later when I finish work on hand.
Thanks.

On Monday, September 19, 2016 1:52 PM, CyrusNajmabadi <[email protected]> wrote:

If you can zip this all up and put it on dropbox/onedrive/etc. that would be great. Thanks!—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[CyrusNajmabadi]

Hi Happy,

I don't seem to have gotten an email from you. Can you resend, or let me know hte subject to see if this got auto-spammed?

Thanks!

[CyrusNajmabadi]

Oh nevermind. I found it. It got autoflagged by our spam system. I'm going to try to repro this now. THanks!

[CyrusNajmabadi]

So this works fine for me locally. Any chance you can supply your ConsoleApplication5 where this is failing?

Thanks!

[happyhebaby]

Could you clarify what exactly you need with ConsoleApplication5? It's basically an "empty" console app solution add a real c# project as a project(one project that I run into issue when running my real code). Do you just need ConsoleApplication5.exe? Or the whole solution files? The latter is harder to get ok as it involves real product code but it you really need it I will try to ask permission.

On Friday, September 23, 2016 4:39 PM, CyrusNajmabadi <[email protected]> wrote:

So this works fine for me locally. Any chance you can supply your ConsoleApplication5 where this is failing?Thanks!—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[CyrusNajmabadi]

Well, i'm just trying to repro what you're seeing. When i run the above code, referencing the DLL you provided, and making my own empty ConsoleApplication5 project to add it to, i get no problems.

So i'm trying to figure out what could possibly be different. One thing was that maybe your ConsoleApplication5 had something interesting about it that was causing the problem :)

[happyhebaby]

I am sure it's this specific project that I included in the repro solution that causes the issue. This project in turn reference quite a few other sibling projects. I run probably 3-5 dozen projects so far this one is the only one that causes the issue. I am not sure what's special about it but I can play around more to try to find a simpler repro: I guess our company is pretty reluctant to share with third party our product code, although I am still waiting for the official word.

On Friday, September 23, 2016 6:40 PM, CyrusNajmabadi <[email protected]> wrote:

Well, i'm just trying to repro what you're seeing. When i run the above code, referencing the DLL you provided, and making my own empty ConsoleApplication5 project to add it to, i get no problems. So i'm trying to figure out what could possibly be different. One thing was that maybe your ConsoleApplication5 had something interesting about it that was causing the problem :)—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[CyrusNajmabadi]

I definitely understnad the reluctance. I'm just not able to make any headway here as there's no repro on our end of this issue.

[CyrusNajmabadi]

@happyhebaby Let me know if you can provide any more data on this. Without a repro of some sort, there is really nothing to go on here.

[happyhebaby]

Hi I checked the management, we couldn't share our product code. Just wonder if there is anything else we can provide to make the repro possible for your team?

On Tuesday, October 11, 2016 4:21 PM, CyrusNajmabadi <[email protected]> wrote:

@happyhebaby Let me know if you can provide any more data on this. Without a repro of some sort, there is really nothing to go on here.—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[fabianoliver]

Hi,

probably not very helpful as I can't provide a lot of evidence or code to reproduce, but just to point out I encountered the same issue under fairly odd circumstances.

I created a project with a few default metadata references, and added a document with some code. Later on, I tried getting the semantic model of that document.

When I ran this code in a standalone Console application, all worked fine.

When I ran the very same code hosted in an ExcelDna Excel Addin, trying to get the document's semantic model threw the above exception "A task was cancelled". When I removed all Metadata references except one to typeof(object).Assembly.Location, it suddenly worked fine even in Excel (at least until this point; later on while compiling I'd get errors due to the missing needed metadata references of course). Adding any other metadata reference, no matter for which assembly (tried some other default ones as well as non-GAC assemblies), would cause the exception to be thrown again.

I didn't find any other useful detail/stacktrace in this Exception, and can't build/debug Roslyn myself in that environment, so unfortunately I didn't find out anything more. Maybe the fact that the same code, but run in different contexts, narrows down possible explanations.

[cchamberlain]

I just experienced this issue - very tricky to debug since there is zero detail in the TaskCancelledException. To figure it out I ran the compilation in a loop with a try / catch, each time it failed with this type of exception, I removed a metadata reference until the project got down to no metadata references. After this I looked at the projects ProjectReferences and noticed the failing project had duplicate references to a couple of the same projects. I opened the csproj and verified there were duplicate references. After fixing this the issue went away. I now use the following code to give me a better exception when this issue occurs:

var project = Solution.GetProject(id);
var dedupedProjectReferences = project.ProjectReferences.Select(x => x.ProjectId).Distinct().ToList();
if (dedupedProjectReferences.Count != project.ProjectReferences.Count()) {
    throw new Exception($"Project {id} cannot be compiled because it contains duplicate project references: {string.Join("|", project.ProjectReferences.Select(x => x.ProjectId))}");
}

Seems a little crazy to me that nothing does any validation that there is no duplicate project references (VS / MSBuild / Roslyn).

[zaccharles]

@cchamberlain You just saved me from pulling my hair out. I spent ages trying to work out what was wrong with this project and in the end it had duplicate keys.

Now I have another TaskCanceledException with no details as a result of a call to SymbolFinder.FindImplementationsAsync. Fantastic...

[cchamberlain]

@zaccharles I'm considering setting up a OS project just to validate the basics that Visual Studio / msbuild should validate. Very easy to end up with a duplicate reference or worse, a misaligned packages.config / csproj after a messy merge.

Hope you get your issue figured out and interested in the results if you do.

[Chetic]

@zaccharles I am in the exact same situation as you. I've figured that AggregateExceptions with inner TaskCancelledExceptions from running the GetSemanticModelAsync task means duplicate DOM elements (project references) in .csproj files. Clearly Roslyn needs better error handling here. In any case, when I get the same exception from running the FindReferencesAsync task I am at a loss. The project does not have any duplicate DOM elements, and it only happens when analyzing certain projects. This is critical to my application and I will put in the work if necessary.

What's more is FindReferencesAsync seems to prevent debugging even with "Just My Code" disabled and debug symbols available. I am unable to share the code base my analyzer runs against that is causing this issue. How can I debug this or provide you with more information?