Roslyn: InvalidOperationException "Cannot enqueue data after PromiseNotToEnqueue." if an analyzer calls GetDiagnostics in the RegisterCompilationAction callback

[dpaoliello]

Version Used: 1.3.2 (also repros in 2.0.0-rc2)

Steps to Reproduce:
Full repro code available at https://github.com/dpaoliello/PromiseNotToEnqueueRepro

This requires two threads:

Thread 1 (Main C# Compiler thread):

1. Calls CommonCompiler.RunCore.
2. Which calls CSharpCompilation.GetDeclarationDiagnostics.
3. Which calls ForceComplete on symbols recursively.
4. Which calls a method that calls SymbolCompletionState.NotePartCompleted before calling CSharpCompilation.SymbolDeclaredEvent (e.g. SourceMemberFieldSymbol.FixedSize).
5. After calling NotePartCompleted the thread is suspended (note that SymbolDeclaredEvent has not yet been called).

Thread 2 (Thread for analyzer):

1. Calls CSharpCompilation.GetDiagnostics.
2. Which calls ForceComplete on symbols recursively.
3. Since Thread 1 has marked part of the compilation complete, continues to complete the compilation.
   4. GetDiagnostics then calls Compilation.EnsureCompilationEventQueueCompleted which marks the EventQueue as completed and promises not to enqueue any other events.

Thread 1 then continues, and calls SymbolDeclaredEvent which attempts to enqueue the event, but finds that the queue has been completed, thus throws the InvalidOperationException.

There are two possible ways I can think of to fix this:

1. A careful review of each call to NotePartCompleted to ensure that all events are queued before completing the part.
2. Ensure that only one thread is walking the ForceComplete chain at a time - perhaps the first thread that does this can atomically set a task that other threads can then wait on.

[gafter]

@dpaoliello You are awesome.

[gafter]

Resolved (fixed) in #16740 .