RoslynCodeTaskFactory compiles a new assembly every time it is called. Check out the cache:

This line probably returns false when comparing two hashsets:
https://github.com/dotnet/msbuild/blob/1ff34e830630ff7c4b6cb008e85ece4f263ace52/src/Tasks/RoslynCodeTaskFactory/RoslynCodeTaskFactoryTaskInfo.cs#L54
Also return 0 for GetHashCode() probably doesn't help either.
Here's what I think a fix could be:
https://github.com/mrward/msbuild/pull/1
Most helpful comment
Also return 0 for GetHashCode() probably doesn't help either.