Runtime: Enable compilation support in System.CodeDom

[stephentoub]

System.CodeDom in corefx provides the object model for describing code as well as code generators for both C# and VB. But attempts to compile source code (which in desktop works by shelling out to csc.exe/vbc.exe) isn't enabled and throws PlatformNotSupportedException. This should be fairly straightforward to build on top of Roslyn's APIs; I expect the main tricky part will be around whether we can take a dependency from this library to Roslyn. If we can't, we could explore indirect dependencies, e.g. via reflection.

[Priya91]

Most of the funtionality in this library is provided through Roslyn, and roslyn is constantly evolving in that space. We don't foresee innovating in codedom, and don't think it's worth the investment here, to light these up using roslyn..

[weshaggard]

I would be very reluctant to take the dependency directly in the System.CodeDom library itself because there is a decent chance that we may be required to pull that library into the shared framework because of other .NET Framework API compat issues and it would be a shame to pull in all of Roslyn because of that.

My suggestion would be to try and build a library on top of System.CodeDom that hooks up Roslyn and CodeDom together so we can always keep that optional. I also want to try and avoid corefx having a dependency on Roslyn packages as that will cause more build cycles, so this library would need to be outside of corefx.

We can also consider a reflection style approach but I suspect that maybe difficult because it isn't going to be one single API we need to call via reflection.

[terrajobst]

I like @stephentoub's suggestion and suggested something similar to @weshaggard.

@Priya91

We don't foresee innovating in codedom, and don't think it's worth the investment here, to light these up using roslyn.

It's not about innovating, it's about how much of the existing code we can make work as-is. Asking folks to change their code to compile with Roslyn isn't straight forward. Asking folks to install the package because we do the work via reflection is reasonable. We could even be more helpful by making sure we detect that case and throw an actionable error message.

[drvink]

@terrajobst

It's not about innovating, it's about how much of the existing code we can make work as-is.

I have a user in fsprojects/FSharp.Compiler.CodeDom#24 who is likely facing the problem that has always presented the greatest difficulty for people at the intersection of CodeDom and F#[1], which is the issue of having to register the provider with machine.config or web.config in order for it to be of any use at all--if all they want is to assemble an AST and get F# source from it, there are better tools than the F# CodeDom provider[2], and if they want a way of programmatically compiling F# source, there are also better ways to do it than via CodeDom.

Anyway, it looks like it's possible to register third-party providers with ASP.NET ~for .NET Core~ (see the web.config transform in Microsoft.CodeDom.Providers.DotNetCompilerPlatform), but it looks like machine.config doesn't even exist at all anymore in .NET Core. This means that the only way the F# provider can be used is by directly referencing the assembly, which is completely useless for users of the provider.

TL;DR: Is there any way to make the provider work "system-wide", as CodeDom used to allow via machine.config, or will the only useful purpose for third-party CodeDom providers under .NET Core be solely for use via ASP.NET? (edit: apparently ASP.NET Core doesn't use CodeDom)

[1] Or really, _any_ language other than C# or VB.NET--even Microsoft has been guilty in a number of places of hard-coding the CodeDom "choices" to only these two, despite allowing for system-wide registration of new providers and for programs to trivially enable users to ask for any registered provider... :)
[2] As the joke goes, CodeDom is an API lets you write C# from any language.

[davidfowl]

ASP.NET Core doesn’t use code Dom, on top of that razor is c# only in asp.net core.

[drvink]

@davidfowl

What is the purpose of web.config for a NuGet-ized CodeDom provider package, then? (Sorry, I don't do web stuff at all; I simply assumed that it was related to ASP.NET)

[davidfowl]

It’s for ASP.NET (not core)

[drvink]

@davidfowl So this is entirely about providing NuGet packages for .NET, and .NET Core/Standard are completely unrelated?

[davidfowl]

Correct, if the CodeDom implementation for C# and VB are stubbed out using roslyn APIs then it will work for things that do use codedom (albeit with a large first time performance hit because using roslyn without ngen is awful). It just so happens that ASP.NET Core does not use the codedom for anything so there's no benefit there.

The API should be ported though, some other systems do rely on it.

[drvink]

@davidfowl So, if I were to provide a package similar to Microsoft.CodeDom.Providers.DotNetCompilerPlatform for the F# provider, it would help people who wanted to use NuGet/Paket/etc., but otherwise, it would have nothing to do with .NET Core?

[drvink]

@davidfowl I simply am 100% ignorant of this segment of the .NET world, so it would help if you can explain the situation here--if the F# provider is only going to be useful from this point on in the context of a compatibility shim for "old .NET", that's not a problem for me, but I need to alert people who have depended on it up until now that it won't work with .NET Core.

[davidfowl]

Ok let me summarize. There are 2 different concerns being discussed here, .NET Core and ASP.NET Core, they are not the same thing.

• CodeDom is made up of 2 pieces
  
  
  
  1. An object model (AST) for generating code in a language agnostic way (which is dubious in itself)
  
  
  2. A way to compile that code that produce and assembly
  
  
• Today on .NET Core System.CodeDom exists, it has code support for generating code but not for compiling code. That's what this issue is about. The old code dom providers in .NET Framework would shell out to vbc and csc that was installed with the framework folder. There's also a new codedom package for .NET Framework that uses roslyn to compile https://www.nuget.org/packages/Microsoft.CodeDom.Providers.DotNetCompilerPlatform/
• ASP.NET on .NET Framework used CodeDom in several ways:
  
  
  
  • For aspx pages and code generation for webforms controls
  
  
  • For compiling aspx and Razor pages (via the BuildManager)
  
  
  • For compiling App_Code and other App_* folders
  
  
  • In this world, adding more code dom providers would add more language support to various parts of ASP.NET
  
  
• ASP.NET Core does not use CodeDom in any way shape or form.

That said, having an F# provider that works on .NET Framework and .NET Core (.NET Standard really) is fine for things that are built on codedom.

Hope that helps

[drvink]

@davidfowl OK, thanks, that clarifies the ASP.NET/ASP.NET Core stuff for me. The question I now have is how to register CodeDom providers (i.e. the way one used to do so via machine.config) on .NET Core.

[trvsysadmin]

@davidfowl Thanks for the detailed explanation. Is there any update on this topic? We have a use case where we compile the VB code inside of excel files inside an app.

After some hours of researching, any clarification with these questions would be really appreciated:

• there is no way to compile VB code using the codedom providers in Core at the moment, correct?
• is it possible to "shell out" from Core by separately compiling the vb code as a project in another folder? Almost a bit like an RPC call to a VB server.
• can we use the Roslyn compiler (via nugetting Microsoft.CodeDom.Providers.DotNetCompilerPlatform) inside Core to achieve the same result? Asking this because @stephentoub said This should be fairly straightforward to build on top of Roslyn's API's and not sure if this was ever done.
• Can we use this: https://www.nuget.org/packages/CoreCompat.System.CodeDOM/ ?

We are using Core 2.1.400 .

Thank You!

[danmosemsft]

Correct, it should be possible but it has not been done. It would be interesting if someone wanted to attempt a PR.

[trvsysadmin]

@danmosemsft Thank you.

Would it still be possible to use Roslyn like this in the meantime?

using System.Runtime.Loader;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.Emit;
using Microsoft.CodeAnalysis.VisualBasic;
using Microsoft.VisualBasic;

MetadataReference[] references = new MetadataReference[]
{
    MetadataReference.CreateFromFile(typeof(object).GetTypeInfo().Assembly.Location)
};

SyntaxTree syntaxTree = VisualBasicSyntaxTree.ParseText(VBSOURCECODESTRING);
VisualBasicCompilation compilation = VisualBasicCompilation.Create(
    "excelVbAssembly",
    syntaxTrees: new[] {syntaxTree},
    references: references,
    options: new VisualBasicCompilationOptions(OutputKind.DynamicallyLinkedLibrary));

MemoryStream ms = new MemoryStream();       
EmitResult result = compilation.Emit(ms);
ms.Seek(0, SeekOrigin.Begin);
Assembly assembly = AssemblyLoadContext.Default.LoadFromStream(ms);

I get this error from the compilation.Emit(ms); statement:

BC35000: Requested operation is not available because the runtime library function 'Microsoft.VisualBasic.CompilerServices.StandardModuleAttribute..ctor' is not defined.

[trvsysadmin]

I managed to fix my issue as it was just the usual problem of MetadataReferences. In the end, I hardcoded all my dlls. This is a well known topic and there is more information here:

https://github.com/dotnet/roslyn/wiki/Runtime-code-generation-using-Roslyn-compilations-in-.NET-Core-App

https://stackoverflow.com/questions/39257074/net-core-amd-roslyn-csharpcompilation-the-type-object-is-defined-in-an-assem

MetadataReference[] references = new MetadataReference[]
{
MetadataReference.CreateFromFile("/home/user/project/packages/microsoft.visualbasic/10.3.0/lib/netstandard2.0/Microsoft.VisualBasic.dll"),
// Where you have your system.runtime.dll etc.
};

[87Alex]

Hello,
Is there any news? We also use the CSharpCodeProvider class as follows:

public Assembly Compile(String sourceCode, String[] references, String assemblyFile) {
            CompilerParameters compilerParameters = new CompilerParameters();
            compilerParameters.ReferencedAssemblies.AddRange(references);
            compilerParameters.OutputAssembly = assemblyFile;
            CodeDomProvider codeProvider = new CSharpCodeProvider();
            CompilerResults compilerResults = codeProvider.CompileAssemblyFromSource(compilerParameters, new String[] { sourceCode });
            return compilerResults.CompiledAssembly;
        }

[Yamboy1]

Hi, Is there any movement on this?

https://stackoverflow.com/questions/63749712/dynamically-compile-multiple-files-into-assembly-using-rosyln

[W3LLD0NE]

up.