Roslyn: Can't reference mscorlib in .Net Core

[Pzixel]

Here is sample code which is working on target 4.5.2, but doesn't on Core 1.0.1.

using System;
using System.IO;
using System.Linq;
using System.Reflection;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;

class Program
{
    static void Main(string[] args)
    {
        var greetings = HelloWorlder.GetGreetings();
        // should write Hello World
        greetings();
    }
}

public static class HelloWorlder
{
    public static Action GetGreetings()
    {
        var tree = CSharpSyntaxTree.ParseText(@"
using System;
public class MyClass
{
    public static void Main()
    {
        Console.WriteLine(""Hello World!"");
        Console.ReadLine();
    }   
}");
        const string testAsmName = "testLib";

        var coreDir = Path.GetDirectoryName(typeof(object).GetTypeInfo().Assembly.Location);

        var mscorlib = MetadataReference.CreateFromFile(Path.Combine(coreDir, "mscorlib.dll"));
        var compilation = CSharpCompilation.Create(testAsmName,
            syntaxTrees: new[] { tree }, references: new[] { mscorlib });

        var emitResult = compilation.Emit($"{testAsmName}.dll");

        if (!emitResult.Success)
        {
            throw new Exception(string.Join(Environment.NewLine, emitResult.Diagnostics.Select((x, i) => $"{i + 1}. {x}")));
        }
        var ourAssembly = Assembly.Load(new AssemblyName(testAsmName));
        var type = ourAssembly.GetType("MyClass");

        var meth = type.GetRuntimeMethod("Main", Type.EmptyTypes);
        return () => meth.Invoke(null, null);
    }
}

As you can examine yourself, it writes Hello world on full framework but fails to compile on .Net Core with following errors:

1. warning CS8021: No value for RuntimeMetadataVersion found. No assembly containing System.Object was found nor was a value for RuntimeMetadataVersion specified through options.
2. (3,14): error CS0518: Predefined type 'System.Object' is not defined or imported
3. (5,19): error CS0518: Predefined type 'System.Void' is not defined or imported
4. (7,27): error CS0518: Predefined type 'System.String' is not defined or imported
5. (7,9): error CS0518: Predefined type 'System.Object' is not defined or imported
6. (7,9): error CS0103: The name 'Console' does not exist in the current context
7. (8,9): error CS0518: Predefined type 'System.Object' is not defined or imported
8. (8,9): error CS0103: The name 'Console' does not exist in the current context
9. (3,14): error CS1729: 'object' does not contain a constructor that takes 0 arguments

[jaredpar]

This behavior is "By Design". On Net Core all of the core types are actually located in a private DLL (believe the name is System.Private.Corlib). The mscorlib library, at runtime, is largely a collection of type forwarders. The C# compiler is unable to resolve these type forwarders and hence issues an error because it can't locate System.Object.

This is a bit of a known issue when working on Net Core. There is a stricter separation of runtime and compile time assemblies. Attempting to use runtime assemblies as compile references is not supported and frequently breaks do to the structure of the runtime assemblies.

The discussion around getting the right compile time assemblies is probably best had on dotnet/corefx as they own the packaging and distribution.

[Pzixel]

So the answer is Roslyn is not working on .Net Core by design? Because I don't see any possibility to compile anything without mscorlib. I know about forwarders but I expected that if runtime can determine the right assembly (obviosly, because .Net Core code is runable), we can do the same thing.

Sorry if I'm missing something, I'm trying to understand how it works, because I issued problem with porting my code-generating open-source library on .Net Core. I wanted to make it more accessible thus started this porting but faced with problem that no type-generation code is available on Core (TypeBuilder and so on are missing) other than Roslyn. And now I see that it's not working too because we just can't compile tree without referencing mscorlib which is not possible refer to.

[jaredpar]

So the answer is Roslyn is not working on .Net Core by design?

No. Roslyn is working as designed here. It's being provided an mscorlib that is incomplete and is responding accordingly. Given a mscorlib that is a suitable reference assembly it will compile just fine.

The root problem here is that mscorlib, on Net Core, is not a stand alone reference library. That part is "By Design". That's not a decision that roslyn controls though.

I know about forwarders but I expected that if runtime can determine the right assembly (obviosly, because .Net Core code is runable),

The compiler is actually pretty dumb when it comes to resolving references. It makes literally no attempt at finding missing references. It requires that all the necessary references are passed to the compiler. If any are missing it will error. The job of finding the set of references (and resolving conflicts) is up to MSBuild, project.json, make, etc ... This is how the basic build responsibilities are divided.

Also the actual mscorlib which contains Object (lets call it System.Private.Corlib) isn't usable as a reference library. It has a number of inconsistencies and duplicate types that make it unsuitable as a reference assembly. Code which is normally compilable won't compile when using System.Private.Corlib due to these changes. Hence even if Roslyn found it, the code isn't guaranteed to compile. It really needs a referencable version of mscorlib in order to function here.

I don't want to dismiss your concerns here. This is a problem we've faced ourselves in a number of areas: scripting + MSBuild in particular. Unfortunately there are no good answers here. Actually just met about this issue just a few weeks ago.

This issue is likely to get more traction in the corefx repo though. This is where the decisions around runtime / reference assemblies occur.

CC @terrajobst, @ericstj

[terrajobst]

Let me try to put another spin on this. @Pzixel's scenario is doing something quite specific: it's blending runtime and design time. @Pzixel is using Roslyn to compile code on the fly and then wants to load & execute that program in the same process.

That's different from "regular" design time. Normally, the Roslyn compiler simply takes text and compiles PE files on disk for a set of assemblies provided as the input. In that world, it doesn't even matter what Roslyn itself is implemented in (in fact, in the past the compiler was a C++ app). And as @jaredpar said: the compiler is pretty dumb and pretends to know nothing about the assemblies your program is referencing. That's why we usually pass the compiler "empty assemblies" that simply declare which APIs it has. At runtime, those assemblies need to be able to be resolved but where they live is irrelevant. We call these things reference assemblies.

However in @Pzixel's scenario runtime and design time happen to be the same thing and that's where things get a bit confusing. The reason being that people often start with the mindset of reflection which never had a separation of design time and runtime. So for example, doing something like this:

C# var coreDir = Path.GetDirectoryName(typeof(object).GetTypeInfo().Assembly.Location); var mscorlib = MetadataReference.CreateFromFile(Path.Combine(coreDir, "mscorlib.dll"))

Seems fine but is actually an example where the code makes the assumption that things are laid out in a specific way at runtime which may not be the case.

So it's not so much that Roslyn doesn't support .NET Core -- it does (first, Roslyn itself can run on .NET Core and you can use Roslyn to produce binaries for .NET Core). It's about that when you're using Roslyn at runtime to emit and load on the fly you've to understand how to do it correctly.

You should follow what Razor views in ASP.NET Core are doing. Require the developer to produce a set of reference assemblies that you can use at runtime to pass to Roslyn as references.

That being said, I understand that this isn't super trivial today and that we lack good docs and features around this. We've discussed this recently with the MSBuild team who need to solve the same problem for their inline tasks. We're thinking of ways to make it easier to get access to the set of assemblies that you be using for compilation.

I suggest that you open an issue in the dotnet/sdk repo as that's where the request logically belongs.

[adamsitnik]

In BenchmarkDotNet we have the same problem. Today we are using dotnet cli to do the compilation, but people start migration from project.json and instead of supporting both project.json and new csprojs I would like to switch to Roslyn.

[Ziriax]

I just upgraded to Visual Studio 15.5.1, and I have a .NET 4.6.2 project that uses Roslyn and now doesn't run anymore, because of this error, while I did not upgrade my Roslyn nuget packages at all. Any idea who to fix this? This is not .NET Core related, but I'm still bitten. I'm a bit stuck now...

[Stamo-Gochev]

@terrajobst @jaredpar

Seems like loading the System.Private.Corlib.dll by using the code from https://github.com/dotnet/roslyn/issues/16211#issuecomment-270510468, e.g.:

var coreDir = Path.GetDirectoryName(typeof(object).GetTypeInfo().Assembly.Location);
var mscorlib = MetadataReference.CreateFromFile(Path.Combine(coreDir, "System.Private.CoreLib.dll"));

//seems like a better option though
var mscorlib = MetadataReference.CreateFromFile(typeof(object).GetTypeInfo().Assembly.Location);

and emitting the result to a memory stream fixes the snippet:

using System;
using System.IO;
using System.Linq;
using System.Reflection;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;

class Program
{
    static void Main(string[] args)
    {
        var greetings = HelloWorlder.GetGreetings();
        // should write Hello World
        greetings();
    }
}

public static class HelloWorlder
{
    public static Action GetGreetings()
    {
        var tree = CSharpSyntaxTree.ParseText(@"
using System;
public class MyClass
{
    public static void Main()
    {
        //Console.WriteLine(""Hello World!"");
        //Console.ReadLine();
    }   
}");
        const string testAsmName = "testLib";

        var coreDir = Path.GetDirectoryName(typeof(object).GetTypeInfo().Assembly.Location);

        var mscorlib = MetadataReference.CreateFromFile(Path.Combine(coreDir, "System.Private.CoreLib.dll"));

        var options = new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary);
        var compilation = CSharpCompilation.Create(testAsmName,
            syntaxTrees: new[] { tree }, references: new[] { mscorlib }, options: options);

        var ms = new MemoryStream();

        var emitResult = compilation.Emit(ms);
        ms.Seek(0, SeekOrigin.Begin);

        if (!emitResult.Success)
        {
            throw new Exception(string.Join(Environment.NewLine, emitResult.Diagnostics.Select((x, i) => $"{i + 1}. {x}")));
        }

        var ourAssembly = Assembly.Load(ms.ToArray());
        var type = ourAssembly.GetType("MyClass");

        var meth = type.GetRuntimeMethod("Main", Type.EmptyTypes);

        ms.Close();
        return () => meth.Invoke(null, null);
    }
}

Here is a diff of the two versions for an easier comparison: https://www.diffchecker.com/veKPhtJv

Is this actually intended to be working?

@terrajobst
On a side note, can you provide more details about the alternative approach you mentioned at https://github.com/dotnet/roslyn/issues/16211#issuecomment-270510468:

You should follow what Razor views in ASP.NET Core are doing. 
Require the developer to produce a set of reference assemblies that you can use 
at runtime to pass to Roslyn as references.

I am doing a similar dynamic compilation to the above code in an Class Library project, which has its target frameworks defined as:

<TargetFrameworks>net451;netstandard1.6</TargetFrameworks>

and used by an ASP.NET Core web app, whose target is:

<TargetFramework>netcoreapp2.0</TargetFramework>

and although System.Private.CoreLib.dll is loaded successfully, the compilation fails with the original problem:

warning CS8021: No value for RuntimeMetadataVersion found. No assembly containing System.Object was found nor was a value for RuntimeMetadataVersion specified through options.

I am currently investigating ways to fix that, but it will be helpful, if you can share an alternative that I can try.

[tmat]

You might find this doc useful: https://github.com/dotnet/roslyn/wiki/Runtime-code-generation-using-Roslyn-compilations-in-.NET-Core-App

[terrajobst]

@Stamo-Gochev

On a side note, can you provide more details about the alternative approach you mentioned

It's a project setting called PreserveCompilationContext. We don't have great docs but it was discussed on Stack Overflow.

The tricky thing is that you have to pass the core assembly (i.e. the one containing System.Object). For .NET Framework, this assembly is at design time and runtime mscorlib. In .NET Core it is in System.Runtime at design time and System.Private.CorLib at runtime, but the latter is an implementation detail.

@tmat Roslyn follows type forwards, right? So is it valid to just pass System.Runtime as the single assembly or do you have to pass in the assembly that contains the definition of object?

[tmat]

@terrajobst You have to pass in all assemblies that are involved in the compilation. Type forwarders are followed but you need to specify both the assembly that defines the forwarder and the assembly that the forwarder forwards to.

[terrajobst]

I see. So that would require people to understand the implementation closure if they use Roslyn to generate code at runtime. We don't have a good mechanism for this today. Look like this would need a properly designed feature.

@richlander we should talk about this and clarify who of us should drive this. I think it's a clear gap in our offering we need to close.

[tmat]

@terrajobst Please include me in the loop.

[tmat]

@terrajobst The wiki describes two ways how to construct references for compilations. They are both pretty straightforward, although certainly not ideal and with room for improvement (providing helpers).

[terrajobst]

Hah! I probably should have read the doc you linked first :-)

The AppContext solution seems like a sensible starting point; the problem will arise as soon as parties want to subset because without reading the assemblies they would have to know the type forwarding relationship (i.e. internal factoring of the runtime) which we're generally changing between releases in order to improve performance/linking/internal engineering.

[tmat]

he problem will arise as soon as parties want to subset

Yeah, using implementation assemblies gives you everything. That's a property of this approach. If you want subset you need to use the second approach.

[terrajobst]

If you want subset you need to use the second approach.

For which the current offering sucks. The closest feature we have is PreserveCompilationContext but I'm not sure that's the best way to go about it.

[tmat]

For which the current offering sucks.

I agree.

[MortenChristiansen]

Edit: This was something unrelated after all. I had a call CSharpCompilation.Create("Things.dll"..., but this should not include .dll.

@tmat I've tried applying the things mentioned in this issue but I can't figure out how to generate an assembly using Roslyn which can be consumed by a .NET Core application. I've created a simple repro - maybe you can see what I'm missing/doing wrong? The idea is that the Codegen console app generates the assembly Thing.dll which the ConsoleApp2 console app consumes. Generating Things.dll works fine, but running ConsoleApp2 gives the error:

System.IO.FileNotFoundException: 'Could not load file or assembly 'Things.dll, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.'

I've tried numerous combinations of referencing assemblies, but all either fail at compile time or runtime.

If I should create a separate issue - here or elsewhere - let me know.

[Gabriel-RABHI]

For first step in runtime compilation, I have done this and it work. It is brutal, and I don't know if it work when deployed :

            var coreDir = Path.GetDirectoryName(typeof(object).GetTypeInfo().Assembly.Location);
            List<MetadataReference> _refs = new List<MetadataReference>();
            foreach (var p in Directory.EnumerateFiles(coreDir)) {
                if (Path.GetFileName(p).StartsWith("System.") && p.EndsWith(".dll"))
                    _refs.Add(MetadataReference.CreateFromFile(p));
            }

            var options = new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary);
            var compilation = CSharpCompilation.Create(testAsmName,
                syntaxTrees: new[] { tree }, references: _refs.ToArray(), options: options);

If deployed, I guess it will get the right assemblies. If it is to dynamically create some classes at runtime, you have to reference your project assemblies too.

.Net Core 2.x

[tmat]

@Gabriel-RABHI See https://github.com/dotnet/roslyn/wiki/Runtime-code-generation-using-Roslyn-compilations-in-.NET-Core-App

[svick]

@tmat That wiki page describes a fairly complicated way of gathering and distributing reference assemblies. Would it make sense to also mention the approach of using <PreserveCompilationContext> and Microsoft.Extensions.DependencyModel? It's less versatile, but also much simpler.

[tmat]

@nguerrera Is there some documentation of PreserveCompilationContext that we can refer to? The linked SO answer is good but I'd prefer to refer to official docs.

[nguerrera]

I don't know if there's a good doc. I'm not finding one. Open an issue on docs.

[nguerrera]

There are essentially two parts to it: 1) copy reference-only assemblies to refs folder 2) record information in deps file that dependency model API seen in SO answer reads.

Razor runtime compilation relies on it.