Fsharp: Regression: IDE does not see provided types generated in referenced projects

[vasily-kirichenko]

1. git clone https://github.com/vasily-kirichenko/TypeProvidersBug.git
2. .paket\paket.bootstrapper.exe
3. .paket\paket restore
4. Open the solution in VS 2015, build it:

1. Open the solution in VS 2017:

1. Open it in Rider:

(I cannot make Ionide to work, sorry).

In VS 2015 VFPT does not see the provided property either:

So it's an old FCS bug, but the VFT compiler service does not have it. @dsyme is it possible we added a filter like if not symbol/entity.IsProvided then ... somewhere?

It's a quite a severe bug.

[forki]

I can confirm this is a regression.

It works for me in VS2015.
It doesn't work in VS2017 or Ionide.

[vasily-kirichenko]

ILSpy shows that Foo is public:

namespace TypeProvidersBug
{
    [CompilationMapping(SourceConstructFlags.Module)]
    public static class Test
    {
        [TypeProviderEditorHideMethods]
        public sealed class Configuration : YamlConfigTypeProvider.Root
        {
            private EventHandler _changed;
            private string _Foo;
            public event EventHandler Changed
            {
                add
                {
                    this._changed = (EventHandler)Delegate.Combine(this._changed, value);
                }
                remove
                {
                    this._changed = (EventHandler)Delegate.Remove(this._changed, value);
                }
            }
            public string Foo
            {
                get
                {
                    return this._Foo;
                }
                set
                {
                    this._Foo = value;
                }
            }
            public Configuration() : base(true)
            {
                this._Foo = "foo";
            }
        }
        [CompilationMapping(SourceConstructFlags.Value)]
        public static Test.Configuration config
        {
            get
            {
                return Test.config@7;
            }
        }
    }
}

[dsyme]

@vasily-kirichenko Great to have the repro. I will look at it now. It only appears to affect cross-assembly references w.r.t. generated types

[vasily-kirichenko]

@dsyme Thanks! :) Yes, it seems so.

VS 2017 C# editor shows the property:

But Rider does not:

That's super strange :(

[dsyme]

@vasily-kirichenko I think I understand what's happening

• VS2017 and VFPT both enabled in-memory cross-assembly references where the DLLs for referenced projects aren't read from disk but rather the projects are type-checked in-memory and the metadata constructed in-memory

• For generative type providers, there is a "late stage" in the process of preparing an assembly's metadata which statically links the generated types into the binary. This stage is being correctly run in the F# 4.1 command line compiler but is not being run in the corresponding in-memory scenario

This explains why this is "only" an IDE problem, and also why it is only in VS2017 and VFPT

To address this problem I think the best approach in the short term is to implicitly turn off in-memory assembly references when referenced project uses a generated provided type. That is consistent with other limitations for in-memory project references, e.g.:

              // Assemblies containing type provider components can not successfully be used via cross-assembly references.

and also described in the FCS docs here: https://fsharp.github.io/FSharp.Compiler.Service/project.html

Then a "compile" would be needed as in VS2015 in order to see the referenced DLL.

[dsyme]

@vasily-kirichenko I will also endeavour to add an option to turn off in-memory cross-project references altogether (leaving them on by default)

[vasily-kirichenko]

Thanks!

[dsyme]

I can't yet add the option because I'm getting this error when using the nightly with 15.2 (which is what I'm developing against)

Not sure what the workaround is for this. I assume it's working in 15.3 but can't verify

[vasily-kirichenko]

I have the same behavior on a version dated May, 17.

[vasily-kirichenko]

I attached to it, no exceptions, nothing in debug output.

[dsyme]

Fix is ready

[dsyme]

A bit more detail on why this was necessary, since @auduchinok and Alex Berezhnykh were asking.

For generative type providers, there is a "late stage" in the process of preparing an assembly's metadata which statically links the generated types into the binary. This stage is being correctly run in the F# 4.1 command line compiler but is not being run in the corresponding in-memory scenario. The “late stage” code being referred to is this https://github.com/dotnet/fsharp/blob/master/src/fsharp/fsc.fs#L1397 and all of this https://github.com/dotnet/fsharp/blob/master/src/fsharp/fsc.fs#L1444. This statically links the actual generated assembly fragments into the final .NET assembly. This code is not run for cross-assembly references in the IDE.

To understand why this is a problem, we need to understand the overall process for cross-assembly provided type definition references.

1. During type checking of the first assembly, TcTyconDefnCore_Phase1C_EstablishDeclarationForGeneratedSetOfTypes generates TAST nodes using Construct.NewProvidedTycon putting a wrapped System.Type in the node.

2. During type checking of the first assembly, when saving these TAST nodes into the pickled F# metadata, this code and this code in p_tycon_repr stores these nodes in a special way, removing the System.Type in the node and saving the equivalent of an ILTypeRef

3. During type checking of the second assembly, when reading the pickled metadata back into the system (i.e. when unpickling the TAST metadata of the assembly being referred to), this code is triggered. This searches through the actual .NET metadata (i.e. the ILModuleDef) for the referenced assembly trying to find the type and then makes the type appear as a .NET type as far as the compilation that refers to the assembly is concerned.

   For this to work, the actual .NET metadata (i.e. ILModuleDef) for the assembly being referred to must be available. However in an IDE setting, the .NET metadata and ILModuleDef are not generated and thus not available for cross-project assembly references. This is because the IDE doesn't run the optimization and IlxGen.fs phases.

This is basically difficult to solve. The only options I can think of are

1. Change TcTyconDefnCore_Phase1C_EstablishDeclarationForGeneratedSetOfTypes (or perhaps p_tycon_repr) to generate F# TAST metadata for all the contents of the generated provided types, including all methods, properties, nested types etc. However this is a big change and could easily hit limitations in F# TAST metadata, or inconsistencies. OR

2. Run the IlxGen phase for those assemblies that include generated provided type definitions,

I guess (2) is plausible and maybe not that hard.

It is a frustrating situation. The existing solution is already a little unsettling because the compiler's interpretation of the .NET view of the type may be subtly different to its interpetation of the provided type. Ideally, we would always make generated provided types produce regular F# TAST nodes all the way down (thus making generated provided types much more like "type definition macros"). The current approach comes from the fact that generated provided types really originated from the desire to statically link C#-generated assembly fragments and types generated by external tools into the compilation. It could be we rethink the whole approach to the underlying compiler infrastructure for generated provided types at some point, and I think this would simply require recompiling existing type providers with a new TPSDK. The compiler's approach to erased types is more reasonable and stable and I wouldn't expect that to change much, though I guess we could revise that too if we're going to make a revision at all.