Runtime: AssemblyLoadContext: Loading platform specific assets in a custom AssemblyLoadContext

[gokarnm]

Context : I'm creating a custom AssemblyLoadContext in my application to load a plugin from a directory. The plugin directory contains the output from _dotnet publish_ command for the plugin project, using a Framework-dependent deployment. If the plugin has dependencies which has platform specific implementations, _dotnet publish_ puts them in the /publish/runtimes/{RID} folders. A {appname}.deps.json file is present in the publish directory which contains metadata about dependencies. This file contains information about managed and native dependencies, which can be specific to a Runtime Identifier (RID). It also contains information about culture specific resource files (satellite assemblies). This file is used by corehost to build TPA list and resolve dependencies.

Snippet from .deps.json's target's section.

        "System.Banana/1.0.0": {
            "type": "package",
            "dependencies": {
                "System.Foo": "1.0.0"
            },
            "runtime": {
                "lib/dnxcore50/System.Banana.dll": { }
            },
            "resources": {
                "lib/dnxcore50/fr-FR/System.Banana.resources.dll": { "locale": "fr-FR" }
            },
            "native": {
                "runtimes/osx.10.10-x64/native/libbananahelper.dylib": { }
            }
        },
        "System.Diagnostics.TraceSource/4.0.0": {
            "dependencies": {
                <snipped>
            },
            "runtimeTargets": {
              "runtimes/unix/lib/netstandard1.3/System.Diagnostics.TraceSource.dll": {
                "rid": "unix",
                "assetType": "runtime"
              },
              "runtimes/win/lib/netstandard1.3/System.Diagnostics.TraceSource.dll": {
                "rid": "win",
                "assetType": "runtime"
              }
            },
            "compile": {
              "ref/netstandard1.3/System.Diagnostics.TraceSource.dll": {}
            }
      },

Questions :

1. Is the resolution logic which uses .deps.json to resolve dependecies implemented in the default Assembly Load Context?
2. Could you point me to the relevent classes/source files where this logic is implemented?
3. Is there a way to reuse this resolution logic without having to replicate the rules in a custom assembly load context?

[jkotas]

cc @schellap @gkhanna79

[gkhanna79]

@gokarnm

The resolution of RID specific assets is done by the .NET Core Host, which starts CoreCLR. This code lives in https://github.com/dotnet/core-setup/tree/master/src/corehost/cli - look for deps_resolver.*

2) No, currently there is no way to reuse this logic from outside the host.

[gokarnm]

Thanks for the details!

I'm trying to understand how RID metadata defined here Microsoft.NETCore.Platforms/runtime.json is used.
This information is available in the dotnet install dir in _C:\Program Files\dotnet\shared\Microsoft.NETCore.App\1.0.1\Microsoft.NETCore.App.deps.json_ under "runtimes" section. Seems like this information file is use by Nuget for packaging convention and when Nuget resolves a dependency as described here.

1. Does this file play any part in dependency resolution at runtime?

Snippet from _C:\Program Files\dotnet\shared\Microsoft.NETCore.App\1.0.1\Microsoft.NETCore.App.deps.json_

"runtimes": {
     "win10-x64": [
      "win10",
      "win81-x64",
      "win81",
      "win8-x64",
      "win8",
      "win7-x64",
      "win7",
      "win-x64",
      "win",
      "any",
      "base"
    ],
}

1. Can I use RID graph in that file to implement dependency resolution in a custom ALC, instead of using {appname}.deps.json?
2. I'm assuming that corehost uses {appname}.deps.json instead of Microsoft.NETCore.App.deps.json/runtimes section, is that an optimization?
3. Or I would be missing something if I don't take {appname}.deps.json into account? I'm trying to understand the big picture here so I can do a custom implementation of the resolution logic.

[schellap]

Can I use RID graph in that file to implement dependency resolution in a custom ALC, instead of using {appname}.deps.json?

For portable apps, the host relies on two dependency files: MNA.deps and appname.deps, these files are merged such that app preferred (override) versions are given priority over framework versions.

For standalone apps, the appname.deps is entirely used.

In the future, there may be many dependency files depending on new features.

I'm assuming that corehost uses {appname}.deps.json instead of Microsoft.NETCore.App.deps.json/runtimes section, is that an optimization?

See above, for portable apps both are needed. This merged view tells what assemblies are needed. The actual resolution, the where these assemblies live follows a resolution policy in deps_resolver.cpp in https://github.com/dotnet/core-setup/tree/master/src/corehost.

Or I would be missing something if I don't take {appname}.deps.json into account? I'm trying to understand the big picture here so I can do a custom implementation of the resolution logic.

I would strongly advise against a custom implementation of the resolution logic as you have to play catch up with the changes we make. However, I believe you could help us export this logic out of hostpolicy.dll so you can call into the DLL for the resolution. I would happy to help here if you have questions.

[gkhanna79]

I believe you could help us export this logic out of hostpolicy.dll so you can call into the DLL for the resolution

I would want to be very explicit here - using RIDs (and to perform asset resolution based upon the RID graph) is an internal implementation detail of .NET Core that could change anytime in future. Please do not rely on this as a formally supported feature.

If your intent is to determine how the plugin's dependencies get resolved automatically, based upon RIDs, then this is a slightly different discussion that is about how a plugin's deps.json is expected to merge with the app.deps.json, something that CLI (the development kit) should look into.

@piotrpMSFT @livarcocc should chime in on the policy around this scenario.

[gokarnm]

@schellap, agreed on your points against custom implementation, that is the least preferred option.

@gkhanna79, thanks for the warning against using internal implementation details. Here is the use case, I would like feedback on best way to move forward. I understand that some of these features may not be available immediately. I would be great to discuss if this is a feature worth adding.

• There is in built resolution logic that corehost uses for dependency resolution that takes into account platform specific assets and culture specific assets. As you said, RID and deps.json files are internal details of this implementation.
• As a developer implementing a host that loads plugins, I would like to use the same resolution logic in some cases. A custom ALC is the mechanism that .NET Core provides to implement isolation, which is used in plugin based scenarios.
• I think there is a valid use case for a custom ALC to be able to call into this default resolution logic. This way the internal details of resolution can change (assuming the end result is the same), but a custom ALC can continue reusing this mechanism.

Let me know your thoughts.

[gkhanna79]

@gokarnm What specific aspect of the resolution you would like to leverage?

[gokarnm]

Hi @gkhanna79, here are the pieces of resolution logic that are interesting for someone implementing hosting/plugin scenarios.

1. In corehost , deps_resolver_t::resolve_probe_paths has logic that merges deps.json, applies RID precedence, and comes up with the list of paths (TPA list, native paths, resource paths)

2. In CoreCLR, AssemblyBinder::BindByTpaList has logic that, given an assembly name, resolves the path and binds to the assembly. It has logic that takes into account satellite assemblies, precedence for file extension and native DLLs, does ref-def check, and supports app path probing.

3. For resolution of native library loaded using DLLImport, NDirect::LoadLibraryModule has code to handle OS specific library names, and probing native DLL search directories. I haven't looked into this method in detail, so I may be missing some things.

It would be great to have APIs that allows reusing this logic. This will help developers implement a host that loads plugins to have dependency resolution behavior similar to what CoreHost and CoreCLR provides today, without reimplementing it themselves. Here are some thoughts on how this logic can be exposed to developers. I think the proposed approach does not expose internals of the resolution logic.

1. An _AssemblyPathResolver_ class that provides the assembly path given an assembly name . It's instantiated with a provider that provides the TPA list and probing paths. _IProbePathProvider_ and _DefaultProvider_ are the provider interface and default implementation. These three types can be part of CoreCLR/CoreFx.

public class AssemblyPathResolver
{
    public AssemblyPathResolver(IProbePathProvider provider) { }

    public Path ResolvePath(AssemblyName assemblyName) 
    {
        // returns assembly path 
    }

    public Path ResolveNativePath(string unmanagedDllName) 
    {
        // returns native library path
    }
}

public interface IProbePathProvider
{
    string TpaList { get; }
    string NativePaths { get; }
    string ResourcePaths { get; }
}

public DefaultProvider : IProbePathProvider
{
    public DefaultProvider(string tpaList, string nativePaths, string resourcePaths) { } 
}

1. CoreHost can provide an implementation of _IProbePathProvider_ that uses deps.json for create TPA list and probing paths.

public DependencyJsonBasedProvider : IProbePathProvider
{
    public DependencyJsonBasedProvider(string fxDir, string fxDepsPath, 
                   string appDir, string appDepsPath) 
    {
        // Run deps.json based resolution logic and 
        // set TPA path, native paths, and resource paths members.
    }
}

These classes can be used in a custom ALC implementation. The ALC's _Load_ and _LoadUnmanagedDll_ method overrides can use _AssemblyPathResolver.ResolvePath_ and _AssemblyPathResolver.ResolveNativePath_ to get assembly path, and subsequently call ALC's _LoadFromAssemblyPath_, _LoadFromNativeImagePath_ and _LoadUnmanagedDllFromPath_ methods to actually load the assembly.

[gokarnm]

@gkhanna79, @piotrpMSFT and @livarcocc, I would appreciate your thoughts on this.

[gokarnm]

@gkhanna79 , any updates on this?

[gokarnm]

@gkhanna79, @schellap do you have comments on the proposed API or any alternatives to reuse the resolution logic?

[MithrilMan]

I'm interested in this plugin loader scenario too

[jhudsoncedaron]

I just got here with the same problem.

[Cyberboss]

Any updates on this?

[jhudsoncedaron]

@Cyberboss : As of .NET 2.1 the AssemblyLoader will descend into a nuget package directory specified in .runtimesettings.json with no help from you.

[Cyberboss]

I'm looking at the issue of loading a netstandard package that's been dotnet published to a directory. It works for the root level Any CPU .dlls but not the platform specific stuff. Here's a picture example:

What would be the best way to go about this?

[yulia-d]

Also interested in this use case, any updates?

[yulia-d]

I would want to be very explicit here - using RIDs (and to perform asset resolution based upon the RID graph) is an internal implementation detail of .NET Core that could change anytime in future. Please do not rely on this as a formally supported feature.

This is unfortunate, since for now as a cross-platform plugin-hosting application we are forced to implement the same behavior ourselves, relying on project.assets.json of the plugin's project for RID-to-path mappings, which is even worse.

If anybody has a better idea how to go around this I'd be very happy to hear.

[Jasdo]

Having the exact same problem. Any ideas?

[vitek-karas]

In .NET Core 3.0 (Preview 5) this scenario should be solved by using AssemblyDependencyResolver. This is a new class added in 3.0 which uses the same code as the startup to read .deps.json file for the "plugin" and implements resolve methods for managed and native dependencies. This can be used to implement custom AssemblyLoadContext which resolves all the dependencies correctly.

Take a look at this sample AppWithPlugins which implements the above for plugins with native dependencies as well as satellite assemblies.