Machinelearning: Error when using OnnxTransformer in two AppDomains

[shaykels]

System information

• OS version/distro: Version Windows 10.0.17763

• .NET Version (eg., dotnet --info): .Net 4.6.2

Issue Error when using OnnxTransformer in two AppDomains

• What did you do?

I have a (production) application with two AppDomains.
I'm trying to run OnnxTransformer in my code within the AppDomain, and get an error when second AppDomain is invoked.
The error is easily reproduced in a Console App with two AppDomains (below).

• What happened?

Error is thrown.

Microsoft.ML.OnnxRuntime.OnnxRuntimeException: '[ErrorCode:RuntimeException] Only one instance of LoggingManager created with InstanceType::Default can exist at any point in time

Source code / logs

Here is simple code to reproduce the error:

namespace MlNetAppDomainErrorRepro
{
using System;
using Microsoft.ML;
using Microsoft.ML.Transforms;

class Program
{
    public static void CreateOnnxTransformer()
    {
        MLContext context = new MLContext();
        OnnxTransformer t = new OnnxTransformer(context, @"PathToYourModel.onnx");
    }

    static void Main(string[] args)
    {
        CreateOnnxTransformer();

        AppDomain secondAppDomain = AppDomain.CreateDomain("SecondAppDomain"); 

        secondAppDomain.DoCallBack(CreateOnnxTransformer); // error here

        Console.ReadLine();
    }
}

}

[Ivanidzo4ka]

Adding @jignparm I'm quite sure this has to do with onnx runtime and not our wrapper, can you pass this issue to right people?

[jignparm]

The OnnxRuntime library can only have a single environment created per process (although multiple sessions within the the environment).

Are these 2 AppDomains running in a single OS process? If that's the case, it will not work in the current design. There can only be one environment (e.g. thread pool, logger) per process.

[OlegLo]

Azure app service deployment slots are creating new domain on swap. Currently there is no easy way to detect when application is in staging slot, this feature finally implemented after 4 years wait is not working. We are using a lot of trickery to be able to use onnxruntime together with deployment slots. May we hope that 2 domains limit will be removed in the not so distant future?

[urikanonov]

@jignparm is there any update on this issue? It's been nearly a year and a half and we're still gravely suffering from this issue :(
I'm working with @shaykels (who opened this issue) on the same project.

[ViggoL]

Any update on this?

[urikanonov]

@ViggoL - updating to the new problems solved the issues for us!

These are the nugets that we use:

<PackageReference Include="Microsoft.ML" Version="1.5.2" />
<PackageReference Include="Microsoft.ML.CpuMath" Version="1.5.2" />
<PackageReference Include="Microsoft.ML.DataView" Version="1.5.2" />
<PackageReference Include="Microsoft.ML.FastTree" Version="1.5.2" />
<PackageReference Include="Microsoft.ML.OnnxRuntime" Version="1.5.2" />
<PackageReference Include="Microsoft.ML.OnnxTransformer" Version="1.5.2" />