Runtime: DllNotFoundException after successful manual dlopen on MacOS

[lostmsu]

I am trying to embed Python 3.6 into my cross-platform C# app.

Since I want the app to rely on an existing Python installation, I let the user pick the Python interpreter, then preload its the dynamic library with LoadLibrary/dlopen, and rely on DllImport to use the preloaded library.

This approach works no problem on Windows and Linux (though in the later case I only tried with the library actually being in the load path), however on MacOS I get Unhandled Exception: System.DllNotFoundException: Unable to load shared library 'python3.6m' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(libpython3.6m, 1): image not found

Here's the repro example:

using System;
using System.Runtime.InteropServices;

namespace DynPreload
{
    class Program
    {
        static void Main(string[] args)
        {
            const string pythonLib = "/usr/local/opt/python36/Frameworks/Python.framework/Versions/3.6/lib/python3.6/config-3.6m-darwin/libpython3.6m.dylib";
            if (dlopen(pythonLib, DLOpenFlags.RTLD_NOW) == IntPtr.Zero) throw new Exception();
            Console.WriteLine("Preloaded!");
            Py_SetProgramName("42"); // <- this throws DllNotFoundException
        }

        enum DLOpenFlags {
            RTLD_NOW = 2,
        }
        [DllImport("dl")]
        static extern IntPtr dlopen(string path, DLOpenFlags flag);
        [DllImport("python3.6m")]
        static extern void Py_SetProgramName(string name);
    }
}

(to get Python 3.6 installed into the specified folder using brew, run brew install sashkab/python/python36)

This happens despite dlopen successfully preloading libpython3.6m.dylib in the second line.

TargetFramework is netcoreapp2.0

[AaronRobinsonMSFT]

@lostmsu The approach being taken here is a trick that has worked on Windows for years. However, the OS loader semantics on non-Windows don't always follow the same behavior and the semantics you are observing can result. At this point the behavior observed is undefined, but there is hope! I would suggest using the AssemblyLoadContext API that provides callbacks for requesting the native library - see https://github.com/dotnet/coreclr/blob/master/Documentation/design-docs/assemblyloadcontext.md. Alternatively you could try the newly added NativeLibrary API - see https://github.com/dotnet/corefx/issues/32015. The idea behind this is that the application can control the lifetime of the load native library and pass around function pointers. If can also be used with the AssemblyLoadContext mentioned above.

cc @swaroop-sridhar

[lostmsu]

@AaronRobinsonMSFT would it be correct to assume, that neither of these methods would work in .NET Standard 2.0, making them inapplicable to .NET Framework without multitargeting?
Actually, are any of these methods planned for .NET Core 3?

[AaronRobinsonMSFT]

@lostmsu Sorry about that. I was in the middle of penning a response because I noticed you were targeting 2.0. The NativeLibrary approach is only on 3.0, but the AssemblyLoadContext should be available. Alternatively you could use the dlopen/dlsym combination with Marshal.GetDelegateForFunctionPointer to create delegates and pass that around or hide behind some static - going this route means forgoing using DllImport.

[lostmsu]

@AaronRobinsonMSFT still wrapping my head around it.

_Actually, I am making a .NET Standard 2.0 class library, not an app (I just reduced the repro to a single file for the issue report)._

Per my understanding, my class library will be loaded into the default AssemblyLoadContext in most cases. And any time my library will try to load an unmanaged DLL (e.g. via DllImport), the attempt will be directed to the LoadUnmanagedDll if the that context. Which means, that it is the host, who determines unmanaged DLL resolution for my library. And for default context it is not overridable.

Now suppose I would want to hide that implementation detail from the host. Then I would create my own AssemblyLoadContext with custom LoadUnmanagedDll. But I would need to load something into that context, and I can not "statically reference" that something, I would have to access it via LoadAssembly + reflection, correct?

Seems a bit backwards, that an override to a very low level operation, such as native interop, requires involvement of reflection.

As for the GetDelegateForFunctionPointer approach. Is there a way to get rid of the delegate invocation overhead?

[jkotas]

As for the GetDelegateForFunctionPointer approach. Is there a way to get rid of the delegate invocation overhead?

The invocation overhead of the delegate returned by GetDelegateForFunctionPointer is comparable to the DllImport PInvoke overhead.

I would recommend using the GetDelegateForFunctionPointer for what you are trying to do.

[lostmsu]

@jkotas was it improved much since 2011?
http://ybeernet.blogspot.com/2011/03/techniques-of-calling-unmanaged-code.html

Related: https://github.com/dotnet/csharplang/issues/80

[AaronRobinsonMSFT]

I could be mistaken, but I believe the SuppressUnmanagedCodeSecurity attribute isn't even read by the .NET Core runtime - see https://github.com/dotnet/coreclr/issues/642.

Those are interesting numbers. I have a lot of questions about them, particularly target architecture. Lots of things have changed in .NET Core so I am not sure how relevant they are. However, I think the fact that we don't have a good answer regarding your question about the specific performance of the available options is the real issue.

cc @jeffschwMSFT

[AaronRobinsonMSFT]

Actually if we assume those numbers are still valid relative to one another and given the fact that the SuppressUnmanagedCodeSecurity is no longer an issue, the function delegate approach appears to be faster than a P/Invoke.

PInvoke - suppressed security           | 5415 ms
Function delegate - suppressed security | 7788 ms
PInvoke                                 | 8249 ms

I will try and write something up this weekend. We really should have some official numbers for these sorts of things.

Edit: Added the PInvoke without security to be complete. This argument is no longer valid.

[lostmsu]

@AaronRobinsonMSFT SuppressUnmanagedCodeSecurity was applicable to both, its in the table. Time to sleep! The remote work caused by snowfall is no excuse for overtime :)

[jkotas]

There is no fundamental throughput performance difference between the two though. It is what I meant by "comparable". Depending on how exactly you write your micro-benchmark or the method signature, one may be faster over the other. The delegate path has to do a few more processor instructions worth of work compared to DllImport, but that's not significant. It is what I meant by saying that the performance is "comparable".

[lostmsu]

@jkotas well, yeah, its not like RAM vs HDD, but 1.5x difference is quite a lot in the numerical compute scenario, which the Python binding is often be used for.

[jkotas]

What is the relative cost of the raw PInvoke vs. the cost of calling empty python function via the Python bindings?

I would expect that that the latter will be orders of magnitude more expensive that makes the cost of the raw PInvoke not something to worry about, ie. if the raw PInvoke became 5x faster by a miracle, you would not really notice that on the Python binding performace.

[AaronRobinsonMSFT]

@lostmsu Oh good point. Missed that, will update the table.

[jkotas]

It sounds like that the table is for .NET Framework (because the suppressing security makes a difference). Numbers for .NET Core are likely going to be different.

[AaronRobinsonMSFT]

but 1.5x difference is quite a lot in the numerical compute scenario, which the Python binding is often be used for.

@lostmsu I previously worked at The MathWorks (makers of MATLAB) and this was an argument we would hear a lot. What is being missed is the fact that the call should be noise relative to the work the function is actually doing. If during a perf measurement the dispatch of the call via some FFI is profound then more than likely the call isn't justified in the first place and the overhead of doing the work in the calling language/runtime is probably warranted - if perf is the greatest concern.

[AaronRobinsonMSFT]

It sounds like that the table is for .NET Framework

@jkotas Yep it is from 2011. I am going to try and get some new numbers in a small benchmark so we have something for future discussions.

[AaronRobinsonMSFT]

@lostmsu I am going to close this issue, but we are going to work on getting official perf numbers so we can provide confidence about making decisions regarding what approach to use. Thanks for the link and bringing this up!

[NextTurn]

@AaronRobinsonMSFT FYI the delegate spends twice the time on a simple function https://github.com/dotnet/coreclr/issues/22320#issuecomment-466736048

Another benchmark with similar results ltrzesniewski/Benchmarks