Runtime: "dotnet test" fails to find .so file in output folder on Ubuntu when LD_LIBRARY_PATH is blank.

[nguerrera]

_From @nguerrera on July 14, 2018 0:14_

_From @mpp-oliverh on July 11, 2018 21:29_

I have a cross-platform project which uses [DllImport] to load a native binary. I have equivalent .dll and .so placed in the application root, set to CopyAlways and they appear in the output folder.

The application runs correctly on both Windows and Linux (Ubuntu) both when published and from the command line with "dotnet run".

I have a test which calls a method from the loaded library. On Windows the test runs correctly, on a Linux system when LD_LIBRARY_PATH is not set (such as the visualstudio.com hosted Linux build agent) the test fails with a DllNotFoundException.

Setting the LD_LIBRARY_PATH to : (just a colon) causes the tests to pass. I believe an incorrect assumption is made somewhere during the test procedure about LD_LIBRARY_PATH.

Further details below, including further tests.

Many thanks!

Steps to reproduce

1. Create a project with a [DllImport] method from a library.so file.
2. Create a test for the method.
3. Set the library.so file to CopyAlways so it appears in the output folder alongside the normal application dlls.
4. Use the VSTS hosted linux build agent or host your own linux vsts-agent which has no LD_LIBRARY_PATH by default.
5. Build the project and run it - the method successfully loads the library.so file from the application folder and runs the imported method properly.
6. Run "dotnet test" against the project and you get DllNotFoundException.

Expected behavior

The tests run successfully - the .so file is found and loaded by the method under test.

Actual behavior

Error Message:
System.DllNotFoundException : Unable to load shared library 'library.so' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: liblibrary: cannot open shared object file: No such file or directory

Additional info

The fact that LD_LIBRARY_PATH is blank is relevant.
If LD_LIBRARY_PATH either starts or ends with a : then the test runs correctly.

export LD_LIBRARY_PATH=
dotnet test
fails with error above

export LD_LIBRARY_PATH=:
dotnet test
passes

export LD_LIBRARY_PATH=:blah
dotnet test
passes

export LD_LIBRARY_PATH=blah:
dotnet test
passes

export LD_LIBRARY_PATH=blah
dotnet test
fails with error above

export LD_LIBRARY_PATH=path_to_application_output_directory
dotnet test
passes

Environment data

dotnet --info output:

root@vsts-agent-5c87858b7b-dsbjf:/vsts# dotnet --info
.NET Core SDK (reflecting any global.json):
Version: 2.1.301
Commit: 59524873d6

Runtime Environment:
OS Name: ubuntu
OS Version: 16.04
OS Platform: Linux
RID: ubuntu.16.04-x64
Base Path: /usr/share/dotnet/sdk/2.1.301/

Host (useful for support):
Version: 2.1.1
Commit: 6985b9f684

.NET Core SDKs installed:
2.1.301 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.1 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.1 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.1 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

_Copied from original issue: dotnet/cli#9649_

_Copied from original issue: Microsoft/vstest#1690_

[nguerrera]

_From @XiaoFaye on July 16, 2018 23:3_

Even set the LD_LIBRARY_PATH as your suggestions, I still got the error in Ubuntu 18.04. The dotnet core version is 2.1.302

[nguerrera]

_From @cltshivash on July 17, 2018 16:15_

@nguerrera Do you see this issue on a simple console app ? (to narrow down the root cause)

[nguerrera]

I haven't reproduced this myself. I moved @mpp-oliverh's issue because they indicated that it only repros in dotnet test. Perhaps they can answer your question...

[nguerrera]

_From @abhishkk on July 19, 2018 10:32_

@mpp-oliverh
Are you seeing this issue on simple console app as well?

[nguerrera]

_From @nigurr on July 30, 2018 12:23_

Closing this issue as there's no further response. Please feel free to re-open this if you are still having an issue.

[nguerrera]

_From @ashkaun on August 1, 2018 0:8_

I am still running into this issue (on CentOS rather than Ubunutu). The workaround mentioned in the original post works for me as well.

I am also able to reproduce the issue on macOS, however it refers to the DYLD_LIBRARY_PATH rather than LD_LIBRARY_PATH due to the difference in loading linked libraries between the two operating systems.

To reiterate:

The application runs correctly on both Windows and Linux (Ubuntu) both when published and from the command line with "dotnet run".

I have a test which calls a method from the loaded library. On Windows the test runs correctly, on a Linux system when LD_LIBRARY_PATH is not set (such as the visualstudio.com hosted Linux build agent) the test fails with a DllNotFoundException.

Setting the LD_LIBRARY_PATH to : (just a colon) causes the tests to pass. I believe an incorrect assumption is made somewhere during the test procedure about LD_LIBRARY_PATH.

[nguerrera]

_From @nigurr on August 1, 2018 5:0_

@ashkaun thanks for updating the thread. This seems to be an issue with dotnet coreclr. Test platform doesn't use any of these variables to load dependencies. They are resolved by the coreclr.
I came across this active issue in coreclr https://github.com/dotnet/coreclr/issues/12707. Are they related? If not, can you please open new issue in dotnet/coreclr repo.

[nguerrera]

_From @ashkaun on August 1, 2018 17:33_

@nigurr From what I can tell this issue is unique to dotnet test, there are no problems running with the *.so file when running it with dotnet run. Additionally the *.so files are in the output & publish directories for the test projects as well as the solution itself.

I'll see if the issue you linked provides additional info in isolating the behaviour I'm experiencing.

[nguerrera]

_From @mpp-oliverh on August 1, 2018 17:39_

@cltshivash the issue report was prompted by a problem with testing a dotnet console application.

@nigurr the issue isn't present at run time, only ever during test. that issue you linked doesn't seem directly related.

this issue was originally posted to the dotnet-cli repository and was moved while I was away, sorry for the delay responding.

[nguerrera]

_From @karanjitsingh on November 23, 2018 6:49_

Sorry for the delayed response. This seems to be a coreclr issue since we don't internally check for or set LD_LIBRARY_PATH. Our recommendation would be to set this externally.

Considering the time that has passed since this issue was opened, maybe they've already fixed it? If not, please report this issue at dotnet/cli

[nguerrera]

_From @karanjitsingh on November 23, 2018 6:52_

Closing this issue for now.

[nguerrera]

_From @oliverholliday on November 23, 2018 8:56_

Hmm they moved it to here from there!

The problem wasn't fixed as of last week, I'll follow up with them thanks.

[nguerrera]

I'll move this to coreclr. This isn't a CLI issue either.

@oliverholliday Is it possible to supply a complete repro project and steps?

[oliverholliday]

@nguerrera thanks for that.

I'm very busy at the moment but I will try and get something going when I can. I don't have the build environment for the .so but I'll see what I can do.

Cheers

[janvorli]

You can try to run the test with LD_DEBUG=all set. It logs all the details about where runtime dynamic linker looks for shared libraries, what paths it checks and what symbols it maps. If you run your test with and without the LD_LIBRARY_PATH set and share the logs from both e.g. via gist, I can take a look.

[Arcanaeum]

I'm getting the same issue - only with dotnet run.

• .so library file is present next to the compiled executable (actually, I put a copy of the .so file in every directory in the project just for testing purposes)
• when LD_LIBRARY_PATH is blank, I get the error (No such file or directory)
• if LD_LIBRARY_PATH contains anything, including just :, the program executes successfully.

Again, I'm using dotnet run (not dotnet test). Version is .net core 2.2. I'm happy to provide a gist for with LD_DEBUG=all if that would be helpful.

[jeffschwMSFT]

This issue has been debated in https://github.com/dotnet/cli/issues/10379. I am in favor of enabling this scenario by adjusting the current directory (though that is looking less likely).
We will take a look, as the native binary lookup should first consider native binaries next to the calling assembly.
cc @jkoritzinsky @AaronRobinsonMSFT

[nguerrera]

I thought loading dlls from CWD was generally frowned upon. @morganbr

[morganbr]

Yes, loading dlls from the working directory is a problem. It's not clear to me that https://github.com/dotnet/cli/issues/10379 is about where to load dlls from though.

[jkoritzinsky]

I can't reproduce this failure on .NET Core 2.1.10, 2.2.4, or any 3.0 preview using the most recent SDKs for each on Ubuntu 18.04.

[jkoritzinsky]

Closing as no-repro since I still can't reproduce this with the given steps.

[mrhelmut]

Hello,

I'd like this issue to be considered for re-opening. The issue is still very present as of 3.1.100 and we have scenarios where loading libraries from the working directory is required and can't be worked around.

The issue is present on both Linux and macOS (and makes DllImport inconsistent with Windows).

The exact scenario is that we have a closed source native library from a third party vendor that we have to DllImport and must ship in the working directory.
This library has another dependency toward a native library from the same vendor that is also in the working directory.

The initial DllImport works but the secondary dependency can't be found because the LD path is not set to anything.

It would be very much appreciated if the working directory is added to the library discoverability path.

P.S.: it is not a vstest issue, it's about dotnet run and self-contained apps.

[oliverholliday]

I agree.

Not sure why it was closed as it's very easy to reproduce - configure an environment with LD_LIBRARY_PATH blank and it happens.

We still have a workaround in place for this for use on MS's own build agents.

[mrhelmut]

If need be, here's a bootstrap script compatible with both Linux and macOS that workaround this issue:

#!/bin/bash

# Move to script's directory
cd "`dirname "$0"`"

# Get the system architecture
UNAME=`uname`

# Setup LD path to the script current path
if [ "$UNAME" == "Darwin" ]; then
    # macOS
    export DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:./
else
    # Linux
    LD_LIBRARY_PATH=./
    export LD_LIBRARY_PATH
fi

# Start the app
./MyApp

[andreakarasho]

same here on Ubuntu 20.04 and netcore 3.1 with selfcontained or just the standard published application

[Bazmodin]

Same on CentOS7 and netcore 2.1