Aws-lambda-dotnet: Nuget dependency is throwing System.IO.FileNotFoundException

[rianjs]

I'm not sure if this is a libphonenumber issue, or an AWS Lambda test tool issue. It has the feel of an issue with the test tool.

Repro steps:
1) Install the AWS Lambda test tool package: https://aws.amazon.com/blogs/developer/debugging-net-core-aws-lambda-functions-using-the-aws-net-mock-lambda-test-tool/
1) Create an empty Serverless Lambda app
2) Add the libphonenumber package as a dependency
3) In the Function constructor, do var foo = PhoneNumberUtil.GetInstance();
4) Run the debugger
5) Send any request

The constructor doesn't run at all.

System.IO.FileNotFoundException: Could not load file or assembly 'PhoneNumbers, Version=8.10.6.0, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.
   at Mercury.Function..ctor()

The smallest repro case I could come up with attached here.
AWSServerlessApp.zip

[rianjs]

I have also downloaded the source code for libphonenumber, and added that csproj as a project reference in a lambda solution. It builds fine, but as soon as I invoke the lambda via the test tool, the same error occurs.

I have opened a bug in the libphonenumber project as well, as I'm not sure which side the problem is on:
https://github.com/twcclegg/libphonenumber-csharp/issues/95

[klaytaybai]

Hi @rianjs, we're looking into this. I've confirmed that it doesn't work in the test tool, but the same library will work in a console app--as you state in the issue for the other repo. I've also verified that some other libraries work with the test tool, so now I'll continue investigating why this one doesn't. Let me know if you have updates on your end. Thanks for reporting it.

[rianjs]

If the code was open, I would have been happy to work through it myself. ;P

I'm guessing it works fine in a "real" lambda environment.

[mikylebaksh]

The Problem:

I have a similar issue, but with a different library. I have a netcore2.1 lambda project that references Microsoft.EntityFrameworkCore 2.2.3 and Microsoft.EntityFrameworkCore.Sqlite 2.2.3. When running in visual studio with the Mock Lambda Test Tool, the following exception is thrown:

System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Data.Sqlite, Version=2.2.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.
   at AWSLambda_EfCoreTest.Function.FunctionHandler(String input, ILambdaContext context)

I have confirmed that this lambda function can be deployed and run successfully in the real AWS cloud based Lambda environment.

See the small reproduction project attached

The Source:

I cloned the aws-lambda-dotnet repo and ran the test tool. I created new unit test to invoke the tool and step into the code. The test tool throws the exception during the Amazon.Lambda.TestTool.Runtime::LambdaAssemblyResolver::OnResolving method. This method cannot resolve AssemblyName = Microsoft.Data.Sqlite and will return null. This method is the last method called before the LoadContext has to call a System.IO.FileNotFoundException if the assembly is not resolved.

---

The solution posted in another issue works. In that issue, the author simply downgraded to a 2.1.* version. If we downgrade to 2.1.8 for both ef core references, the test tool starts working again. However this is not an option for me. I'm trying to take advantage of new features in ef core 2.2.*. IMO the tool should be able to load assemblies that are also supported in lambda.

This open issue in the corefx repo may give hints into the underlying issue.

I would explore further, but I don't have the time. Any ideas?

AWSLambda_NetCore21-EfCore22Test.zip

[normj]

I pushed out version 0.9.2 that has a change to how dependencies are searched for. With this change I was able to execute a lambda function with this tool that used libphonenumber.

If you are using Visual Studio and the AWS Toolkit for Visual Studio the update to 0.9.2 should happen automatically the next time you open your solution assuming you are online.

[normj]

Also @rianjs, the code for the mock test tool is open source and can be found in this repo at https://github.com/aws/aws-lambda-dotnet/tree/master/Tools/LambdaTestTool

[mikylebaksh]

Hey @normj,

Thanks for the help so far. I updated the tool to 0.9.2. I'm still getting System.IO.FileNotFoundException. Are you able to replicate using my sample project listed above?

For now, I can work around by kicking off my lambda functions for local development via a console app.

[rianjs]

Hmm, this still doesn't work for me.

[rianjs]

My previous comment, which I've edited, was wrong. Latest problem was just PEBKAC. The LibPhoneNumber assembly was found just fine.

[mrcunninghamz]

I am experiencing this issue in 0.9.2 with a nuget package that is being referenced from another project that my lambda project is dependent on.

[jiabiao]

I am experiencing the same issue in 0.9.2.
The test tool already loaded the Microsoft.Extensions.* 2.1., so It will fail if we want to use Microsoft.Extensions. 2.2.*.

Can you resolve this issue?

[AngelVenchev]

@jiabiao I am having the same issue.

System.IO.FileLoadException: Could not load file or assembly 'Microsoft.Extensions.DependencyInjection.Abstractions, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. Could not find or load a specific file. (Exception from HRESULT: 0x80131621)

Version 0.9.2

[IdresAhmed]

@AngelVenchev Could you try changing version from 2.2 to 2.1? This worked for me for a similar issue. May have to do with Lambda not supporting .Net Core 2.2

[AngelVenchev]

@IdresAhmed this is what I ended up doing, although I now have two versions 2.2 for deployment to lambda and 2.1 for running locally.

[normj]

@IdresAhmed @AngelVenchev I opened up this feature request that would address the issue of using different versions of Microsoft.Extensions.* then were loaded by the Testing tool's ASP.NET Core web app

[maj-soccermanager]

I have the same issue with the Google.Apis.Auth dependency. It works fine when deployed to AWS but when I try to run it locally with the test tool, I get this error. I'm using .NET Core 2.1 and test tool 0.9.2.

System.IO.FileLoadException: Could not load file or assembly 'Google.Apis.Auth, Version=1.35.1.0, Culture=neutral, PublicKeyToken=4b01fa6e34db77ab'. An operation is not legal in the current state.

[thepenguinmaster]

I ran into this issue and believe I may have found the cause for "An operation is not legal in the current state.". Please excuse the poor formatting in the response below.

Tools\LambdaTestTool\Amazon.Lambda.TestTool\Runtime\LambdaAssemblyResolver.cs OnResolving Line 79
and more specifically Line 93:

return this.loadContext.LoadFromAssemblyPath(assemblies[0]);

The issue here, is that if an assembly contains multiple runtimes, it will only select the first in the list. In some cases, this is a unix runtime which is not compatible with a windows machine which in my case, is the operating system I am using for development.

This will not cause an issue when deployed because publishing the project renders the dependencies properly.

For example, when a project is compiled for debug, a file named "[projectName].deps.json" is generated.
In this file in my case, I saw this as the first entry for the library I was referencing:

"runtimeTargets": {
    "runtimes/unix/lib/netstandard1.6/Microsoft.Management.Infrastructure.Native.dll": {
    "rid": "unix",
    "assetType": "runtime",
    "assemblyVersion": "1.0.0.0",
    "fileVersion": "1.0.0.0"
},

To verify my suspicions I ran the project, and in visual studio opened debug->windows->modules and could see that the assembly being loaded was in fact incorrect. The code mentioned above was taking only the first dependency in the list, which will not work on a windows platform.

For my needs, I was able to modify LambdaAssemblyResolver.cs OnResolving starting line 78 to the following:

if (library != null){
        var deps = library.RuntimeAssemblyGroups.Distinct().Count() > 1
        ? library.RuntimeAssemblyGroups.Where(x => x.Runtime.StartsWith("win") && x.Runtime.Contains("x64")).SelectMany(g => g.AssetPaths)
        : library.RuntimeAssemblyGroups.SelectMany(g => g.AssetPaths);
        var wrapper = new CompilationLibrary(
                library.Type,
                library.Name,
                library.Version,
                library.Hash,
                deps,
                library.Dependencies,
                library.Serviceable);
        var assemblies = new List<string>();
        this.assemblyResolver.TryResolveAssemblyPaths(wrapper, assemblies);
        if (assemblies.Count > 0)
        {
            return this.loadContext.LoadFromAssemblyPath(assemblies.FirstOrDefault(x => !x.EndsWith(".Native.dll")));
        }
}

I added a filter to pull only runtime versions that are built for win 64 and later, exclude native binaries.

I did not submit this as a PR since it is only 'good enough for me' and my specific case, but something like it in combination with operating system and platform detection would allow for proper assembly loading.

Not sure if this should be escalated to feature request or Bug.

Also if there is anything I can do to contribute to this project, please let me know.

[carlosrfernandez]

Having a similar issue.

We're trying to add Swagger: <PackageReference Include="Swashbuckle.AspNetCore" Version="4.0.1" />

It works in the deployed asp.net app that's behind the API Gateway.

In the test tool however, it throws this exception.

It's very easy to reproduce, standard Serverless project template, add the assembly above, add the services.AddSwaggerGen() and the app.UseSwagger(); lines in the ConfigureServices and Configure methods respectively.

Start app with Mock Lambda tool. Make any request using the API Gateway template from the tool. It will throw the FileNotFoundException

[asmodat]

I got the same issue

System.IO.FileNotFoundException: Could not load file or assembly 'XYZ, Version=1.0.0.32, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.
   at SomeClass.Function..ctor()

where XYZ is .NET Standard 2.0 lib and Lambda project is Core 2.1

[skewl84]

Same Issue

System.IO.FileLoadException: Could not load file or assembly 'Microsoft.Extensions.DependencyInjection, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. Could not find or load a specific file. (Exception from HRESULT: 0x80131621)
   at Namespace.Function..ctor()

[vlade-rc]

Hi I got the same issue, any help to solve this?

[KFayal]

Same issue today with System.Data.SqlClient.dll

[KFayal]

Just curious about this.. but I ran ProcMon and watched the path System.Data.SqlClient.dll - (the file with which I'm having issues in this context) I noticed that twice, when the dotnet-lambda-test-tool-2.1.exe is trying to do CreateFileMapping against System.Data.SqlClient.dll it gets a result of FILE LOCKED WITH ONLY READERS. Which after further research shows that it can cause a "Could not load file or assembly" error. I thought that by then restarting Visual Studio 2019 as Administrator I could get that error to go away, but no such luck. I suspect that others having this issue will also find this issue with the assemblies in the Lambda's they are trying to debug as well. I hope this helps someone. I'm still looking for a workaround.

[KFayal]

Ok, ok, ok.. For me this issue was due to the fact that I had to lower the version of the System.Data.SqlClient Nuget package to 4.5.1. For some reason, anything higher produces that "Could not load file or assembly" error that this thread is about. The version 4.6 and above give me that error when debugging locally with the Lambda debugging tool.

[normj]

Wanted to give a status update. I'm currently working on the .NET Core 3.1 version of this tool for the upcoming .NET Core 3.1 Lambda runtime. The work for that can be tracked in the mock-testtool-31.

In this version I'm using the AssemblyDependencyResolver which was added in .NET Core 3.0 to load the Lambda function in a separate AssemblyLoadContext. That will help these issues if the Lambda function using a different version of an assembly then was loaded by the test tool.

[stevecookms]

How can we test 3.1?

[Edulopez]

I was having the same issue with my super simple template project,
System.IO.FileLoadException: Could not load file or assembly 'Microsoft.Extensions.DependencyInjection.Abstractions, Version=3.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. Could not find or load a specific file. (Exception from HRESULT: 0x80131621) at AWSLambda1.Functions.MyFunction..ctor()
Also with Microsfot.Extensions.Configuration.Abstractions (latest 3.1.3)

After reading all the comments I use the same solution as some of you. I had to downgrade my packages and the ones referenced by them to a lower version. In this case I'm using 2.1.

This is what I have now so my Mock tool can work.

Just to remind some of the facts about the issue.

• Project compiles with no errors
• Test project runs successfully
• Mock tool does not work when code that use the dll is executed.

This is the line that cause the error in my case.
public MyFunction() : this(StartUp.Container.BuildServiceProvider()) {}
This is my Startup.cs

using AWSLambda1.Config;
using AWSLambda1.Services;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;

namespace AWSLambda1
{
    public class StartUp
    {
        public static IServiceCollection Container => ConfigureServices(LambdaConfiguration.Configuration);

        private static IServiceCollection ConfigureServices(IConfigurationRoot root)
        {
            var services = new ServiceCollection();

            var a = root.GetSection("MySection");
            services.Configure<EnvMySection>(options =>
                root.GetSection("MySection").Bind(options));

            var b = new MyFunctionEnvironment() { Something = LambdaConfiguration.Configuration["Hello"] ?? "" };
            services.AddSingleton(b);
            services.AddTransient<IMyService, MyService>();
            return services;
        }
    }
}

[normj]

Version 0.10.0 was released today for both .NET Core 2.1 and .NET Core 3.1 which now loads the Lambda code in a separate AssemblyLoadContext. I believe that will fix a lot of the reported issues here. I'm going to close this issue as it is old and I believe there are several separate issues. After using the new version if there is still an issue please open a separate issue to make it easier to track.

[vlade-rc]

Thanks @normj and congratulations, I'll test

[Edulopez]

@normj
Hey there. I Had the same issues 6 days ago. I tried to cover all the info I had in the last post of this thread. I don't think this issue is solved yet until we got proof.

[vlade-rc]

Ar you tested with 3.1?

[normj]

@Edulopez Have you tried the latest version that came out yesterday? If the issue does still exist then please create a separate issue with your repo. I just felt this thread had gotten to unwieldy and I do suspect many reported issues will be fixed with the new version.