Aws-lambda-dotnet: Lambda suddenly started to throw error

[genifycom]

I was running a lambda ASP.NET Core application successfully when suddenly it started to thow:

Could not find file '/var/lang/Microsoft.NETCore.App.deps.json'.: FileNotFoundException

I was having a couple of people try hitting it at the same time. Is this to do with starting a new instance?

Thanks,
Dave

[gokarnm]

Can you provide the full exception stack trace? How frequently does this occur? Which region did you use?

[automagic]

I have also been getting this since yesterday!! Thought I was going crazy... its tentatively working for me again as of evening 8:30 PST

Was happening in us-west-2 for me. It was almost completely constant. I have been combing through library versions thinking there was some sort of mismatch. The stack-trace was pretty useless- happens in services.UseMvcCore(). I am pretty convinced now its an AWS issue since i'm not the only one.

[gokarnm]

@automagic , thanks for the details! Could you share the stack trace, it would be helpful to diagnose the issue.

[automagic]

An exception was thrown when the constructor for type 'PlatformAPI.LambdaFunction' was invoked. Check inner exception for more details.: LambdaException


at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck)
at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)
at System.Activator.CreateInstance(Type type, Boolean nonPublic)
at System.Activator.CreateInstance(Type type)
Could not find file '/var/lang/Microsoft.NETCore.App.deps.json'.: FileNotFoundException
at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
at Interop.CheckIo[TSafeHandle](TSafeHandle handle, String path, Boolean isDirectory, Func`2 errorRewriter)
at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
at System.IO.UnixFileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, FileStream parent)
at System.IO.UnixFileSystem.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, FileStream parent)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
at Microsoft.Extensions.DependencyModel.FileWrapper.OpenRead(String path)
at Microsoft.Extensions.DependencyModel.DependencyContextLoader.LoadRuntimeContext(IDependencyContextReader reader)
at Microsoft.Extensions.DependencyModel.DependencyContextLoader.Load(Assembly assembly)
at Microsoft.AspNetCore.Mvc.Internal.DefaultAssemblyPartDiscoveryProvider.DiscoverAssemblyParts(String entryPointAssemblyName)
at Microsoft.Extensions.DependencyInjection.MvcCoreServiceCollectionExtensions.GetApplicationPartManager(IServiceCollection services)
at Microsoft.Extensions.DependencyInjection.MvcCoreServiceCollectionExtensions.AddMvcCore(IServiceCollection services)
at PlatformAPI.Startup.ConfigureServices(IServiceCollection services)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.ConfigureServices(IServiceCollection services)
at Microsoft.AspNetCore.Hosting.Internal.WebHost.EnsureApplicationServices()
at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()
at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
at Amazon.Lambda.AspNetCoreServer.APIGatewayProxyFunction..ctor()
at PlatformAPI.LambdaFunction..ctor()

[gokarnm]

Thanks for the stack trace! We are looking into this issue.
@genifycom , @automagic , would you be able to share a trimmed down/sample ASP.NET Core app that triggers this error?

[automagic]

I will look into getting you a sample app tomorrow morning.

[pfeurean]

region:us-east-1

I can force the exception. Subsequent calls to the function will fail until I update an environment variable, forcing a reload of the "machine?".

An exception was thrown when the constructor for type 'Swell.AspNetCore.LambdaEntryPoint' was invoked. Check inner exception for more details.: LambdaException

at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck)
at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)
at System.Activator.CreateInstance(Type type, Boolean nonPublic)
at System.Activator.CreateInstance(Type type)
Could not find file '/var/lang/Microsoft.NETCore.App.deps.json'.: FileNotFoundException
at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func2 errorRewriter) at Interop.CheckIo[TSafeHandle](TSafeHandle handle, String path, Boolean isDirectory, Func2 errorRewriter)
at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
at System.IO.UnixFileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, FileStream parent)
at System.IO.UnixFileSystem.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, FileStream parent)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
at Microsoft.Extensions.DependencyModel.FileWrapper.OpenRead(String path)
at Microsoft.Extensions.DependencyModel.DependencyContextLoader.LoadRuntimeContext(IDependencyContextReader reader)
at Microsoft.Extensions.DependencyModel.DependencyContextLoader.Load(Assembly assembly)
at Microsoft.AspNetCore.Mvc.Internal.DefaultAssemblyPartDiscoveryProvider.DiscoverAssemblyParts(String entryPointAssemblyName)
at Microsoft.Extensions.DependencyInjection.MvcCoreServiceCollectionExtensions.GetApplicationPartManager(IServiceCollection services)
at Microsoft.Extensions.DependencyInjection.MvcCoreServiceCollectionExtensions.AddMvcCore(IServiceCollection services)
at Microsoft.Extensions.DependencyInjection.MvcServiceCollectionExtensions.AddMvc(IServiceCollection services)
at Microsoft.Extensions.DependencyInjection.MvcServiceCollectionExtensions.AddMvc(IServiceCollection services, Action1 setupAction) at Microsoft.Extensions.DependencyInjection.IServiceCollectionExtensions.AddApiVersioning(IServiceCollection services, Action1 setupAction)
at Swell.AspNetCore.Startup.ConfigureServices(IServiceCollection services)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.ConfigureServices(IServiceCollection services)
at Microsoft.AspNetCore.Hosting.Internal.WebHost.EnsureApplicationServices()
at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()
at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
at Amazon.Lambda.AspNetCoreServer.APIGatewayProxyFunction..ctor()

[automagic]

@pfeurean exactly how mine acts. It kills the function until a new container is instantiated (Lambda is just Docker underneath)

Also I have an enterprise support ticket out with AWS about this, so I will keep this thread updated with the response there as well.

[DerrickF]

I've got this same issue. It seems to work for a period of time right after deployment but after a few mins it starts throwing the same FileNotFoundException that everyone is having. Hopefully this gets sorted out soon.

[JNBarnes]

I'm seeing the exact same behavior. I've been pulling my hair out over for it for > 24 hours now.

Stack trace here:

An exception was thrown when the constructor for type 'LeadEngineASP.LambdaEntryPoint' was invoked. Check inner exception for more details.: LambdaException
at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck)
at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)
at System.Activator.CreateInstance(Type type, Boolean nonPublic)
at System.Activator.CreateInstance(Type type)
Exception has been thrown by the target of an invocation.: TargetInvocationException
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Microsoft.AspNetCore.Hosting.Internal.ConfigureServicesBuilder.Invoke(Object instance, IServiceCollection exportServices)
at Microsoft.AspNetCore.Hosting.Internal.WebHost.EnsureApplicationServices()
at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()
at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
at Amazon.Lambda.AspNetCoreServer.APIGatewayProxyFunction..ctor()
Could not find file '/var/lang/Microsoft.NETCore.App.deps.json'.: FileNotFoundException
at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
at Interop.CheckIo[TSafeHandle](TSafeHandle handle, String path, Boolean isDirectory, Func`2 errorRewriter)
at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
at System.IO.UnixFileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, FileStream parent)
at System.IO.UnixFileSystem.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, FileStream parent)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
at Microsoft.Extensions.EnvironmentAbstractions.FileWrapper.OpenRead(String path)
at Microsoft.Extensions.DependencyModel.DependencyContextLoader.LoadRuntimeContext(IDependencyContextReader reader)
at Microsoft.Extensions.DependencyModel.DependencyContextLoader.Load(Assembly assembly)
at Microsoft.AspNetCore.Mvc.Internal.DefaultAssemblyPartDiscoveryProvider.DiscoverAssemblyParts(String entryPointAssemblyName)
at Microsoft.Extensions.DependencyInjection.MvcCoreServiceCollectionExtensions.GetApplicationPartManager(IServiceCollection services)
at Microsoft.Extensions.DependencyInjection.MvcCoreServiceCollectionExtensions.AddMvcCore(IServiceCollection services)
at Microsoft.Extensions.DependencyInjection.MvcServiceCollectionExtensions.AddMvc(IServiceCollection services)
at LeadEngineASP.Startup.ConfigureServices(IServiceCollection services)

[gokarnm]

We are actively working this issue and will have a fix soon. @genifycom , @DerrickF , @JNBarnes which regions are you using?

[JNBarnes]

@gokarnm eu-west1. I'm putting together an SCCE for you to help repro it too...

[DerrickF]

@gokarnm us-west2

[automagic]

AWS Support is also aware (from my ticket):

Hello,

The Lambda team is aware of the issue and currently taking steps to address it. A fix should be applied shortly.

[lthompson-cartasite]

We are experiencing the same issue (since 23:33 UTC) in US-East-1

An exception was thrown when the constructor for type 'redacted' was invoked. Check inner exception for more details.: LambdaException

at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck)
at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)
at System.Activator.CreateInstance(Type type, Boolean nonPublic)
at System.Activator.CreateInstance(Type type)
Exception has been thrown by the target of an invocation.: TargetInvocationException
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Microsoft.AspNetCore.Hosting.Internal.ConfigureServicesBuilder.Invoke(Object instance, IServiceCollection exportServices)
at Microsoft.AspNetCore.Hosting.Internal.WebHost.EnsureApplicationServices()
at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()
at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
at Amazon.Lambda.AspNetCoreServer.APIGatewayProxyFunction..ctor()
Could not find file '/var/lang/Microsoft.NETCore.App.deps.json'.: FileNotFoundException
at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func2 errorRewriter) at Interop.CheckIo[TSafeHandle](TSafeHandle handle, String path, Boolean isDirectory, Func2 errorRewriter)
at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
at System.IO.UnixFileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, FileStream parent)
at System.IO.UnixFileSystem.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, FileStream parent)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
at Microsoft.Extensions.EnvironmentAbstractions.FileWrapper.OpenRead(String path)
at Microsoft.Extensions.DependencyModel.DependencyContextLoader.LoadRuntimeContext(IDependencyContextReader reader)
at Microsoft.Extensions.DependencyModel.DependencyContextLoader.Load(Assembly assembly)
at Microsoft.AspNetCore.Mvc.Internal.DefaultAssemblyPartDiscoveryProvider.DiscoverAssemblyParts(String entryPointAssemblyName)
at Microsoft.Extensions.DependencyInjection.MvcCoreServiceCollectionExtensions.GetApplicationPartManager(IServiceCollection services)
at Microsoft.Extensions.DependencyInjection.MvcCoreServiceCollectionExtensions.AddMvcCore(IServiceCollection services)
at Microsoft.Extensions.DependencyInjection.MvcServiceCollectionExtensions.AddMvc(IServiceCollection services)
at wv3.fieldFlowAPI.Startup.ConfigureServices(IServiceCollection services)

[gokarnm]

A fix has been released for Lambda to address this issue. Please update this thread if you still encounter this error.

[automagic]

Thanks! Any chance you could give us any technical insight into what went wrong on the lambda end?

[gokarnm]

We are working on an in-depth analysis of what went wrong and how to prevent this in the future, there should be an update with details in a week.

[gokarnm]

Hi @automagic , @JNBarnes , I've sent a PM over gitter to get some details about your application.

[gokarnm]

Here are the details of the bug, closing this issue.

Starting 5/9/2017, AWS Lambda deployed an update for .NET Core 1.0 runtime. The update allowed libraries like Microsoft.Extensions.DependencyModel to pick up the correct path for application and framework dependencies files (.deps.json). An example of usage of the dependency files is the ASP.NET Core Razor pages that are dynamically compiled, which uses these files to discover the dependencies at runtime.

The _Microsoft.NETCore.App.deps.json_ file, which is the framework dependency file, was missing in the runtime environment, which caused code paths in ASP.NET applications to fail. We got an initial report of this bug on 5/11/2017 on this GitHub issue, and received additional reports through AWS Forums and AWS Support tickets on 5/12/2017. After the issue was reported, we started a rollback of the update on 5/12/2017 and completed rollback in all regions by 5/13/2017. The root cause was identified to be a bug that resulted in the _Microsoft.NETCore.App.deps.json_ file to be absent in the runtime environment. An internal review of the events that caused this issue, and corrective actions is underway. As part of corrective actions, we are adding additional tests and checks in place to prevent similar issues in the future.

[sondlerd]

.NetCore 2.0 Lambda test (console.aws.amazon.com) throws: "Could not find the required '[AppName].deps.json'. This file should be present at the root of the deployment package." It looks like this was fixed for .netCore 1.0. Is it a problem for .netCore 2.0? Nope... I just switched my Target Framework to .net core 1.0 and get the same error. So what is going on here?
I see that the tutorial I'm using was created via Visual Studio 2015 and I'm using the AWS toolkit for Visual Studio 2017. I'm going to try to recreate my project using Visual Studio 2015 and the AWS Toolkit for VS 2013- 2015.

[vellozzi]

Is that file in the deployment package?
Do you have the following line in your csproj file?

<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>

That property in the csproj will include the deps.json file in your publish directory and your deployment package.

[vellozzi]

Sorry. I confused the runtime.json and the deps.json files.

It's likely that the handler string is configured incorrectly.
Check out the Handler Signatures section in https://docs.aws.amazon.com/lambda/latest/dg/dotnet-programming-model-handler-types.html . Make sure there are no typos in your handler string. In this case it's likely a problem in the assembly name.

[sondlerd]

Thanks for the quick response. I have checked the handler string in the AWS Lambda Console and I don't see a mistake. I went back to the configuration page and copied and pasted all of the info, paying attention to the syntax and nothing changed. After I upload it to AWS, I can run a test in Visual Studio and I get a successful response. I also see that the [AppName].deps.json file in the bin\Debug\netcoreapp2.0 folder with the assembly. However, it still fails when I run a test in the AWS Lambda test console.

[vellozzi]

So it works in Visual Studio but not the console? Hm, that's a tough one.
Does the console give you the same error you were getting before? Did you test it successfully in Visual Studio before testing in the console? The the bootstrapping process only happens on cold start. So if it works correctly once it shouldn't run that part of the bootstrap again (normally) if you try it it immediately after. Are you 100% sure you're testing the same function in the console that you're testing from Visual Studio?

[sondlerd]

I fixed it by recreating the project under a different name. It looks like I had inconsistent type usage between the Alexa.Net.Requests and Slight.Alexa.Framework.Models.Requests assemblies. It was my fault.
The odd thing is that the Visual Studio upload test worked... very odd. The good news is that I'm able to get past this problem and move on with development. Thanks so much for your time.

[vellozzi]

Cool. I'm glad it's resolved.