Aws-lambda-dotnet: Process exited before completing request

[betimd]

After updating AWS Tools in Visual Studio 2015, when I update lambda (that has dynamodb triggers attached) from VS, function stopped working and in cloud watch I get this line
"Process exited before completing request"
(RequestId: ffe0bbe3-1c91-4f82-b9f7-d1c40e4bfc41 Process exited before completing request).

Same error is happening in VS 2017.

After fighting with this error for hours I dropped function in console and recreated from VS by publishing it and it worked.

If I wanted to update same error happened.

[betimd]

Now I noticed that I have to manually invoke from VS to make it work.
So, when I upload an update to function, it is not working if I do not invoke it from VS.

[betimd]

Is the issue related to this one described here (event that's in nodejs) https://forums.aws.amazon.com/message.jspa?messageID=739341

[normj]

The error indicates that something is causing the .NET runtime to crash. Is there anything in your code or dependencies that would trigger an Environment.Exit() or a stack overflow?

[betimd]

Here is all scenario, CW logs & code (simplified).

15:04:16 START RequestId: 95bc39ec-399b-4562-ae79-1aa9672acbab Version: $LATEST
15:04:16 Beginning to process 1 records
15:04:16 Starting to process sensor data
15:04:16 Get cache post data item.
15:04:16 Initiating connection
15:04:18 END RequestId: 95bc39ec-399b-4562-ae79-1aa9672acbab
15:04:18 REPORT RequestId: 95bc39ec-399b-4562-ae79-1aa9672acbab Duration: 2080.11 ms    Billed Duration: 2100 ms Memory Size: 960 MB    Max Memory Used: 84 MB
15:04:18 RequestId: 95bc39ec-399b-4562-ae79-1aa9672acbab Process exited before completing request

public void FunctionHandler(DynamoDBEvent dynamoEvent, ILambdaContext context)
{
    var logger = context.Logger;

    logger.LogLine($"Beginning to process {dynamoEvent.Records.Count} records");

    try
    {
        foreach (var record in dynamoEvent.Records)
        {
            //we need just insert operation
            if (record.EventName != OperationType.INSERT || record.Dynamodb.NewImage == null)
                continue;

            var obj = ToObject(record.Dynamodb.NewImage);
            ProcessSensorData(obj.RegistrationDateTime, obj.Request).Wait();
            DynamoDbContext.DeleteAsync<DevicePostItem>(obj.PostId).Wait();
        }
    }
    catch (Exception e)
    {
        Console.WriteLine(e);
    }
}


private async Task ProcessSensorData(long registrationDate, string comingRawData)
{
    LambdaLogger.Log("Starting to process sensor data");

    var sensorData = SerializationHelper.DeserializeSensorDataToObject(comingRawData);

    if (sensorData == null) return;

    //huh, I need this field a lot
    var deviceId = sensorData.DeviceId;

    try
    {
        //previous post cached item
        //when code is executing this method is dying, staying there forever 
        //but when I invoke once manually, it has no problem then
        var previousItemStr = _sensordataAccess.GetCachePostSensorDataItem(deviceId);
        var previousItemObj = SimpleSerializationHelper.ToObject<SensorDataItem>(previousItemStr);

        var deviceDetails = _deviceDataAccess.GetDeviceDetailsForProcessor(deviceId);

        //....

        Console.WriteLine($"{sensorDataItem.RecordId}");
    }
    catch (Exception ex)
    {
        Console.WriteLine($"{deviceId}|{registrationDate} \n{comingRawData}");
        Console.WriteLine(ex);
    }
}

public class CacheDbDataAccess : BaseDataAccessTable
{
    public CacheDbDataAccess(TivePlatformConfig config) : base(config) { }

    public string GetCachePostSensorDataItem(string deviceId)
    {
        Console.WriteLine("Get cache post data item.");

        try
        {
            string dataPost;

            using (var cnn = new SqlConnection(CacheDbConnectionString))
            {
                Console.WriteLine("Initiating connection");

                var result = cnn.Query<string>("spGetSensorDataPostCache", new { DeviceId = deviceId.Trim() },
                    commandType: CommandType.StoredProcedure).FirstOrDefault();

                dataPost = result;
                Console.WriteLine("Got results");
            }
            Console.WriteLine("returning result");
            return dataPost;
        }
        catch (Exception exception)
        {
            Console.WriteLine(exception);
            return string.Empty;
        }
    }
}

So when I update the function from VS toolkit (right click publish) it halts in that place where I call db (Sql Server - RDS) using Dapper (cnn.Query...). The trick makes to work is, a single Invoke from VS or AWS Console, and then code continues to work. This is happening to me since yesterday, after VS2017 update.

[betimd]

Basically Lambda function works completely fine once being updated and then triggered manually, but doesn't work if hasn't been triggered manually after update.
So if Lambda was updated but wasn't triggered manually it gets timeout trying access RDS instance all the time.

[AndreiMarukovich]

Have the same issue - trying to access RDS MySQL and Lambda returns "Process exited..." if connection.Open() is calles (connection is MySqlConnection).

Same code perfectly works on the local machine - establishes the connection and reads the data.

[timdinh-grex]

I encountered the same error. I managed to work out that it actually crashed the .net process if you include the nuget package Microsoft.IdentityModel.Tokens version 5.1.4 in the .csproj (i'm using VS2017). Here's what i did to come to that conclusion

1. Create a new C# AWS Serverless Web API from the provided template
2. Add the nuget package Microsoft.IdentityModel.Tokens
3. Build and deploy to AWS lambda
4. Send a png file to the API via the S3Proxy endpoint
5. Check CloudWatch log

[normj]

Version 5.1.4 of Microsoft.IdentityModel.Tokens depends on .NET Core 1.1 libraries. Which is why it didn't work. I am surprised it actually killed the process through.

[nanestev]

Same error is thrown if your Lambda runs out of memory. I know it's obvious in the logs, but it still took me a while before realizing that and wasted time checking libraries for incompatibility etc.

[AndreiMarukovich]

In my case the issue was with the official MySQL library. Replacing it with one of the alternatives completely solved the problem.

[wtesler]

@nanestev That was my problem! Error could be more descriptive for sure.

[normj]

Closing this as it look likes the issues have been resolved. In the next iteration of .NET Core Lambda functions we hope to use some of the new API in .NET Core to capture what happens when the process dies and report that back in the logs.

[dzfweb]

@normj is there a way to capture what happens when the process dies?