Aws-lambda-dotnet: Unable to connect to RDS/Postgres in Lambda but works fine locally

[NatElkins]

Hi,

I am trying to connect to Postgresql using Npgsql. What I have works fine on my local machine, but not when deployed to Lambda.

The exception I'm getting looks similar to the following, depending on where I try and open the connection:

Lambda encountered an UnobservedTaskException via 'TaskScheduler.UnobservedTaskException' event:
{
    "errorType": "AggregateException",
    "errorMessage": "A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Connection timed out 18.206.134.251:5432)",
    "cause": {
        "errorType": "ExtendedSocketException",
        "errorMessage": "Connection timed out 18.206.134.251:5432",
        "stackTrace": [
            "at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()",
            "at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)",
            "at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)",
            "--- End of stack trace from previous location where exception was thrown ---",
            "at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()",
            "at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)",
            "at Npgsql.TaskExtensions.<WithCancellation>d__3.MoveNext()"
        ]
    },
    "causes": [{
        "errorType": "ExtendedSocketException",
        "errorMessage": "Connection timed out 18.206.134.251:5432",
        "stackTrace": [
            "at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()",
            "at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)",
            "at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)",
            "--- End of stack trace from previous location where exception was thrown ---",
            "at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()",
            "at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)",
            "at Npgsql.TaskExtensions.<WithCancellation>d__3.MoveNext()"
        ]
    }]
}

I tried adding

  "additional-files": [
    "runtimes/linux/lib/netstandard1.3/System.Net.NetworkInformation.dll"
  ]

to my aws-lambda-tools-defaults.json file but that did not work. I also tried creating a custom package and uploading it as described here: http://www.philliphaydon.com/2017/03/21/part4-building-and-publishing-a-dotnet-core-app-using-aws-tools/

but that did not work either.

This is a test DB that is publicly accessible. The connection string is hard coded. And like I said before, this works just fine from my local machine. What can I look into next?

Any help you can provide would be greatly appreciated. Please let me know if I can provide any additional information.

[NatElkins]

BTW, I am targeting netcoreapp2.0, and I'm using Npgsql 4.0.

Here's another stack trace that might contain more information:

Error] Giraffe.Middleware.GiraffeErrorHandlerMiddleware: Unhandled exception.|ex=System.AggregateException: One or more errors occurred. (The operation has timed out.) ---> System.TimeoutException: The operation has timed out.
at Npgsql.TaskExtensions.<WithTimeout>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.NpgsqlConnector.<ConnectAsync>d__155.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.NpgsqlConnector.<RawOpen>d__153.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.NpgsqlConnector.<Open>d__149.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.ConnectorPool.<AllocateLong>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.NpgsqlConnection.<>c__DisplayClass32_0.<<Open>g__OpenLong|0>d.MoveNext()
--- End of inner exception stack trace ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.FSharp.Control.AsyncBuilderImpl.commit[a](AsyncImplResult`1 res)
at Microsoft.FSharp.Control.CancellationTokenOps.RunSynchronouslyInCurrentThread[a](CancellationToken token, FSharpAsync`1 computation)
at Microsoft.FSharp.Control.CancellationTokenOps.RunSynchronously[a](CancellationToken token, FSharpAsync`1 computation, FSharpOption`1 timeout)
at Microsoft.FSharp.Control.FSharpAsync.RunSynchronously[T](FSharpAsync`1 computation, FSharpOption`1 timeout, FSharpOption`1 cancellationToken)
at Masse.SearchProduct.Service.makeSem3idProductRequests(String[] sem3Ids) in /home/nat/Projects/shopmasse-backend/fsharp/src/Masse.SearchProduct/Service.fs:line 94
at Masse.SearchProduct.Service.handle$cont@250(FSharpFunc`2 makeSearchProductRequest, FSharpFunc`2 makeAsinProductRequests, FSharpFunc`2 makeSem3idProductRequests, Input input, Unit unitVar) in /home/nat/Projects/shopmasse-backend/fsharp/src/Masse.SearchProduct/Service.fs:line 282
at Masse.Backend.App.SearchProduct.continuation@1(FSharpFunc`2 next, HttpContext ctx, String _arg1) in /home/nat/Projects/shopmasse-backend/fsharp/src/Masse.Backend/App.fs:line 47
at FSharp.Control.Tasks.TaskBuilder.StepStateMachine`1.nextAwaitable()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at [email protected](Unit unitVar0)
at FSharp.Control.Tasks.TaskBuilder.StepStateMachine`1.nextAwaitable()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at [email protected](Unit unitVar0)
at FSharp.Control.Tasks.TaskBuilder.StepStateMachine`1.nextAwaitable()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at [email protected](Unit unitVar0)
at FSharp.Control.Tasks.TaskBuilder.StepStateMachine`1.nextAwaitable()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at [email protected](Unit unitVar0)
at FSharp.Control.Tasks.TaskBuilder.tryWith[a](FSharpFunc`2 step, FSharpFunc`2 catch)
---> (Inner Exception #0) System.TimeoutException: The operation has timed out.
at Npgsql.TaskExtensions.<WithTimeout>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.NpgsqlConnector.<ConnectAsync>d__155.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.NpgsqlConnector.<RawOpen>d__153.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.NpgsqlConnector.<Open>d__149.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.ConnectorPool.<AllocateLong>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.NpgsqlConnection.<>c__DisplayClass32_0.<<Open>g__OpenLong|0>d.MoveNext()<---

[NatElkins]

In that other issue, you suggested adding

  "additional-files": [
    "runtimes/linux/lib/netstandard1.3/System.Net.NetworkInformation.dll"
  ]

Where does that path come from? Is that an AWS Lambda specific thing?

[NatElkins]

I've also created a StackOverflow issue here: https://stackoverflow.com/questions/50744391/npgsql-with-netcoreapp2-0-on-aws-lambda-is-unable-to-connect-has-timeout-how

[vellozzi]

It looks like you're having VPC/network/lambda function configuration issues. Check this out and see if it helps: https://docs.aws.amazon.com/lambda/latest/dg/vpc.html

[NatElkins]

@vellozzi Thanks for chiming in. The lambda has this execution role: AWSLambdaVPCAccessExecutionRole

Wouldn't that give it access to the VPC? Or do I need to add something more explicit?

BTW, the DB is publicly accessible. I'm able to access it just fine from Datagrip without being on any VPC.

I admit though, I am an AWS newbie.

[NatElkins]

Looks like this was security group related. Turns out when you set up a Postgres instance on RDS, the IP of the computer you set it up with is automatically whitelisted. Had to add an inbound rule to the security group allowing other connections. Guess it was PEBCAK this time :)

[KPatel91]

@NatElkins - thank you. I had been banging my head for ages and reading this comment made me have a "oh damn" moment. As it happens the latency is so bad I'm going the VPC route now anyway but still you saved me a lot of time buddy.

[NatElkins]

@KPatel91 ha, I remember that feeling. Pretty sure I had at least two days of head banging myself. Moral of the story: read the docs closely, and when you find a solution post it online so only one person has to suffer :)