Azure-functions-host: Reading from DocumentDB in Cosmos DB fails frequently with this error

[dfinke]

This payload is written by a different function and rarely fails. We're trying to figure out how to get better diagnostics to solve the problem.

• Invocation ID: b648680e-3c56-4da7-8f87-3a53a6d3bd9a
• Region: East US 2

StartOpportunitySearch.Run: Starting search for opportunity report phrase: abcd, execution context id: b648680e-3c56-4da7-8f87-3a53a6d3bd9a
Exception while executing function: Functions.StartOpportunitySearch
Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.StartOpportunitySearch ---> System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation. ---> System.AggregateException : One or more errors occurred. ---> An error occurred while sending the request. ---> Unable to connect to the remote server ---> An attempt was made to access a socket in a way forbidden by its access permissions
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout,CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at Aimee.Research.Phrases.Domain.PhraseService.GetByKey(String phrase,String channel)
   at Aimee.Research.Opportunities.Domain.OpportunityGenerator.StartOpportunitySearch(CreateOpportunityParameter parameter)
   at Aimee.Research.Services.EngineFunctions.Application.CalcOpportunityStats.Run(CreateOpportunityParameter parameter,ExecutionContext executionContext,TraceWriter log)
---> (Inner Exception #0) System.Net.Http.HttpRequestException : An error occurred while sending the request. ---> System.Net.WebException : Unable to connect to the remote server ---> System.Net.Sockets.SocketException : An attempt was made to access a socket in a way forbidden by its access permissions
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot,SocketAddress socketAddress)
   at System.Net.Sockets.Socket.InternalBind(EndPoint localEP)
   at System.Net.Sockets.Socket.BeginConnectEx(EndPoint remoteEP,Boolean flowContext,AsyncCallback callback,Object state)
   at System.Net.Sockets.Socket.UnsafeBeginConnect(EndPoint remoteEP,Async…

[anthonychu]

What does the function app do and how often are functions being executed? That error might indicate a problem with socket exhaustion under high load.

[dfinke]

The functions are all queued, and they are sending requests to a webservice fire and forget.
They save the dummy record in document db of the request that is filled later.

The socket exhaustion error/exception is when we are trying to read from document db not related to the sockets in web job.

[tohling]

@dfinke, I checked the logs and it seems that your Function ran out of outbound connection sockets. Is your Function implementation creating its own DocumentDB client? If yes, could you kindly see if you can leverage the DocumentDB client that is built-in as part of an input-binding instead? See https://github.com/Azure/azure-webjobs-sdk-script/issues/984

[ericjohnolson]

@tohling I tried to create an input-binding to get the document client but am getting a bunch of errors trying to bind it. For this use case the function runs and reads some data from an api and saves many documents to one collection. From what I can see from that binding it requires a document ID for an input binding, and I don't have that yet. Also I am using the Visual Studio 2017 Azure Function Tools and leveraging the attributes for input bindings, do you have an example of using a DocumentDb client that way? Thanks.

[dersia]

Actually Socket exhaustion is a big problem, especially if you're using 3rd party libs and those use HttpClient not statically and so not managed by the OS. I had a few customers running into this issue and I know, that there is solution in the documents, explaining the issue and suggesting to use HttpClient as a static resource than with the using pattern, but you'll never know, if all your 3rd party libs are doing it correctly.

I think it would make sense to have a perf-counter on open sockets while scaling functions and then spawning a new function app when a specific threshold is reached, at least for the consumption plan. But I'm not sure how this would affect the billing metrics and if this is a feasible solution, but it would solve a lots of issues...

[brettsam]

@ericjohnolson -- your issue looks unrelated. Would you mind opening a new issue and share what your function signature looks like? I can help you narrow down the problem there.

[ericjohnolson]

@brettsam Sorry Doug and I work together so we are talking about the same issue.

[ericjohnolson]

@brettsam I also just looked through the github issues and see that you have lots of experience with the socket exhaustion problem. I keep trying to change how I am doing the processing, but it keeps pushing the issue of running out of sockets to another function. I see this is due to a hard limit of 250 concurrent socket connections. Would this scale better using the dedicated server plan instead of consumption? I was under the probably wrong assumption that the consumption plan scaled better.

What I am trying to do is process a set of 2000 searches that each return about 100 related terms from a call to an API. Each of these 200,000 results are saved in a document db collection and sent to another service that processes some data and pushes a notification when it is done, where a function wakes up to crunch the and save the results.

Just trying to process this relatively small load I get socket exhaustion errors every step of the way, either when trying to do the initial REST API call, the initial save to a document client, or the subsequent saving to the document client when the result completes. I create my own HttpClients and DocumentClients but also make sure to dispose of them after each use.

Is there any way to have the system manage scaling in a way that doesn't hit those limits? I'm am willing to re-architect the system in whatever way is most performant, so any suggestions would be greatly appreciated.

[brettsam]

Try not disposing of your HttpClient and DocumentClient. They're built to be re-used and creating new ones adds to the problem. Instead, create static instances in your function and re-use them for each function invocation. See if that helps at all.

Are your 2000 searches going to the same host, or are there lots of different hosts? If it's relatively few hosts, then you may be able to make it work with the 250 socket limit. If it's a lot of hosts, then you'll hit the socket limit in Dynamic plans if you issue them all at once.

If you ran this code in a dedicated plan (Basic or higher), you wouldn't have that limit imposed and the calls will likely work. See this chart for the details (see Connections): https://github.com/projectkudu/kudu/wiki/Azure-Web-App-sandbox#per-sandbox-per-appper-site-numerical-limits

[ericjohnolson]

@brettsam Thanks for the feedback!

[christopheranderson]

Please reopen if you still see this issue after implementing brett's suggetions

[CANUSA]

Have you folks found a solution to this? I see the socket exception quite often and am struggling to find a solution. It mainly manifests as my functions scale under load. Not disposing of your HttpClient and/or DocumentClient objects seems wrong.

[brettsam]

Those clients are designed to be re-used, so it's recommended that you have a single static instance that you re-use across all functions. Some links to explain why:

HttpClient:

• https://aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/. This one specifically explains the connection exhaustion issue.
• https://stackoverflow.com/a/15708633/3516125
• https://stackoverflow.com/a/22561368/3516125

DocumentClient:

• https://stackoverflow.com/a/43422615/3516125
• https://docs.microsoft.com/en-us/azure/cosmos-db/performance-tips#sdk-usage