Azure-webjobs-sdk: [AppInsights] -- Some background calls are appearing as dependency failures

[brettsam]

@lmolkova FYI.

Ideally these calls would be filtered out as they're part of our host operations and not technically failures. But they happen in the context of the function execution, so they're appearing today. Most people using AI should disable the dashboard, but filing this to track as multiple people have mentioned it.

See https://github.com/Azure/azure-functions-host/issues/3363#issuecomment-417759496.

[davemurphysf]

FYI, I deleted the AzureWebJobsDashboard Application Setting and the dependency errors went away.

[lmolkova]

@brettsam is there anything special about this calls that we may filter based on? Or can we add another scope to dashboard calls to distinguish them from 'user' calls?

[brettsam]

We probably want to add something more generic to cover non-dashboard calls too (I see another one with Singleton that I need to narrow down) -- like maybe a "HostCall" scope value. If that thing is present, it wins out and we ignore the call for dependency tracking. We can then use that wherever we see something like this happening in the scope of an executing function.

[harshithkashyap]

@brettsam I found a couple of more issues with the 409 errors.

1. Only HTTP triggered functions are running. Queue and Timer triggered functions do not start. Although manually triggering the queue/timer functions using /admin/functions endpoint works.
2. Functions runtime keeps spawning new instances even when there is no load on the application.

I've emailed you the name of our function app.

[brandonh-msft]

Add me to the list of those experiencing 4xx errors from storage associated w/ a V2 app. So far I've seen 404, 409, and 412 (had to look that one up! 😃)

Am able to repro the 404 by simply creating a v2 HTTP Trigger function in the portal and hit it with postman. 409/412 have been hit by Durable Function executions (Note these would not be rid of by simply removing the Dashboard app setting)

[harshithkashyap]

@brettsam Any update on this issue? We are still seeing the issue on the GA release. I'm curious to see this issue added to Backlog milestone and wasn't included in the GA release.

[ianrathbone]

Can confirm also seeing a bunch of 409 PUT errors from storage on a V2 app (Runtime version: 2.0.12115.0).

Container: azure-webjobs-hosts/locks/

[muratboduroglu]

I am also on the same boat. Just switched a sample benchmark app V1 to V2 (that's built to check performance under heavy load in durable functions so it's doing a lot of http requests) and currently seeing a huge number of httprequest failures, including 404's, 409's and socket connection errors. It almost looks like it's not just an issue of dependency failures, but the socket limit being hit for http requests and causing further requests to fail instantly.

Sample socket error msg:
"Exception message An attempt was made to access a socket in a way forbidden by its access permissions"

Currently the test basically fails instantly, all the invocations fall back to retry (via storage queues) and the connected server list changes to "No servers online".

The same code is working fine in V1.

[brettsam]

@muratboduroglu -- this seems unrelated to the dependency logging. Can you create a new issue and possibly share your benchmark app? I'd specifically be interested if you are using an HttpClient directly or using some other SDK to make your requests.

[muratboduroglu]

Just to follow up, after a discussion with @brettsam and creating a sample app, the issue I have been having turned out to be socket limitation issue.

Our own code is setting the connection limit of the HttpClient in the handler, but durable tasks is making a high level of requests to azure storage and as .Net Core ServicePointManager settings don't affect HttpClient, and there is no way of setting this for the SDK the function can not control the number of connections requested, which causes the App Service sandbox to kick in and reject the socket access.

[brettsam]

Can anyone following this issue let me know -- are you always seeing this in the QuickPulse (Live Metrics) stream? Or are you also seeing these errors flow into your "real" App Insights logs? If you query for them in App Insights Analytics, for example, do they show up?

The reason I ask -- we don't filter anything flowing to QuickPulse so you can see everything happening. But then we apply a filter just after these are sent. We did this originally so you can, for example, disable all App Insights logging (meaning you want nothing to be stored long-term) yet still see logs in the QuickPulse view so you can still get a live view of what's happening. I think all the screenshots I've seen are from this view, so I may have to move our dependency filter forward in the pipeline so they're not so confusing...

[NickMoores]

Hi @brettsam . For us, these are appearing in both QuickPulse and the real App Insights logs (as dependency failures).

What you're saying about QuickPulse makes sense. We got caught off guard after a recent update to the Azure Functions runtime, because we suddenly started seeing lots of dependency failures in our App Insights Failures tab. This also triggered an App Insights alert, saying there was an abnormal rise in the dependency failure rate for the service.

(Let me know if you'd like to have a peruse of our AppInsights!)

[brettsam]

If you can share some examples of these failures (either here as screenshots, or feel free to mail me directly; my email is on my profile), that would help. App Insights watches all Http traffic and logs it as dependencies, so we're trying to strategically hide the host operations since, as you can tell, they can be confusing. But we're obviously missing a few gaps here-and-there.

[brettsam]

Thanks to @NickMoores -- one binding scenario has been identified. When we internally make a call to container.CreateIfNotExistsAsync() before passing a CloudBlockBlob as an input parameter to a function, storage will return a 409 if that container already exists. Since it almost always exists, that means a lot of (harmless, but confusing) 409s.

I've contacted @lmolkova to see if we have any recommended approaches here. One thing we can do internally is check ExistsAsync() first, which will always return a 404 on the first invocation, but then will be successful for the rest of the run (unless the container was deleted, of course).

If anyone else has examples, feel free to keep sending them here and I will investigate.

[prabhakarreddy1234]

Any update on this? Is there anyway we can make AI doesn't show these harmless 409, 412 errors. or fixing these errors to not occur ?

[brettsam]

Fix is in (#1946) and will be part of our next release after 2.0.12134 (it just missed the cutoff).

@prabhakarreddy1234 -- are you seeing any of these Dependency failures in your App Insights queries (i.e. using Analytics) or are they only showing up in the Live Metrics view? See my comment above for the explanation.

[prabhakarreddy1234]

@brettsam I am seeing these errors only in Live Metrics View. Is #1946 fixes this issue ? I am currently using 2.0.12134.

[brettsam]

Yes, this will fix the issue. I'm closing this issue as the issue is fixed in our repo. I'll come back and update this issue when the fix is released. If anyone continues seeing issues after that, we can re-investigate.

[andrii-baranov]

@brettsam clarification: is fix for this issue already deployed to production? I am still seeing 409 dependency errors for table storage bindings in AI Live Metrics.
Runtime version: 2.0.12246.0
AzureWebJobsStorage: enabled

[brettsam]

@andrii-baranov -- The known fixes are deployed, yes. There may still be some one-off calls that we're not handling properly. There wasn't a general fix that would work across-the-board, so I've been trying to fix all of the reports that come in on a case-by-case basis. This one is likely due to us calling CreateIfNotExists somewhere. The Storage API tries to create, which returns a (benign) 409 if it already exists.

I can see if we can do something to suppress it, though. Would you mind opening a new issue with your specific failure you're seeing? It'd be helpful to see:

• What table operation is failing
• Which bindings you're using.

That should be enough for me to narrow it down.

[andrii-baranov]

@brettsam created issue #2098

[Kumar9676]

Is there any way I can avoid this error when using Output binding and using the return value as Blob content in Java?

Below is my function annotation and I am seeing 404 in the HEAD request of Blob.

@FunctionName("ProcessZipFiles2Cosmos")
@StorageAccount("blobStorageAccount")
@BlobOutput(name = "$return",
path = "nonpii/{name}.json")
public static String run(
@BlobTrigger(name = "file", dataType = "binary", path = "transactedreturn/{name}") byte[] content,
@BindingName("name") String filename,
//@BlobOutput(name = "blob_redacted_json", path = "nonpii2/{name}.json") OutputBinding outputItem,
@CosmosDBOutput(name = "database",
databaseName = "manoj",
collectionName = "manoj3",
connectionStringSetting = "AzureCosmosDBConnection") OutputBinding outputItem,
final ExecutionContext context

[brettsam]

@Kumar9676 -- can you share a sample path for the 404s you're seeing? That would help identify where it's coming from.