Azure-functions-host: Host crashes caused by log contention

Created on 6 Nov 2018 · 20Comments · Source: Azure/azure-functions-host

We are experiencing unexpected host restarts when running a function. When this happens, the diagnostic logs show the normal trace messages from the execution of the function but occasionally, before the invocation completes, the logs show that the host restarted, with no indication of what has caused it to crash.

As I said, the streaming logs do not provide any information, but in the event log messages in Kudu (eventlog.xml) we have observed that every time that this happens, it seems to have been caused by contention while writing to the host logs. Here's an excerpt of the log showing the two events that we always see before each crash.

<Event>
    <System>
        <Provider Name="ASP.NET 4.0.30319.0"/>
        <EventID>1325</EventID>
        <Level>1</Level>
        <Task>0</Task>
        <Keywords>Keywords</Keywords>
        <TimeCreated SystemTime="2018-10-31T01:37:53Z"/>
        <EventRecordID>777728515</EventRecordID>
        <Channel>Application</Channel>
        <Computer>…</Computer>
        <Security/>
    </System>
    <EventData>
        <Data>
            An unhandled exception occurred and the process was terminated.

            Application ID: /LM/W3SVC/1134994283/ROOT

            Process ID: 1780

            Exception: System.IO.IOException

            Message: The process cannot access the file 'D:\home\LogFiles\Application\Functions\Host\2018-10-31T01-34-39Z-7f29d1a4c5.log' because it is being used by another process.

            StackTrace:    at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
            at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
            at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
            at System.IO.FileInfo.Open(FileMode mode, FileAccess access, FileShare share)
            at Microsoft.Azure.WebJobs.Script.FileTraceWriter.AppendToFile(FileInfo fileInfo, String content) in C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Diagnostics\FileTraceWriter.cs:line 279
            at Microsoft.Azure.WebJobs.Script.FileTraceWriter.Flush(Boolean timerCallback) in C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Diagnostics\FileTraceWriter.cs:line 151
            at Microsoft.Azure.WebJobs.Script.FileTraceWriter.FlushCallback(Object state) in C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Diagnostics\FileTraceWriter.cs:line 91
            at System.Threading.TimerQueueTimer.CallCallbackInContext(Object state)
            at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
            at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
            at System.Threading.TimerQueueTimer.CallCallback()
            at System.Threading.TimerQueueTimer.Fire()
            at System.Threading.TimerQueue.FireQueuedTimerCompletion(Object state)
            at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
            at System.Threading.ThreadPoolWorkQueue.Dispatch()
            at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
        </Data>
    </EventData>
</Event>
<Event>
    <System>
        <Provider Name=".NET Runtime"/>
        <EventID>1026</EventID>
        <Level>1</Level>
        <Task>0</Task>
        <Keywords>Keywords</Keywords>
        <TimeCreated SystemTime="2018-10-31T01:37:54Z"/>
        <EventRecordID>777729218</EventRecordID>
        <Channel>Application</Channel>
        <Computer>…</Computer>
        <Security/>
    </System>
    <EventData>
        <Data>
            Application: w3wp.exe
            Framework Version: v4.0.30319
            Description: The process was terminated due to an unhandled exception.
            Exception Info: System.IO.IOException
            at System.IO.__Error.WinIOError(Int32, System.String)
            at System.IO.FileStream.Init(System.String, System.IO.FileMode, System.IO.FileAccess, Int32, Boolean, System.IO.FileShare, Int32, System.IO.FileOptions, SECURITY_ATTRIBUTES, System.String, Boolean, Boolean, Boolean)
            at System.IO.FileStream..ctor(System.String, System.IO.FileMode, System.IO.FileAccess, System.IO.FileShare)
            at System.IO.FileInfo.Open(System.IO.FileMode, System.IO.FileAccess, System.IO.FileShare)
            at Microsoft.Azure.WebJobs.Script.FileTraceWriter.AppendToFile(System.IO.FileInfo, System.String)
            at Microsoft.Azure.WebJobs.Script.FileTraceWriter.Flush(Boolean)
            at Microsoft.Azure.WebJobs.Script.FileTraceWriter.FlushCallback(System.Object)
            at System.Threading.TimerQueueTimer.CallCallbackInContext(System.Object)
            at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
            at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
            at System.Threading.TimerQueueTimer.CallCallback()
            at System.Threading.TimerQueueTimer.Fire()
            at System.Threading.TimerQueue.FireQueuedTimerCompletion(System.Object)
            at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
            at System.Threading.ThreadPoolWorkQueue.Dispatch()
            at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

        </Data>
    </EventData>
</Event>

Timestamp: 2018-11-06T12:53:45.889
Function App version (1.0 or 2.0): 1.0
Invocation ID: d2c4757a-7444-498c-b07c-f766fabef199
Region: East US

Any suggestions or workarounds?

Thanks

P1 bug

Source

f2bo

👍3

Most helpful comment

I have the same issue today
The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\ItemsTrigger\2019-05-05T16-16-01Z-8d04d9d4f1.log' because it is being used by another process.

function app: cosmosdemo-function
functionName: ItemsTrigger

This incident have been open for 6 months now, is somebody going to take a look at it? :)

crgarcia12 on 5 May 2019

👍3

All 20 comments

Having the exact same issue randomly while having more load on the functions. As a workaround, can I limit the use the logging methods within the functions until a fix is provided?

lonesomegeek on 23 Nov 2018

We have found something that may be useful for further analysis.

We just went live a few days ago on a project with an architecture of Azure Servicebus with Azure Functions. Within our beta test, we were often taking a look at Log Streamers (within Azure Functions) and if I correlate the quantity of errors and the timeframe of the errors, these errors are mostly related to our beta phase when we were in the Log Streamers almost all day ;)

Hope it can help!

lonesomegeek on 23 Nov 2018

Same happening for us in the last 2 days. Azure Function v2 crashes. It runs on AppService. But it is a long-running one.

Nemeczek on 29 Nov 2018

Same issue here. I have an HTTP triggered function that logs an exception " log.LogError("An error has occurred.", ex);". I'm getting the error below when logging the exception.

carloscamposcoelho on 18 Dec 2018

👍3

We are a team inside Microsoft having the same issue. We discussed this problem with Azure support (ticket 119021321000209) and so far they suggested to turn off local logging, since we also have Application Insights active. However, turning off local logging did not fix this issue.

We use Azure Functions v2 and we Publish updates using the package (zip) option.

zmarty on 14 Mar 2019

👍1

function app: cosmosdemo-function
functionName: ItemsTrigger

This incident have been open for 6 months now, is somebody going to take a look at it? :)

crgarcia12 on 5 May 2019

👍3

Same issue here, Azure Functions v2, publishing using package option.

CaioCavalcanti on 7 May 2019

👍1

@fabiocav Any ideas?

zmarty on 7 May 2019

We are having the same issue in our Event Hub triggered Function App on Runtime v1 (1.0.12299.0).
It worked fine until friday May 3 2019.
Is there a status update on this issue?

robrechtdelanghe on 10 May 2019

I am seeing the same issue. I am running a Durable Function in fan-out pattern. So there are up to a couple of hundred threads running in parallel and for some threads I get the error.

sebader on 20 May 2019

I have a similar issue where I try to log an error in the exception block it gave me the below error
The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\DetailsFunction\2019-05-21T22-12-30Z-1b572f146a.log' because it is being used by another process

A30001344 on 22 May 2019

Also experiencing this issue running Azure Functions v2. As much as I'm sure we all love paying to be your beta testers Microsoft, how about we see some hustle on this for our bucks if you please?

Exception:

Exception while executing function: MyFunction One or more errors occurred. (An error occurred while writing to logger(s). (The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\MyFunction\2019-05-21T09-47-15Z-1e265b5431.log' because it is being used by another process.)) (An error occurred while writing to logger(s). (The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\MyFunction\2019-05-21T09-47-15Z-1e265b5431.log' because it is being used by another process.)) (An error occurred while writing to logger(s). (The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\MyFunction\2019-05-21T09-47-15Z-1e265b5431.log' because it is being used by another process.)) (An error occurred while writing to logger(s). (The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\MyFunction\2019-05-21T09-47-15Z-1e265b5431.log' because it is being used by another process.)) (An error occurred while writing to logger(s). (The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\MyFunction\2019-05-21T09-47-15Z-1e265b5431.log' because it is being used by another process.)) (An error occurred while writing to logger(s). (The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\MyFunction\2019-05-21T09-47-15Z-1e265b5431.log' because it is being used by another process.)) (An error occurred while writing to logger(s). (The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\MyFunction\2019-05-21T09-47-15Z-1e265b5431.log' because it is being used by another process.)) (An error occurred while writing to logger(s). (The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\MyFunction\2019-05-21T09-47-15Z-1e265b5431.log' because it is being used by another process.)) (An error occurred while writing to logger(s). (The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\MyFunction\2019-05-21T09-47-15Z-1e265b5431.log' because it is being used by another process.)) (An error occurred while writing to logger(s). (The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\MyFunction\2019-05-21T09-47-15Z-1e265b5431.log' because it is being used by another process.)) (An error occurred while writing to logger(s). (The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\MyFunction\2019-05-21T09-47-15Z-1e265b5431.log' because it is being used by another process.)) (An error occurred while writing to logger(s). (The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\MyFunction\2019-05-21T09-47-15Z-1e265b5431.log' because it is being used by another process.)) (An error occurred while writing to logger(s). (The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\MyFunction\2019-05-21T09-47-15Z-1e265b5431.log' because it is being used by another process.)) (An error occurred while writing to logger(s). (The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\MyFunction\2019-05-21T09-47-15Z-1e265b5431.log' because it is being used by another process.)) (An error occurred while writing to logger(s). (The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\MyFunction\2019-05-21T09-47-15Z-1e265b5431.log' because it is being used by another process.)) (An error occurred while writing to logger(s). (The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\MyFunction\2019-05-21T09-47-15Z-1e265b5431.log' because it is being used by another process.)) An error occurred while writing to logger(s). (The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\MyFunction\2019-05-21T09-47-15Z-1e265b5431.log' because it is being used by another process.) The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\MyFunction\2019-05-21T09-47-15Z-1e265b5431.log' because it is being used by another process. An error occurred while writing to logger(s). (The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\MyFunction\2019-05-21T09-47-15Z-1e265b5431.log' because it is being used by another process.) The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\MyFunction\2019-05-21T09-47-15Z-1e265b5431.log' because it is being used by another process. An error occurred while writing to logger(s). (The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\MyFunction\2019-05-21T09-47-15Z-1e265b5431.log' because it is being used by another process.) The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\MyFunction\2019-05-21T09-47-15Z-1e265b5431.log' because it is being used by another process. An error occurred while writing to logger(s). (The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\MyFunction\2019-05-21T09-47-15Z-1e265b5431.log' because it is being used by another process.) The process cannot access the file 'D:\home\LogFiles\Application\Functions\Function\MyFunction\2019-05-21T09-47-15Z-1e265b5431.log' because it is being used by another process. The number of inner exceptions was 34 which is larger than 10, the maximum number allowed during transmission. All but the first 10 have been dropped.

Call stack:

Microsoft.Azure.WebJobs.Host.FunctionInvocationException:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__19.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 321)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsyncCore>d__16.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 117)
Inner exception System.Collections.Async.ParallelForEachException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at ***.MoveNext ([removed for security])
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at ***.MoveNext ([removed for security])
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2+<InvokeAsync>d__2.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\VoidTaskMethodInvoker.csMicrosoft.Azure.WebJobs.Host, Version=3.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 20)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2+<InvokeAsync>d__10.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.csMicrosoft.Azure.WebJobs.Host, Version=3.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 52)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<InvokeAsync>d__27.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 584)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithWatchersAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 531)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__25.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 467)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__19.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 277)
Inner exception System.AggregateException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
   at Microsoft.Extensions.Logging.Logger.Log (Microsoft.Extensions.Logging, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.Extensions.Logging.LoggerExtensions.Log (Microsoft.Extensions.Logging.Abstractions, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.Extensions.Logging.LoggerExtensions.LogError (Microsoft.Extensions.Logging.Abstractions, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at ***.MoveNext ([removed for security])
Inner exception System.IO.IOException handled at Microsoft.Extensions.Logging.Logger.Log:
   at System.IO.FileStream.ValidateFileHandle (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.IO.FileStream.CreateFileOpenHandle (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.IO.FileStream..ctor (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.IO.FileStream..ctor (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.IO.FileInfo.Open (System.IO.FileSystem, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at Microsoft.Azure.WebJobs.Script.FileWriter.AppendToFile (Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: C:\azure-webjobs-sdk-script\src\WebJobs.Script\Diagnostics\FileWriter.csMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: 114)
   at Microsoft.Azure.WebJobs.Script.FileWriter.Flush (Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: C:\azure-webjobs-sdk-script\src\WebJobs.Script\Diagnostics\FileWriter.csMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: 92)
   at Microsoft.Azure.WebJobs.Script.Diagnostics.FileLogger.Log (Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: C:\azure-webjobs-sdk-script\src\WebJobs.Script\Diagnostics\FileLogger.csMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: 94)
   at Microsoft.Extensions.Logging.Logger.Log (Microsoft.Extensions.Logging, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
Inner exception System.AggregateException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
   at Microsoft.Extensions.Logging.Logger.Log (Microsoft.Extensions.Logging, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.Extensions.Logging.LoggerExtensions.Log (Microsoft.Extensions.Logging.Abstractions, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.Extensions.Logging.LoggerExtensions.LogError (Microsoft.Extensions.Logging.Abstractions, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at ***.MoveNext ([removed for security])
Inner exception System.IO.IOException handled at Microsoft.Extensions.Logging.Logger.Log:
   at System.IO.FileStream.ValidateFileHandle (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.IO.FileStream.CreateFileOpenHandle (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.IO.FileStream..ctor (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.IO.FileStream..ctor (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.IO.FileInfo.Open (System.IO.FileSystem, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at Microsoft.Azure.WebJobs.Script.FileWriter.AppendToFile (Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: C:\azure-webjobs-sdk-script\src\WebJobs.Script\Diagnostics\FileWriter.csMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: 114)
   at Microsoft.Azure.WebJobs.Script.FileWriter.Flush (Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: C:\azure-webjobs-sdk-script\src\WebJobs.Script\Diagnostics\FileWriter.csMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: 92)
   at Microsoft.Azure.WebJobs.Script.Diagnostics.FileLogger.Log (Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: C:\azure-webjobs-sdk-script\src\WebJobs.Script\Diagnostics\FileLogger.csMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: 94)
   at Microsoft.Extensions.Logging.Logger.Log (Microsoft.Extensions.Logging, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
Inner exception System.AggregateException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
   at Microsoft.Extensions.Logging.Logger.Log (Microsoft.Extensions.Logging, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.Extensions.Logging.LoggerExtensions.Log (Microsoft.Extensions.Logging.Abstractions, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.Extensions.Logging.LoggerExtensions.LogError (Microsoft.Extensions.Logging.Abstractions, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at ***.MoveNext ([removed for security])
Inner exception System.IO.IOException handled at Microsoft.Extensions.Logging.Logger.Log:
   at System.IO.FileStream.ValidateFileHandle (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.IO.FileStream.CreateFileOpenHandle (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.IO.FileStream..ctor (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.IO.FileStream..ctor (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.IO.FileInfo.Open (System.IO.FileSystem, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at Microsoft.Azure.WebJobs.Script.FileWriter.AppendToFile (Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: C:\azure-webjobs-sdk-script\src\WebJobs.Script\Diagnostics\FileWriter.csMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: 114)
   at Microsoft.Azure.WebJobs.Script.FileWriter.Flush (Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: C:\azure-webjobs-sdk-script\src\WebJobs.Script\Diagnostics\FileWriter.csMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: 92)
   at Microsoft.Azure.WebJobs.Script.Diagnostics.FileLogger.Log (Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: C:\azure-webjobs-sdk-script\src\WebJobs.Script\Diagnostics\FileLogger.csMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: 94)
   at Microsoft.Extensions.Logging.Logger.Log (Microsoft.Extensions.Logging, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
Inner exception System.AggregateException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
   at Microsoft.Extensions.Logging.Logger.Log (Microsoft.Extensions.Logging, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.Extensions.Logging.LoggerExtensions.Log (Microsoft.Extensions.Logging.Abstractions, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.Extensions.Logging.LoggerExtensions.LogError (Microsoft.Extensions.Logging.Abstractions, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at ***.MoveNext ([removed for security])
Inner exception System.IO.IOException handled at Microsoft.Extensions.Logging.Logger.Log:
   at System.IO.FileStream.ValidateFileHandle (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.IO.FileStream.CreateFileOpenHandle (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.IO.FileStream..ctor (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.IO.FileStream..ctor (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.IO.FileInfo.Open (System.IO.FileSystem, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at Microsoft.Azure.WebJobs.Script.FileWriter.AppendToFile (Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: C:\azure-webjobs-sdk-script\src\WebJobs.Script\Diagnostics\FileWriter.csMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: 114)
   at Microsoft.Azure.WebJobs.Script.FileWriter.Flush (Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: C:\azure-webjobs-sdk-script\src\WebJobs.Script\Diagnostics\FileWriter.csMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: 92)
   at Microsoft.Azure.WebJobs.Script.Diagnostics.FileLogger.Log (Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: C:\azure-webjobs-sdk-script\src\WebJobs.Script\Diagnostics\FileLogger.csMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: 94)
   at Microsoft.Extensions.Logging.Logger.Log (Microsoft.Extensions.Logging, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
Inner exception Microsoft.ApplicationInsights.DataContracts.InnerExceptionCountExceededException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:

nero120 on 28 May 2019

hello, any update on this? I'm experiencing the same with a Durable Function applying the fan-out pattern.

regards, F.

fume on 25 Jun 2019

@fume this is assigned to the current sprint. We'll be looking at this ASAP.

fabiocav on 26 Jun 2019

Thanks for the feedback @fabiocav , really appreciated.

fume on 26 Jun 2019

I've just been doing some re-creation of this issue (I'd had a notification that this issue was resolved in the latest runtime release so was trying to test it before I realised it hadn't been).
I had been experiencing the issue with repeated large errors being logged (to be picked up by app insights), in recreating the issue I have found that high volume small errors have the same effect, and that the file is blocked until app insights has finished reading in the entries.
From the discussion and looking at the PR for the fix I'm concerned this won't actually be resolved.
It's easy to recreate the issue with the below code on a function with app insights connected, on my initial run it got to ~4000 errors logged, which did eventually make it into app insights, but took a while to catch up, on subsequent runs it didn't get anywhere near as far through.
for (int i = 0; i < 10000; i++) { log.LogError(new Exception("error"+i), "error"+i); }
The same code but with LogInformation() gets significantly further but still suffers from the same issue.

davemckerral on 1 Jul 2019

👍1

@gzuber Quick question, is this fixed in V1, V2, or both?

f2bo on 18 Jul 2019

Good question. The fix was for V2 only, but this is something we should backport.

@gzuber

fabiocav on 18 Jul 2019

Yes, please. I originally started this thread last year. We're still stuck on V1 because of a dependency that's not supported under .NET Core and even though we haven't seen these crashes recently (maybe because we're not relying on streaming logs as much?), we'd be much happier if it didn't happen any more.

f2bo on 18 Jul 2019

There's actually an issue I've opened a few days ago to track this work for V1 (https://github.com/Azure/azure-functions-host/issues/4573), let's use that to follow the progress on getting this backported. Shouldn't take long.

fabiocav on 18 Jul 2019

👍1

Was this page helpful?

0 / 5 - 0 ratings

Related issues

A command that prompts the user failed

paulbatum · 4Comments

Service Bus AutoRenewTimeout not configurable

silencev · 3Comments

[PHP] Using Blob as input fails with `mscorlib` exception

ElvenSpellmaker · 3Comments

Access to XMLHttpRequest at 'http://localhost:8080/register' from origin 'http://localhost:4200' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: It does not have HTTP ok status.

slim-souissi · 3Comments

Metadata generation failed. Exit code: '-1073741502' Error: '' #6769

krillehbg · 3Comments