Azure-functions-host: Azure Functions throw TaskCanceledException

[vijayrkn]

Create new Azure Function 2.0 project with an HttpTrigger function. Run it locally. Open the URL to trigger the funtion

Systems.Threading.Tasks.TaskCanceledException

in System.Private.CoreLib.dll

"A task was cancelled"

This happens for each request.

More details in this bug report : https://developercommunity.visualstudio.com/content/problem/549316/azure-functions-throw-taskcanceledexception.html

[ColbyTresness]

@Francisco-Gamino can you investigate?

[GuilhermeMorais]

I got the same error, in my case the project was created using the cmd: func init
This error happens while debugging or when you publish to Azure.

Error detail:
C# System.Threading.Tasks.TaskCanceledException HResult=0x8013153B Message=A task was canceled. Source=System.Private.CoreLib StackTrace: at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult() at Microsoft.Azure.WebJobs.Script.WebHost.Diagnostics.MetricsEventManager.FunctionActivityTracker.<<-ctor>b__8_0>d.MoveNext() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Diagnostics\MetricsEventManager.cs:line 341

[zorkind]

Any news on this issue? for me this is happening in 1.0 az function. after it run for some time, exactly 229936ms (around 1 hour) i don't think this is function timeout, my function is on an app service so the timeout should be zero as default.

[arby50]

I can duplicate as well, but calling multiple HTTP clients simultaneously to the same HTTPTrigger. For example, when starting 20 simultaneous calls, 5 will error with the TaskCanceledException within the first 30 seconds.

[Francisco-Gamino]

Hi @ankitkumarr, could you please take a look? If this is not Core-Tools related, we'll move it to the correct repo. Thanks.

[karinpm]

I am also experiencing this issue - is there any update?

[ankitkumarr]

This looks like a host issue. Moving to that repo.

[ankitkumarr]

@yojagad would you have an insight on this? I remember you were doing some stuff with MetricsEventManager.

[spettifer]

I would be good to have at least a vague explanation for this as we are also experiencing this issue. Whilst not a show stopper for us it is at the very least inconvenient.

[mikelor]

I'm getting the same error. When I activate historical debugging, it appears to be in the MetricsEventManager.cs in Microsoft.Azure.WebJobs.Script.WebHost.Diagnostics

The exception is thrown at on this line, in the FunctionActivityTracker
await Task.Delay(TimeSpan.FromSeconds(1), _etwTaskCancellationSource.Token);

    private class FunctionActivityTracker : IDisposable
    {
        private readonly string _executionId = Guid.NewGuid().ToString();
        private readonly object _functionMetricEventLockObject = new object();
        private ulong _totalExecutionCount = 0;
        private int _functionActivityFlushInterval;
        private CancellationTokenSource _etwTaskCancellationSource = new CancellationTokenSource();
        private ConcurrentQueue<FunctionMetrics> _functionMetricsQueue = new ConcurrentQueue<FunctionMetrics>();
        private Dictionary<string, RunningFunctionInfo> _runningFunctions = new Dictionary<string, RunningFunctionInfo>();
        private bool _disposed = false;
        private IOptionsMonitor<AppServiceOptions> _appServiceOptions;
        private IMetricsPublisher _metricsPublisher;

        internal FunctionActivityTracker(IOptionsMonitor<AppServiceOptions> appServiceOptions, IEventGenerator generator, IMetricsPublisher metricsPublisher, int functionActivityFlushInterval)
        {
            MetricsEventGenerator = generator;
            _appServiceOptions = appServiceOptions;
            _functionActivityFlushInterval = functionActivityFlushInterval;
            _metricsPublisher = metricsPublisher;
            Task.Run(
                async () =>
                {
                    try
                    {
                        int currentSecond = _functionActivityFlushInterval;
                        while (!_etwTaskCancellationSource.Token.IsCancellationRequested)
                        {
                            RaiseMetricsPerFunctionEvent();

                            if (currentSecond >= _functionActivityFlushInterval)
                            {
                                RaiseFunctionMetricEvents();
                                currentSecond = 0;
                            }
                            else
                            {
                                currentSecond = currentSecond + 1;
                            }

                            await Task.Delay(TimeSpan.FromSeconds(1), _etwTaskCancellationSource.Token);
                        }
                    }
                    catch (TaskCanceledException)
                    {
                        // This exception gets throws when cancellation request is raised via cancellation token.
                        // Let's eat this exception and continue
                    }
                },
                _etwTaskCancellationSource.Token);
        }

`

[ankitkumarr]

Thanks @mikelor for sharing that. That is weird that the exception from there is being bubbled up. As you can see, there is a catch (TaskCanceledException) in place to take care of that exception.

I am having a hard time reproing this. I even added code to manually cancel the cancellation token right before that line, and it gets accurately handled in the catch block.

I will look more into this.

Adding @fabiocav as he may have some insight on why this is bubbling up, or a workaround.

What version of core-tools / Azure Functions Runtime are you on? You can check by running func

[mikelor]

Hi @ankitkumarr, I created a simple app the repros the problem when reading from a Service Bus queue. See the TaskCanceledFunctionApp repo.

As you mentioned the Exception gets handled, so it's probably okay. I'm showing the exceptions occurring in the debugger, but since their being swallowed up, everything continues to run.

Is this the "expected behavior"? Looks like it, but would like some confirmation.

Exception has occurred: CLR/System.Threading.Tasks.TaskCanceledException
Exception thrown: 'System.Threading.Tasks.TaskCanceledException' in System.Private.CoreLib.dll: 'A task was canceled.'
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.Script.WebHost.Diagnostics.MetricsEventManager.FunctionActivityTracker.<<-ctor>b__10_0>d.MoveNext() in C:\azure-functions-host\src\WebJobs.Script.WebHost\Diagnostics\MetricsEventManager.cs:line 363

This sample writes to a Service Bus Queue via a Timer Trigger and reads from the queue via a Queue Trigger.

Also included is a createResourceGroup.sh shell script that will create the Azure Service Bus to write/read from.

This successfully reproduces the TaskCanceledException for me. See ReadMe for more info. You will need to add your own localsettings.json file to add a reference to the ServiceBusConnectionString generated by the script.

func -v returns 3.0.2009

I'm using Visual Studio 16.5.0 Preview 1.0
And have implemented this workaround Debug startup fails "no Function runtime available" #5145

cc: @fabiocav

[yangyadi1993]

Any updates or workaround? I am having the same issue.

[SergeyOpalev]

Do not use async void - change it to async Task instead.

[LarrySmith-1437]

same problem, with function defined using "async Task". Same exception every execution.

Funny... Converted back to non-async, but every execution is STILL throwing a task canceled exception from System.Private.CoreLib.dll

[fabiocav]

@yangyadi1993 @LarrySmith-1437 is this with the same stack shared above? If so, this is not an exception that will impact runtime behavior.

[LarrySmith-1437]

@fabiocav I updated my comment above, it's happening elsewhere once I made my method non-async. So now it's not throwing from my code, but somewhere else.

[fabiocav]

Understood, but is this causing a runtime exception that is impacting your application, or are you seeing that in VS while debugging and setting exceptions to throw? If your stack is different, can you please open a separate issue with the details? Thanks!

[LarrySmith-1437]

Does not seem to be impacting execution, except of course it does draw some attention and one would love to know where/why the exception is happening prior to putting code into production. I found it interesting that the exception is being thrown whether the Run method is async or not.

[masaab]

I was missing an await keyword for my method. Make sure all your async methods contain await keyword.

[fabiocav]

Closing as resolved.