Azure-webjobs-sdk: Continuous Web Job crashes with code -1073610685

[raeffs]

Hi,

We have a continuous web job that gets triggered by a queue trigger. That web job takes a message and generates several messages out of it, and adds them to another queue with a collector. The web job crashes quite often with the exit code -1073610685 (it crashes more often than it succeeds...).

We can reproduce that with this simple web job deployed to an Azure web app (we cannot reproduce it on a local development system):

using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;

namespace TestJob
{
    public class WebJob
    {
        public async Task ProcessNotificationAsync([QueueTrigger("notifications")] string message, [Queue("notifications-queued2")] IAsyncCollector<string> outgoingMessages, ILogger logger)
        {
            logger.LogInformation("> ProcessNotificationAsync");

            for (var i = 0; i < 50; i++)
            {
                await outgoingMessages.AddAsync(message);
            }

            logger.LogInformation("< ProcessNotificationAsync");
        }

        public async Task ProcessQueuedNotificationAsync([QueueTrigger("notifications-queued2")] string message, ILogger logger)
        {
            logger.LogInformation("> ProcessQueuedNotificationAsync");

            await Task.Delay(0);

            logger.LogInformation("< ProcessQueuedNotificationAsync");
        }
    }
}

We use the following packages:

    <PackageReference Include="Microsoft.Azure.WebJobs" Version="3.0.0-beta4" />
    <PackageReference Include="Microsoft.Extensions.Logging.Console" Version="2.0.0" />

Thanks,
Raphael

[christopheranderson]

Can you share a time window and info about the site? (https://github.com/projectkudu/kudu/wiki/Reporting-your-site-name-without-posting-it-publicly)

This isn't something we see often, so it's likely being caused by something you're doing or something else in your environment.

[mathewc]

Do you see any errors logged to your log files?

[raeffs]

We tried to catch and log any unhandled exception with a AppDomain.UnhandledException handler, but we were not able to catch any errors this way.

I deployed the code above to the site https://dummy-support.azurewebsites.net, the last time it failed was [01/23/2018 19:49:26 > b33f2c: SYS ERR ] Job failed due to exit code -1073610685. Can you work with this?

[hackked]

I'm having the exact same issue with a function triggered using a service bus queue. This function has no content other than a single line to log the string content bound as an input. A number of handlers will be invoked printing the contents and then the following is printed in the logs:

[03/01/2018 14:55:35 > 677ac4: SYS ERR ] Job failed due to exit code -1073610685

Here's the current list of packages I'm using:

Anything you would like me to try or additional information I can provide?

[codecadwallader]

I am seeing this issue as well, using version 3.0.0-beta5 of Microsoft.Azure.WebJobs. I am also using the AysncCollector the same as the OP. Please let me know if there's any additional information I can provide!

It does seem to be an intermittent issue. I have 5 jobs running with similar infrastructure and whereas 3 of them in one app service are running fine at the moment, 2 of them in a different app service and encountering the issue.

[mdsharpe]

I am seeing this problem when running a .NET Core 2.0 WebJob using Microsoft.Azure.WebJobs.ServiceBus version 3.0.0-beta4. My job is triggered with a [ServiceBusTrigger].

The problem does not occur if I set serviceBusConfiguration.MessageOptions.MaxConcurrentCalls to a small number such as 1 or 2. It seems mostly stable at 3, but if I set it to 4 or higher then the WebJob is guaranteed to crash.

The exact same code works perfectly on my local workstation, with MaxConcurrentCalls set to 16 or even 100.

[gregsika]

We are also experiencing this problem ("Job failed due to exit code -1073610685") with multiple .NET Core 2.0 WebJobs using Microsoft.Azure.WebJobs 3.0.0-beta5 (and Microsoft.Azure.WebJobs.Extensions).

Our WebJobs use both EventHubTrigger and TimerTrigger triggers and send output to multiple destinations including EventHub (via IAsyncCollector), a custom web service, and Azure SQL. All combinations are encountering the same OP issue intermittently.

We have not been able to discern a pattern in the failure frequency based on execution volumes or trigger frequency. It has failed 5 times in a row on occasion and after 100 successful runs on another occasion but is usually mixed between "Success" and "Never Finished" (as reported by KUDU in WebJob Logs UI). However, there are fewer job failure completions than successful job completions.

[tdjastrzebski]

What caused error 1073610685 in my case was .Net Framework 4.7.2.
After changing to version to 4.6.2 the problem is gone.