Sendgrid-csharp: "A task was canceled" exception

[VeselovAndrey]

We getting 1-2 exceptions per day with "A task was canceled" message. According to stack tracke it happens inside sendgrid-csharp library (strack trace copied below). Also I noticed that it happens for emails with attachement, and attachement size can be different: from few KB to few MB.

Can you advice?

Technical details:

• sendgrid-csharp Version: 9.9.0 (installed from NuGet)
• Platform ASP.NET
• .NET Version: 4.6.1

Stack trace

MESSAGE: A task was canceled.
STACK TRACE:
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at SendGrid.Helpers.Reliability.RetryDelegatingHandler.d__4.MoveNext() in C:\Users\ethomas\Documents\GitHub\sendgrid-csharp\src\SendGrid\Reliability\RetryDelegatingHandler.cs:line 50
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.Http.HttpClient.d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at SendGrid.SendGridClient.d__23.MoveNext() in C:\Users\ethomas\Documents\GitHub\sendgrid-csharp\src\SendGrid\SendGridClient.cs:line 250
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at SendGrid.SendGridClient.d__24.MoveNext() in C:\Users\ethomas\Documents\GitHub\sendgrid-csharp\src\SendGrid\SendGridClient.cs:line 290
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at SendGrid.SendGridClient.d__25.MoveNext() in C:\Users\ethomas\Documents\GitHub\sendgrid-csharp\src\SendGrid\SendGridClient.cs:line 301
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at EmailService.Send(EmailData data)

[willtam]

We are also experiencing the exact same scenario except we have:

sendgrid-csharp Version: 9.9.0 (installed from NuGet)
Platform ASP.NET
.NET Version: 4.6.2

[thinkingserious]

Hello @VeselovAndrey @willtam,

It seems like you may have a timeout issue. To debug further, please try catching that exception and looking into the error message.

Thanks!

With Best Regards,

Elmer

[VeselovAndrey]

Actually we catching and logging all exceptions. Here is exception message from log: "A task was canceled."
Or you mean something else?

[thinkingserious]

Hi @VeselovAndrey,

Could you please share a snippet of the code you are using to catch the exception? Thanks!

[VeselovAndrey]

I removed some code to make it compact and more readble.

try
{
// prepare HTML and text messages for email here. No async code here
var apiKey = _cfg.Get(CfgNames.SendGridAPIKey);
var client = new SendGridClient(apiKey);
var response = client.SendEmailAsync(msg).GetAwaiter().GetResult();
var successResponse = response.StatusCode == HttpStatusCode.OK || response.StatusCode == HttpStatusCode.Accepted;
// log "response" here
return successResponse;
}
catch (Exception e)
{
string errorMessage = $"Exception - MESSAGE: {e.Message} STACK TRACE: {e.StackTrace}";
// Exception logging code here
return false;
}

[willtam]

We are also catching the exception in similar format. We’ve also added more info to our logging and found that these tend to happen more often when the email’s total attachment size goes over the 5MB mark.

[thinkingserious]

@willtam Thanks for the additional information!

@VeselovAndrey are you also sending attachments over 5MB in size?

[VeselovAndrey]

@thinkingserious Sometimes. But it also happens even if attachment size less 100KB. I didn't find any dependency on attachment size, but I found no error for emails without attachments (but also such emails are rare in our system).

[thinkingserious]

Hi @VeselovAndrey,

Can you try increasing the timeout value on your server side? And also, add some logic in your code that will retry when this happens? Or is is that once you get this exception, that email will never send?

With Best Regards,

Elmer

[DavidRouyer]

I have the same problem, a timeout occurs sometimes while sending an attachment.

• sendgrid-csharp Version: 9.10.0 (installed from NuGet)
• Platform ASP.NET Core
• .NET Core Version: 2.2

System.TimeoutException: The operation has timed out. at SendGrid.Helpers.Reliability.RetryDelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at SendGrid.SendGridClient.MakeRequest(HttpRequestMessage request, CancellationToken cancellationToken) at SendGrid.SendGridClient.RequestAsync(Method method, String requestBody, String queryParams, String urlPath, CancellationToken cancellationToken) at SendGrid.SendGridClient.SendEmailAsync(SendGridMessage msg, CancellationToken cancellationToken) at Hopfab.Emails.EmailSender.SendEmailAsync(String email, String subject, String htmlContent, IList`1 attachments, IList`1 ccList, IList`1 bccList) in /app/src/Hopfab.Emails/EmailSender.cs:line 89    

My settings:

var clientOptions = new SendGridClientOptions()
{
    ApiKey = "mykey",
    ReliabilitySettings = new ReliabilitySettings(2, TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(10), TimeSpan.FromSeconds(3)),
};

Application Insights allows me to see the SendGrid dependency:

I guess it's being killed because it's running for too long (almost 2 minutes)

[thinkingserious]

Hi @DavidRouyer,

Wow, that should definitely not happen. I will check with our support team to find out if there are any delays receiving files on our end.

With Best Regards,

Elmer

[thinkingserious]

@VeselovAndrey, @DavidRouyer, @willtam,

The support team has asked for you all to please open tickets and include the IP that you are connecting with for further debugging.

With Best Regards,

Elmer

[willtam]

@thinkingserious

I added extra logging and found that whenever a request is taking 100+ seconds to complete, then this error will get thrown. Thanks for looking into it.

[graham-saunders]

@willtam I'm also seeing this issue.

@thinkingserious has there been any updates here? Seems like a fairly common issue. Should we also open tickets? And should the tickets be for each subuser or as a general account?

[VeselovAndrey]

@thinkingserious Actually I was sent to GitHub repo from support ticked. Should I open a new one?

[thinkingserious]

Hi @VeselovAndrey,

Could you please just reference this thread in your previous ticket? Thanks!

[VeselovAndrey]

@thinkingserious I can't add new message because ticket already closed as solved. But I can provide the ticked number: 1855796

[thinkingserious]

Hi @VeselovAndrey,

I have let the support team know. Thanks!

With Best Regards,

Elmer

[graham-saunders]

@thinkingserious any updates on this issue?

[thinkingserious]

Hello @graham-saunders,

Last I heard was that this was still under investigation. If this is happening to you, please do reach out to our support team. More data could likely help us get to resolution quicker. Thanks!

With Best Regards,

Elmer

[VeselovAndrey]

@thinkingserious Finally we integrate changes you suggested into production. But still we get 1 failed email sometimes even after all retries.

[willtam]

I ended up passing in a HttpClient with a longer timeout value into the SendGridClient constructor as follows:

System.Net.Http.HttpClient longerTimeoutValueClient = new System.Net.Http.HttpClient();
longerTimeoutValueClient.Timeout = new TimeSpan(0, 5, 0);
SendGridClient sgc = new SendGridClient(longerTimeoutValueClient, apiKey);

Then I also wrapped the SendEmailAsync() call within a loop that retry up to 3 times.

Now, instead of getting about 5-7 of these errors per day, I only get 1-2 per week. Still not ideal but it helps.

[thinkingserious]

@VeselovAndrey, @willtam,

Please be sure to update your support tickets with the new information. Thanks!

And thanks for taking the time to post here!

With Best Regards,

Elmer

[AlquistArj]

I'm also getting "A task was cancelled" exceptions. I'm sending via a flaky satellite link so the quality of connection is sometimes poor, or non existent. I notice the problem occurs more frequently as the attachments gets bigger. Does the upload happen in one chunk? Is there any way to upload in smaller chunks to potentially improve the reliability particularly if sending on a flaky network link.

[thinkingserious]

Hello @AlquistArj,

I'm not aware of way to upload the attachments in smaller chunks. How about storing your attachment behind a link? If you did that, you would also gain the advantage of click tracking.

With Best Regards,

Elmer

[Shilp]

@thinkingserious Getting similar exceptions everyday with our sendgrid v3 integration,
Exception Info: System.Net.WebException
at System.Net.HttpWebRequest.EndGetRequestStream(System.IAsyncResult, System.Net.TransportContext ByRef)
at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(System.IAsyncResult)

Exception Info: System.Net.Http.HttpRequestException
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at SendGrid.Helpers.Reliability.RetryDelegatingHandler+d__4.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at SendGrid.SendGridClient+d__21.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at SendGrid.SendGridClient+d__22.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at SendGrid.SendGridClient+d__23.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

Seems to be a timeout , we are not using any reliability settings as such. Api used is
MailHelper.CreateSingleEmailToMultipleRecipients(...)
await client.SendEmailAsync(msg)

What is the solution? Do we need to re try sending this email on exception?

[thinkingserious]

Hi @Shilp,

Have you tried this solution?

With Best Regards,

Elmer

[Shilp]

@thinkingserious Some more queries on this, Does the timeout exception thrown after the default retries exhaust or is it on first call itself? What is the reason for timeout? Is it safe to retry on catching exceptions? We do not want duplicate emails to be delivered. Is the ReliabilitySettings recommended before adding a retry logic on exceptions? I would like to know what is the default wait time before a default re try happens in SendGrid. Thanks

[thinkingserious]

@Shilp These are questions that are best answered by our support team. Please reach out to them here.

[samifahmy]

i have same issue when i try to run my ASP.net web application locally using local IIS or IIS Express but it seems working fine on Azure!
note console application work fine locally

[thinkingserious]

Hello Everyone!

I am closing this issue out as I believe support on this issue is best handled by our support team.

With best regards,

Elmer