Botframework-sdk: Intermittent 403, 502 errors | Is something up with the services?

[JPThorne]

We are intermittently receiving these errors:

POST https://directline.botframework.com/v3/directline/conversations/JrdN8uvQS0UKzaLE2pFFuf/activities 502 (Bad Gateway)

And now we've also been seeing 403 errors and Access-Control-Allow-Origin errors.

Has something changed on the directline side of things? Do we now need to send additional headers?

[McGern]

I've been getting a lot of "503 (service unavailable)" messages and "Access-Control-Allow-Origin" errors. Are they linked?

[JPThorne]

Yes, that's exactly what we're getting too. Now we're getting this intermittently:

[dandriscoll]

Hi @McGern and @JPThorne, we're experiencing load issues in our European datacenter at the moment. This is our top priority and we're working on fixing this ASAP.

[willemodendaal]

I notice the failures occur because calls to the bot is timing out after 15 seconds. However, if AppInsights is accurate, this is because the Bot seems to call directline.botframework and that takes a long time to succeed (European datacenter issues I assume).

Is this really the case? Our bot uses the bot connector, so does the Bot connector call the DirectLine API? And if so, why?

I haven't been able to pinpoint the code that causes the call to the DirectLine API because it does not happen locally with the emulator.

I'm very sure it's not our code specifically calling the DirectLine API.

[dandriscoll]

Hi @willemodendaal, which channel are you using to talk to your bot?

[willemodendaal]

Hi @dandriscoll . We are using the directline channel. Thinking about it more... I guess the Bot Connector will call that channel to send the response back? I tend to think of the bot framework calling the Bot and waiting for a simple Http response, but I guess it's more complicated than that. The Bot probably has to do a call-back of some sort to the channel.

Am I on the right track?

[dandriscoll]

That's correct, replies are sent through their own calls back to the channel. The ServiceUrl field in each activity you receive indicates where replies should be sent.

[pkaldawy]

We are facing an issue @dandriscoll. Wanted to know if that is related. I have attached the stack trace.

Exception: System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.Bot.Builder.Dialogs.Internals.Extensions.MaxBy1 items, Func2 selectRank, IComparer`1 comparer> at Microsoft.Bot.Builder.Dialogs.LuisDialog1.b__6_0(LuisResult result) at System.Linq.Enumerable.WhereSelectArrayIterator2.MoveNext() at System.Linq.Enumerable.WhereSelectEnumerableIterator2.MoveNext() at Microsoft.Bot.Builder.Dialogs.Internals.Extensions.MaxByT,R at Microsoft.Bot.Builder.Dialogs.LuisDialog1.

[dandriscoll]

Hi @pkaldawy, that is not related to the issue reported in this thread.

[AlavudeenRafiq]

I am getting the response from the Bot intermittently. While sending the request, I am getting the below error.

POST https://directline.botframework.com/v3/directline/conversations/B56KT664xddDwc9SBbL4md/activities 502 (Bad Gateway)

I am also getting error in botchat.js. How long will it take to resolve your loading issues in your European datacenter?

[dandriscoll]

Hi @AlavudeenRafiq, I'm very sorry, I just noticed your issue is a 502 and not a 503.

A 502 indicates the problem occurred within your bot.
A 503 indicates our server is not responsive, and is the issue we are dealing with in this thread.

If you're seeing a 502, please inspect the logs inside your bot to see if it is generating any errors.

Very sorry about miscategorizing the issue.

[AlavudeenRafiq]

Hi @dandriscoll,
If the problem is within the bot, I should not get any response right? But I am getting responses randomly. And my bot code works fine when I test with emulator.
So, can you suggest me how to resolve this issue?

[dandriscoll]

Hi @AlavudeenRafiq, when your Direct Line client issues a POST to /v3/directline/.../activities, Direct Line contacts your bot.

If your bot returns an HTTP 2xx code, we return 200 back to your client.
If your bot returns an HTTP 4xx or 5xx code, we return 502 back to your client.
If your bot does not respond within our timeout period, we return 504 back to your client.

Your bot may be experiencing an issue that isn't easily triggered with the emulator.

One common error occurs if you have a C# bot that uses optimistic concurrency when writing to state storage. You can read more about that here: https://docs.botframework.com/en-us/technical-faq/#what-causes-precondition-failed-412-or-conflict-409-http-errors

Your bot may be encountering a different error entirely, as well.

[willemodendaal]

Hi @dandriscoll, @AlavudeenRafiq . Related to the comments above: we have received many 502's, but never any 504's. However, if we send AppInsights traffic via the Bot Channel, it indicates that calls timed out to our bot (15 seconds +). In those cases we got 502's as well, even though our Bot never threw any errors of its own.

In our case the 15 second timeouts were happening because responses from our bot to directline were taking too long (EU datacenter issue if I understand correctly).

[dandriscoll]

I'll take a look at that -- we added the 504 code recently and it may actually not be live yet.

[willemodendaal]

We're on v3.5.0 at the moment and I see v3.5.2 was released recently. Could be that we're just a bit out of date. Would definitely be useful to distinguish 502s from 504s.

[AlavudeenRafiq]

Hi @dandriscoll,
Since the 504 code is not live yet, I may get 502 instead of 504. Consider I am facing 504 issue. What is the fix to that issue?

[JPThorne]

Hi @dandriscoll we are unfortunately still seeing these timeouts to the directline endpoint. It seems quite stable in the morning on our side, but then come midday the frequency of the errors increase. Our appinsights logging is indicating that calls to directline timed out after 15s.

Is there any way you can provide us with an indication of a time frame around these fixes?

[dandriscoll]

Yes, we're verifying a fix and hope to have it in production today.

[AlavudeenRafiq]

Hi @dandriscoll,
Still, I am facing the issue(502 bad gateway error). Have you made a fix for that issue?

[dandriscoll]

Hi @AlavudeenRafiq, the platform issues are resolved, and any 502s indicate your bot is returning an error.

[dandriscoll]

Closing as the platform issues have been resolved.

[alexandruf]

Hello,
Today, I experience 503 from 16:00. It's a production bot. Can you please tell us if there are any alarms?
Failed to load https://directline.botframework.com/v3/directline/conversations/6908fV1xj6v4IQxXDLu6gZ?watermark=: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'https://www.xxxxxxxxx' is therefore not allowed access. The response had HTTP status code 503.

[CalvinFengDatacom]

@dandriscoll Hi I'm so queries about why we are getting 502 bad Gateway issue intermittently from the directline:
https://directline.botframework.com/v3/directline/conversations/1qFnZ0XBpw3IkOJZjBxH0J/activities

From your comments above, the 502 indicates the issue from bot. But, if it's the issue from my bot, why I got it intermittently, not always? Are there any other aspects that may effects the bot? Like something may cause the time out or Azure App service?
We are based on New Zealand, so does the distance may cause some issue as well?

[prashantsoni]

Is this fixed, I am facing the same issue. Getting 502 intermittently for both web chat client and direct line.

[mutanttech]

This has been happening at different time when Bot returning 502 Bad Gateway Errors. On checking Application Insights noticed BotTimeoutException Logged.
9/25/2018, 12:43:55 PM
POST to wab-nl2-XXX-chatbot-1 timed out after 15s
Problem Id:Microsoft.Bot.Schema.BotTimeoutException at Microsoft.Bot.ChannelConnector.BotAPI+d__30.MoveNext

How to resolve this issue as it comes as surprise and production bots gets unresponsive to users? Please help.

[EricDahlvang]

@mutanttech Please open a new issue. This issue is closed. The team does not actively monitor closed issues.