Botframework-sdk: Webchat # Directline APIs performance severely impacting user experience

[sharmadine]

Bot Info

• SDK Platform: Node.js
• SDK Version: 3.14.1
• Active Channels: Direct Line
• Deployment Environment: Azure Bot Service

Issue Description

For last 5-6 hours today, the response times from WebChat initiated DirectLine APIs are crossing 5+ seconds severely impacting user experience.

A few observations:

1. receiving frequent '403' errors
   GET https://directline.botframework.com/v3/directline/conversations/xxxxxx/activities?watermark=44 403 (Forbidden) botchat.js:20
2. Receiving frequent '503' errors

• Failed to load resource: the server responded with a status of 503 (Service Unavailable)
• Failed to load https://directline.botframework.com/v3/directline/conversations/xxxxxxxx/activities?watermark=37: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'https://URL.com' is therefore not allowed access. The response had HTTP status code 503.

Code Example

Reproduction Steps

The bot / webchat is working perfectly in production for last few months. The behaviour is observed for last 4-6 hours.

Expected Behavior

Directline APIs should not fail and lead to 'Retry' message behaviour.

Actual Results

Direct Line API long-poll calls are failing leading to 'couldn't send, Retry' message to users.

Please support to help us providing consistent and well performing user experience.

[JasonSowers]

Are you using the default state store or have you implemented a custom one?

[sharmadine]

We have been using custom state since beginning of Production deployment i.e. 3-4 months.

[JasonSowers]

Sorry for the delay, Is this still happening?

[sharmadine]

Not observed after that day but message deliveries were failing yesterday. No errors but messages were not getting delivered to connector and Webchat was too slow to interact. Looking at our bot service logs, we found that messages were not getting delivered to bot service.

Directline response times are not consistent, sometimes it gives error and sometimes it just goes silent impacting the user response times / interaction severely.

[JasonSowers]

take a look at this as it may provide some useful information for you https://blog.botframework.com/2017/11/02/optimizing-latency-bot-framework/

[sharmadine]

@JasonSowers we are getting 503 again now in our production bot hosted in West Europe.

Error: POST to 'https://directline.botframework.com/v3/conversations/xxxxx/activities/xxxxx' failed: [500]  
    at Request._callback (/usr/src/app/node_modules/bot-services/node_modules/botbuilder/lib/bots/ChatConnector.js:559:46)

Please suggest and support. The problem is becoming frequent and causing production bots downtimes.

[easkerov]

@JasonSowers @sharmadine We have have the same issue too on our production bot deployment

[JPThorne]

We are also having the same errors on all our bot environments. A sample error:

Failed to load https://directline.botframework.com/v3/directline/conversations/CJNVaedUKzPAuf4kKEn2hL/activities: 
Response to preflight request doesn't pass access control check: 
No 'Access-Control-Allow-Origin' header is present on the requested resource. 
Origin 'https://www.ouroriginwhichwasworking.com' is therefore not allowed access. 
The response had HTTP status code 503

This is on the C# SDK. Latest version.

[willemodendaal]

Same issue too. (and we definitely use a custom state client)

[mikethetike]

Same

[sharmadine]

The error that I am getting now is:

Request URL: https://directline.botframework.com/v3/directline/conversations
Request Method: POST
Status Code: 500 Internal Server Error

[viralgandhi]

Do we have a solid solution for this? This is affecting in production. We are getting below error
Failed to load https://directline.botframework.com/v3/directline/conversations: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'https://preview.c9users.io' is therefore not allowed access. The response had HTTP status code 503.

[kanishk-mehta]

Having the same issue. Bots are in production.

[bhushanvadgave]

Same issue for us as well, causing production downtime. :(

[EricDahlvang]

@sharmadine @easkerov @JPThorne @willemodendaal @mikethetike @viralgandhi @kanishk-mehta @bhushanvadgave The issue from 13 hours ago is tracked here: https://github.com/Microsoft/BotBuilder/issues/4738

4686 is a different issue from two weeks ago.

[JasonSowers]

@sharmadine are you still experiencing any issue?

[sharmadine]

@JasonSowers I have not seen Direct Line 500/503 errors in last couple of days. We are monitoring it closely and will share if observe the errors.

[JasonSowers]

@sharmadine I am going to close this issue as it was overtaken with non-related posts. If you find you are still experiencing this or something new please make a new issue detailing the behavior.

[Bradrajkumar]

i am also facing the issue that after chatting with bot i left for 2 hours, again i returned and tried to chat but bot is not sending the message, it is showing "sent Retry"

[EricDahlvang]

Hi @Bradrajkumar

This is a closed issue. Closed issues are not monitored by the team. Please direct your question to StackOverflow The Bot Framework Support Team monitors Stack Overflow daily.

That said: please ensure you are on the latest version of the v3 or v4 sdk. Also, ensure AlwaysOn is set for the web application hosting the bot.