Botframework-sdk: 500 server error using facebook

[tpowellmeto]

Bot Info

• Bot handle: informatics-lab-sol
• App ID: c0dd83ca-0370-4310-96a6-c418c924870f
• SDK Platform: Node.js
• SDK Version: 3.9.1
• Active Channels: Facebook
• Deployment Environment: aws ec2

Issue Description

My service is returning 500s constantly

Error: Request to 'https://state.botframework.com/v3/botstate/facebook/users/1420943404688109' failed: [500] Internal Server Error

[jeffhee]

I am experiencing the same thing. Is there a status page somewhere?

[jacobtomlinson]

I found this page, but it shows the service as being up.

https://azure.microsoft.com/en-gb/status/

[advl03]

Same error 500 here. Like 3 hours ago, we started to have this kind of errors in both our development (own openstack servers) and production environments (azure North EU).
Azure services status page does not show any alarm.
Any help is appreciated...

[EricDahlvang]

Thanks for reporting this. We will investigate, and report back our findings asap.

(Note, the azure status page does not show Bot Framework connector status. We are working on a status page.)

[KwstasBol]

the service is up, but it keeps having the same issue.. Thanks Eric,hope we find it soon,because it was all of the sudden without any change..

[tpowellmeto]

can you give us any more information?

[yichengsun]

Would appreciate an update soon, as we have a release slated for later this afternoon that will be impacted...

[edgarw19]

I have the exact same issue and this is a serious, high priority bug for our users. We have an enormous launch to thousands of people in just 2 hours...

I went into chatconnector.js and printed the body response back from Microsoft, and I get this

{ error:
{ code: 'ServiceError',
message: 'Null properties cannot be encrypted. Please assign a default value to the property if you wish to encrypt it.' } }
Error: Request to 'https://state.botframework.com/v3/botstate/facebook/users/2014981635209074' failed: [500] Internal Server Error
at Request._callback (/app/node_modules/botbuilder/lib/bots/ChatConnector.js:514:46)
at Request.self.callback (/app/node_modules/request/request.js:188:22)
at emitTwo (events.js:106:13)
at Request.emit (events.js:191:7)
at Request. (/app/node_modules/request/request.js:1171:10)
at emitOne (events.js:96:13)
at Request.emit (events.js:188:7)
at IncomingMessage. (/app/node_modules/request/request.js:1091:12)
at IncomingMessage.g (events.js:292:16)
at emitNone (events.js:91:20)
at IncomingMessage.emit (events.js:185:7)
at endReadableNT (_stream_readable.js:974:12)
at _combinedTickCallback (internal/process/next_tick.js:80:11)
at process._tickCallback (internal/process/next_tick.js:104:9)

[dan-cohn]

Likewise here. We have a user study and demo this afternoon. This is a very serious problem.

[JasonSowers]

We are actively investigating this. Will update when we have more details

[jphgross]

We're also experiencing the issue and its having a fairly large impact to us.

[sonnenhaft]

ddos? Big brother, speak to us. ETA?
Seems like SMS chat works, slack should not.
Also guess it is on FB and Slack side.

[sagarpomal]

Looks like it has started working.

[advl03]

Yes, confirmed. My dev and prod bots are working again! Thanks!

[tombarton]

We were experiencing similar issues earlier; looks like things have settled now. Is there any change of getting a status page anytime soon?

[edgarw19]

Oh wow, thank goodness, up and running 1.5 hours before our launch. Hopefully it's stabilized now.

[tpowellmeto]

Yes, thank you! Back up and running now. Any information on what caused this?

[JasonSowers]

There was an issue with null values in bot properties. We have done a rollback to fix the issue and are working on a hot fix to correct it permanently.

[EricDahlvang]

Thank you everyone, for reporting this to us. Please open a new issue if you run into any new problems.

[EricDahlvang]

Everyone please note: "By default, the Bot Framework SDK for .NET stores state data using the Bot Framework State service, which is intended for prototyping only and is not designed for use by bots in a production environment." https://docs.microsoft.com/en-us/bot-framework/dotnet/bot-builder-dotnet-state

Please consider using the Azure Extensions, or implementing a custom state client: https://blog.botframework.com/2017/07/18/saving-state-azure-extensions/

Note: Bots using a custom state client did not experience these issues on September 15th.

[dan-cohn]

You mention the .NET SDK, but this issue affected my Node.js client as well.

I'm a little surprised to hear that the State Service isn't meant for production. This is one of the best features of the Bot Framework. It significantly speeds up bot development and maintenance. Without it everyone has to implement their own solutions, many of which are likely to incur an operational cost on top of the added dev costs. Have you considered hardening the service so it's production-ready and perhaps even charging usage fees?

[EricDahlvang]

"It significantly speeds up bot development and maintenance." <-- yes, there are definitely benefits to the default connector state service. However, it is also relatively simple to include the Azure Extensions and use CosmosDb, Sql Server, or Azure Table Storage in either Node or .NET. https://github.com/Microsoft/BotBuilder-Azure

There are Nuget and NPM packages:
https://www.nuget.org/packages/Microsoft.Bot.Builder.Azure/
https://www.npmjs.com/package/botbuilder-azure

I do not know the plans for the current state service.

[KwstasBol]

?Thank you for the crucial information. I tried using bot service and it worked, i totally recommend it.

---

Από: Eric Dahlvang [e-mail]
Στάλθηκε: Τρίτη, 19 Σεπτεμβρίου 2017 9:31 πμ
Προς: Microsoft/BotBuilder
Κοιν.: Bolotis Konstantinos; Comment
Θέμα: Re: [Microsoft/BotBuilder] 500 server error using facebook (#3466)

"It significantly speeds up bot development and maintenance." <-- yes, there are definitely benefits to the default connector state service. However, it is also relatively simple to include the Azure Extensions and use CosmosDb, Sql Server, or Azure Table Storage in either Node or .NET. https://github.com/Microsoft/BotBuilder-Azure

There are Nuget and NPM packages:
https://www.nuget.org/packages/Microsoft.Bot.Builder.Azure/
https://www.npmjs.com/package/botbuilder-azure

I do not know the plans for the current state service.

-
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/Microsoft/BotBuilder/issues/3466#issuecomment-330442753, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ANrwG0bwjBrVJe13U5c4CWOhKkYs0o9vks5sj1-kgaJpZM4PX9hK.

[KwstasBol]

So, i guess that every bot framework solution already stored in the old way has to change to bot service? I am not capable of understanding this part of the problem.

[advl03]

Please confirm:
1) if node.js based bots must also implement a custom state client for storing conversation and user data.
2) if we need to remove every reference to session.userData, session.privateConversationData and son on in order to future proof our bots

Thanks!

[tombarton]

@advl03 yes, Node.js based Bots should be using external state storage, as advised in the bot framework documentation. You can then continue to use session based storage (e.g. session.privateConversationData) as normal.

[EricDahlvang]

@advl03

1) node.js bots should use one of the Azure Extensions (documentdb, azure table storage, or sql: coming soon), or implement your own state client (not that difficult). Also: this is not just for conversation and user data, but dialog data (like in a waterfall in node, or formflow in .net) are also stored in the state client.

2) no need to remove session.userData, etc: if the azure extensions implementation are used. Azure Extensions will substitute the default.