Botframework-sdk: Bot Stopped Working After Migration "[Error] Microsoft.Azure.WebJobs.Host: Timeout value of 00:05:00 was exceeded by function"

[justinh00k]

Bot Info

• SDK Platform: Node.Js
• SDK Version:
• Active Channels: Skype, Slack, Messenger, Kik
• Deployment Environment: Azure Bot Service

Issue Description

Since migrating, bot times out on almost all requests.

Code Example

https://github.com/justinh00k/autocompetebot-update

Reproduction Steps

Bot was just migrated.

Expected Behavior

Bot still works when it doesn't time out, which is almost always. Bot had been running continuously for over a year previously.

Actual Results

2018-04-05T19:15:03.711 [Error] Microsoft.Azure.WebJobs.Host: Timeout value of 00:05:00 was exceeded by function: Functions.messages.

[EricDahlvang]

Hi @justinh00k

What is the bot handle and MicrosoftAppId?

Which version of the BotBuilder sdk are you using?

Is the bot using the default state service?

[justinh00k]

@EricDahlvang Thank you.

What is the bot handle and MicrosoftAppId?

@autoCompeteBot https://bots.botframework.com/bot?id=autoCompeteBot

Which version of the BotBuilder sdk are you using?

3.14.1

Is the bot using the default state service?

I assume so. How would I check this?

[EricDahlvang]

If you have not implemented a custom state store, then you are using the default (not intended for production bots).

There is information on our blog about the deprecation of the default state service: https://blog.botframework.com/2018/04/02/reminder-the-bot-framework-state-service-has-been-retired-what-you-need-to-know/ There you will find links to resources describing how to use BotBuilder-Azure to implement a custom state store using CosmosDb, Azure Table Storage and Azure SQL.

[justinh00k]

The instructions said no downtime, click Migrate. Where was all the info about rewriting the entire code? This is a nightmare and my bot is just permanently down till I can parse through all this new and uncollected info????

[EricDahlvang]

You can rollback the migration if you believe it has caused issues with your bot. This is the first report I've seen where performance is impacted by migration.

Please share your actual BotHandle and MicrosoftAppId so I can investigate.

[EricDahlvang]

Also, the project you've linked looks to have a partial implementation of a custom state store using botbuilder-azure: https://github.com/justinh00k/autocompetebot-update/blob/master/index.js#L15

Just missing this: .set('storage', tableStorage); Here is an example: https://github.com/Microsoft/BotBuilder-Azure/blob/master/Node/examples/feature-documentDb/app.js

[justinh00k]

It's partial because I just started it. It's only incomplete because Azure STILL hasn't pushed the Github changes.

MiscrosoftAppId is 1bd06027-2491-4709-ba31-01d101f57e2d

Not sure what you mean by "actual" bot handle if it's not autoCompeteBot

[justinh00k]

The problem continues. When I redo the entire bot from scratch it works fine, but the migrated one won't work, and that's the one that's published to numerous platforms.

[Andrea-Orimoto]

Hi @justinh00k, first of all, sorry for the inconvenience. Have you updated your code to use FuncPack? I see your bot is a Functions bot in Node.js, which is the one case that we highlight in the migration doc https://docs.microsoft.com/en-us/azure/bot-service/bot-service-migrate-bot

Note

Before migrating a Functions bot using Node.js, it is recommended that you use the Azure Functions Pack to package the Node.js modules together. Doing so will improve performance during migration and execution of the Functions bot after it is migrated. For more information, see Azure Functions Pack.

The fact that a brand new bot works leads me to believe that this might be the issue.

Again, apologies for the inconvenience, and please let us know if this fixes your issue.

[justinh00k]

Thanks @Andrea-Orimoto. Yeah i tried setting it up for funcpack and that didn't seem to help. I tried any number of things. Eventually i had to unmigrate and remigrate, and that resulted in continual "Error 500"s that creating and switching users didn't fix. That's where I stopped, because now when I login to try, it says "Provisioning your bot..." and that never changes.

[Andrea-Orimoto]

oh I see, I'll get an engineer involved tomorrow. We might need to ask you to give us temporary access to your subscription. I'll let you know if that's the case.
Sorry for the inconvenience.

[justinh00k]

That would be totally fine. Appreciate your help!

[Andrea-Orimoto]

Hi @justinh00k, we spent some time digging in the logs, have some clues, but it would help to get access to your subscription. Can you please temporarily add andreo(at)microsoft.com and hailiu(at)microsoft.com as contributors to your subscription?

Also, a few questions:

1. this bot appears to be a legacy Azure Bot Service functions bot. I would have expected to see an endpoint with a functions code, but I see a clean url like /api/messages. are you currently using a webapp for this bot?
2. the portal now loads instead of saying "Provisioning your bot...", have you changed anything?

[justinh00k]

You are added!

1) It was a functions bot, and when that wasn't working I switched the endpoint to a new webapp. That worked for a time (without migration), then that stopped working too.
2) Nope!

[Andrea-Orimoto]

hi @justinh00k, only hailiu has received the invitation, can you please check andreo?

A few things going on here... but first, can you check that the MicrosoftAppPassword in the app settings is correct for MiscrosoftAppId 1bd06027-2491-4709-ba31-01d101f57e2d? We see one that starts with "stwX" in your app settings. Please check on apps.dev.microsoft.com that it is indeed correct. If not, generate a new one and update the appSettings.

Finally, the new webapp you created is actually a new Azure Bot Service webapp, instead of a simple web app. Here's what we're thinking comes next:

1. we work together to make the bot work again
2. once it works, we patch your AutoCompeteBot bot record so that it becomes a registration bot
3. we delete the new "googlefeud2018" bot record you created (only the bot record, not the new "googlefeud2018" appservice)
4. you migrate your AutoCompeteBot (this time it's a simple migration, it shouldn't cause any issue)

makes sense? let me know if you have any questions.
andrea

[justinh00k]

Fixed andreo.
Fixed the password. (That was why it stopped working again!)
Patch ahoy. Thanks!

[EricDahlvang]

Great!

Closing this issue. Thanks @Andrea-Orimoto !!

Sorry, jumped the gun. :/

[Andrea-Orimoto]

That's great news. So here's what we're doing now:

1. we'll patch your AutoCompeteBot bot record, so that it becomes a registration bot
2. you run the migration, which we're confident will work. If it doesn't allow you to select an existing resource group, then let it create a new one, and then move the AutoCompete bot service to the autoCompeteBotGroup Resource Group
3. we delete the googlefeud2018 bot service record

After this you will have the AutoCompeteBot bot service in Azure and your googlefeud2018 App Service.
Makes sense?

I'll let you know when 1. is done

[Andrea-Orimoto]

Ok, #1 is done. please proceed with #2 and let me know when done.

[justinh00k]

Hey, it works! (Feels so good to hit Migrate and have it succeed haha). Thanks so much for your assistance!!

[Andrea-Orimoto]

yeah! it created a new Resource Group as I suspected. just move the autoCompeteBot Bot Service to the autoCompeteBotGroup Resource Group and you're set. We'll go ahead and delete the googlefeud2018 Bot Service (we'll do it, don't do it yourself).
btw, what is the AutoCompete2018 Bot Service? another test you were doing? If so, you might delete that one yourself.

[justinh00k]

Yes that was another attempt. Just deleted most of it, but got this error:

Server farm 'AutoCompete2018' cannot be deleted because it has web app(s) googlefeud2018 assigned to it.

And I swapped the resource group for the autoCompeteBot Bot Service too.

[justinh00k]

Could you tell me or show me how to find out which resources the new bot is using? There are tons of services in there (including adding Cosmos) I'm sure I don't need anymore.

[Andrea-Orimoto]

oh I see, you have put the googlefeud2018 webapp into that app service plan. I'd say, move the AutoCompete2018 app service plan into the autoCompeteBotGroup resource group, and then delete the AutoCompete2018 and autoCompeteBot resource groups (the latter was created by the migration).

I have deleted the googlefeud2018 bot service. once you do the above you should be set. I'll close this issue, let me know if you have any further questions. And sorry for the inconvenience!

[Andrea-Orimoto]

oh, let me check your resources...

[Andrea-Orimoto]

these seem to be unused, but please double check.
AppInsights: autoCompeteBotyveu4q (not used by the bot)
Storage: autocompetebotyveu4q (not used by the bot)
App Service Plan: WestUSPlan (no apps in it)
App Service Plan: autoCompeteBot-farm (no apps in it)

[Andrea-Orimoto]

oh, btw, you can remove us from your subscription.

[justinh00k]

:pray: Thanks, @Andrea-Orimoto.