Botbuilder-dotnet: Cannot use AzureTableStorage to persist ConversationState for dialogs

[ADefWebserver]

Trying to use Azure table storage to handle dialog state.
File Storage works, but not Azure Storage.

When using this code for dialogs:

   public class EchoBot : IBot
    {
        private readonly DialogSet dialogs;
        public EchoBot()
        {
            dialogs = new DialogSet();
            dialogs.Add("greetings", new WaterfallStep[]
            {
                async (dc, args, next) =>
                {
                    // Prompt for the guest's name.
                    await dc.Prompt("textPrompt","What is your name?");
                },
                async (dc, args, next) =>
                {
                    await dc.Context.SendActivity($"Hi {args["Text"]}!");
                    await dc.End();
                }
            });

            // add the prompt, of type TextPrompt
            dialogs.Add("textPrompt", new Microsoft.Bot.Builder.Dialogs.TextPrompt());
        }

        public async Task OnTurn(ITurnContext context)
        {           
            // This bot is only handling Messages
            if (context.Activity.Type == ActivityTypes.Message)
            {
                // Dialogs
                var state = ConversationState<Dictionary<string, object>>.Get(context);
                var dc = dialogs.CreateContext(context, state);

                await dc.Continue();

                if (!context.Responded)
                {
                    await dc.Begin("greetings");
                }
            }
        }
    }

In startup.cs this works:

var dataStore = new FileStorage("c:/temp");
options.Middleware.Add(new ConversationState<Dictionary<string, object>>(dataStore));

But this doesn't

IStorage dataStore =
new Microsoft.Bot.Builder.Azure.AzureTableStorage("DefaultEndpointsProtocol=https;AccountName=aihelpwebsitestorage;AccountKey=**account Key**;EndpointSuffix=core.windows.net","helloworldbotsimpledialog");

options.Middleware.Add(new ConversationState<Dictionary<string, object>>(dataStore));

[ADefWebserver]

Working version: https://www.dropbox.com/s/fv31dqt8wr2oycw/HelloWorldBot-works.zip?dl=0
Not working version: https://www.dropbox.com/s/sbp5vl4c0drrtr2/HelloWorldBot-doesnotwork.zip?dl=0

[drub0y]

hey @adefwebserver, thanks for reporting this and supplying a repro. Can I ask you to please include the full details of the exception being thrown (i.e. $exception.ToString()) as well? Might help reach a resolution a little faster.

[ADefWebserver]

More details on the error:

exception.Message:
Parameter count mismatch.

exception.StackTrace:
at System.Reflection.RuntimeMethodInfo.InvokeArgumentsCheck(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.RuntimePropertyInfo.GetValue(Object obj, Object[] index)
at Microsoft.WindowsAzure.Storage.Table.EntityPropertyConverter.<>c__DisplayClass5_0.b__1(PropertyInfo propertyInfo) in C:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\Common\TableEntityPropertyConverter.cs:line 194
at System.Linq.Enumerable.AllTSource
at Microsoft.WindowsAzure.Storage.Table.EntityPropertyConverter.Flatten(Dictionary2 propertyDictionary, Object current, String objectPath, HashSet1 antecedents, EntityPropertyConverterOptions entityPropertyConverterOptions, OperationContext operationContext) in C:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\Common\TableEntityPropertyConverter.cs:line 184
at Microsoft.WindowsAzure.Storage.Table.EntityPropertyConverter.Flatten(Object root, EntityPropertyConverterOptions entityPropertyConverterOptions, OperationContext operationContext) in C:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\Common\TableEntityPropertyConverter.cs:line 75
at Microsoft.Bot.Builder.Azure.AzureTableStorage.ToTableEntity(String key, Object value)
at Microsoft.Bot.Builder.Azure.AzureTableStorage.<>c.b__7_0(KeyValuePair2 kv) at System.Linq.Enumerable.SelectListIterator2.MoveNext()
at System.Linq.Enumerable.WhereEnumerableIterator1.MoveNext() at System.Linq.Enumerable.Any[TSource](IEnumerable1 source)
at Microsoft.Bot.Builder.Azure.AzureTableStorage.d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Bot.Builder.Core.Extensions.BotState1.<Write>d__9.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Bot.Builder.Core.Extensions.BotState1.d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Bot.Builder.Core.Extensions.BotState1.<OnTurn>d__5.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Bot.Builder.Core.Extensions.CatchExceptionMiddleware1.d__2.MoveNext()

[drub0y]

ah, OK so that's telling me you have a NuGet package versioning issue somewhere... specifically related to Azure Storage SDK. I'm not in front of a PC at the moment, but check your version and see if it aligns with what the Bot SDK is expecting or if you maybe rev'd up a bit too high. Will check your repro code when I can.

[ADefWebserver]

This is all I have in my assembly configuration:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.All" Version="2.0.5" />
    <PackageReference Include="Microsoft.Bot.Builder.Azure" Version="4.0.1-preview" />
    <PackageReference Include="Microsoft.Bot.Builder.Core" Version="4.0.1-preview" />
    <PackageReference Include="Microsoft.Bot.Builder.Core.Extensions" Version="4.0.1-preview" />
    <PackageReference Include="Microsoft.Bot.Builder.Dialogs" Version="4.0.1-preview" />
    <PackageReference Include="Microsoft.Bot.Builder.Integration.AspNet.Core" Version="4.0.1-preview" />
  </ItemGroup>

</Project>

[ADefWebserver]

The only thing I can add is that I do have the latest .Net Core SDK installed on my system.

[chintan3100]

I am facing the same issue. Any solution?

[GhostAtom]

I have the same issue.

[drub0y]

Ok, so I finally got a sec to look into this. It's not the NuGet package mismatch issue that I thought it might be at first glance, so let's forget about that.

This is due to a bug in the EntityPropertyConverter inside of the Azure Storage SDK where, when it tries to serialize types that have an indexer property (such as Dictionary<K, V>) it will throw this error because it's attempting to read what it thinks is a plain-old-property, but really it needs to be walking through all the keys and calling the indexer with each key as a parameter.

So the bad news is: right now, you can't use anything with an indexer in the hierarchy of your bot state classes when using the Azure Table Storage provider w/the current preview bits. 😢

Perhaps worse news that you should probably prepare for is: the Azure Table Storage provider has been removed in the more recent bits and will only be provided as a sample. 😭

[ADefWebserver]

@drub0y - What storage should we use? Asking because I need to update my article: http://aihelpwebsite.com/Blog/EntryId/1031/Saving-User-and-Conversation-Data-MS-Bot-Framework-V4-Preview-Edition Thanks!

[drub0y]

If you take a look at the current state of master, the only Azure based storage provider that's still "in-the-box" as of right now is the CosmosDbStorage. So, for now, I suppose that's your safest bet.

Again, to be clear, I don't think the team's absolutely finalized this decision at this point so it might make a return. Keep your eyes peeled... if you really need blob or table storage, and it's not there in the next officially released preview bits, definitely file an issue so the team understands how important it is to you. Also, since the v4 framework _is_ actually so extensible in this regard, it's not inconceivable that an implementation is shipped as a standalone package that you can just plug-in.

[ADefWebserver]

@drub0y - Thanks for the heads up. I will update the code to use CosmosDbStorage for now. This is how pre-release software goes, I'm used to it.

[ADefWebserver]

I updated my article to use Cosmos DB http://aihelpwebsite.com/Blog/EntryId/1031/Saving-User-and-Conversation-Data-MS-Bot-Framework-V4-Preview-Edition

[jorupp]

It'd definitely be nice to still use table storage for persistence even if it gets a not-great-for-production tag (is it's latency a problem?) or something. Unless I'm misreading it, for a not-used-a-lot bot, bottom-end price point for a Cosmos instance is $25/mo, while azure table storage is trivial.

Of course, given the extensiblity goal there, writing it myself for apps probably won't be hard anyway.

[JasonJShuler]

What must we do to encourage the team to maintain Azure Storage account implementations?

CosmosDB is considerably more expensive than table or blob storage, and I would strongly disagree with any claims that it is not fine for production (VMs use storage accounts)

From what I have read, the performance cosmos offers comes at a high cost, and the value just isn't there.

I certainly hope this decision is not financially motivated...

[sgellock]

@ADefWebserver we have decided to deprecate table store and replace it with blob storage, cosmosDB, and memory storage as our storage providers.

[jorupp]

Won't claim to speak for the others who chimed in here, but for my use case, blob storage will work fine - just looking for something persistent, out-of-process, and able to scale cost way down when usage is very low. Glad to hear blob storage support will be in the box even if table isn't.

[ADefWebserver]

@sgellock - That works. Thanks!