Botframework-sdk: Buffer cannot be null. Parameter name: buffer

[MohammedAbdulMateen]

Bot Info

• SDK Platform: .NET
• SDK Version: 3.13.1
• Active Channels: Skype, MS Teams, Slack, Skype for Business, WebChat, Emulator
• Deployment Environment: Azure Bot Service, Azure App Service and local development & testing with Emulator and ngrok.

Issue Description

Getting ArgumentNullException with the following message, Buffer cannot be null. Parameter name: buffer. ParamName: "buffer", Source: "mscorlib"

The following is the stack trace,

   at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
   at System.IO.MemoryStream..ctor(Byte[] buffer)
   at Microsoft.Bot.Builder.Azure.BotDataEntity.Deserialize(Byte[] bytes)
   at Microsoft.Bot.Builder.Azure.TableBotDataStore.<Microsoft-Bot-Builder-Dialogs-Internals-IBotDataStore<Microsoft-Bot-Connector-BotData>-LoadAsync>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Bot.Builder.Dialogs.Internals.CachingBotDataStore.<LoadFromInnerAndCache>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Bot.Builder.Dialogs.Internals.CachingBotDataStore.<Microsoft-Bot-Builder-Dialogs-Internals-IBotDataStore<Microsoft-Bot-Connector-BotData>-LoadAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at AppName.Filters.RefreshTokenFilter.<OnActionExecutingAsync>d__0.MoveNext() in D:\TFS\Source\Workspaces\TP_APPNAME\Development\Source\Dev_AppName_Version1\FolderName\AppName\Filters\RefreshTokenFilter.cs:line 49

Everything was working fine and we didn't change anything in our code but this issue started showing up from 2/8/2018. We are using Azure Table Storage by following this article here

Code Example

The following statement is throwing the exception
var userData = await botDataStore.LoadAsync(key, BotStoreType.BotUserData, CancellationToken.None);

Registering the Azure Table Storage

Conversation.UpdateContainer(builder =>
{
    builder.RegisterModule(new AzureModule(Assembly.GetExecutingAssembly()));

    // Bot Storage: register state storage for your bot
    var connectionString = ConfigurationManager.ConnectionStrings[Constants.Database.BotStorageConnectionString].ConnectionString;
    var store = new TableBotDataStore(connectionString);

    builder.Register(c => store)
    .Keyed<IBotDataStore<BotData>>(AzureModule.Key_DataStore)
    .AsSelf()
    .SingleInstance();
});

The code that throws the exception,

public class RefreshTokenFilter : ActionFilterAttribute
{
    public async override Task OnActionExecutingAsync(HttpActionContext actionContext, CancellationToken cancellationToken)
    {
        try
        {
            object activity;
            actionContext.ActionArguments.TryGetValue(Constants.MessagesAndOptions.Activity, out activity);
            IMessageActivity message = activity as IMessageActivity;

            if (message != null &&
                message.Text != null &&
                message.Type == ActivityTypes.Message &&
                !Constants.MessagesAndOptions.BotResetCommand.Contains(message.Text.ToLower()))
            {
                using (var scope = DialogModule.BeginLifetimeScope(Conversation.Container, message))
                {
                    var botDataStore = scope.Resolve<IBotDataStore<BotData>>();
                    var key = new AddressKey
                    {
                        BotId = message.Recipient.Id,
                        ChannelId = message.ChannelId,
                        UserId = message.From.Id,
                        ConversationId = message.Conversation.Id,
                        ServiceUrl = message.ServiceUrl
                    };

                    var userData = await botDataStore.LoadAsync(key, BotStoreType.BotUserData, CancellationToken.None); // problem occurs here
                    // ... code omitted for brevity
                }
            }
        }
        catch (HttpOperationException ex)
        {
            // error handling due to concurrency issues (another instance of the bot has changed the data)
            ErrorSignal.FromCurrentContext().Raise(ex);
        }
        catch (Exception ex)
        {
            ErrorSignal.FromCurrentContext().Raise(ex);
        }

        await base.OnActionExecutingAsync(actionContext, cancellationToken);
    }
}

Reproduction Steps

1.
2.
3.

Expected Behavior

Exception shouldn't occur and the user data should load properly.

Actual Results

Buffer cannot be null. Parameter name: buffer

[EricDahlvang]

i'm unable to repeat this. Is the error occurring on all channels? or just the emulator?

[NiteLordz]

Yea, this happens on all channels (at least on FB and emulator).

So I exported my old botdata table and deleted it. I was hoping to reproduce the table, and start fresh. However, when I run the code when the bot calls PromptDialog.Confirm(...) I get an error message of

Exception: Bad Request [File of type 'text/plain']

Checking within Azure Storage Explorer, I see the table was created, but it doesn't have the fields, and no data is present. My guess is it is throwing an exception during the write operation (i am speaking out of turn here I know).

[EricDahlvang]

@NiteLordz Which BotBuilder libraries are you using?
For instance, the package.json for one of my projects is:

  <package id="Microsoft.Bot.Builder" version="3.13.1" targetFramework="net46" />
  <package id="Microsoft.Bot.Builder.Azure" version="3.2.5" targetFramework="net46" />
  <package id="Microsoft.Bot.Builder.History" version="3.13.1" targetFramework="net46" />
  <package id="Microsoft.Bot.Connector" version="3.13.1" targetFramework="net46" />

[NiteLordz]

@EricDahlvang

<package id="Microsoft.Bot.Builder" version="3.13.1" targetFramework="net461" /> <package id="Microsoft.Bot.Builder.Azure" version="3.2.5" targetFramework="net461" /> <package id="Microsoft.Bot.Builder.History" version="3.13.1" targetFramework="net461" /> <package id="Microsoft.Bot.Connector" version="3.13.1" targetFramework="net461" />

[EricDahlvang]

I am unable to reproduce this with the emulator, or the webchat channels. I've tried starting with sdk v3.12.2.4 saving some state, upgrading to v3.13.1, then re-connecting with the same user: no errors. I tried starting with empty Table Storage table and 3.13.1: no errors.

Can either of you provide a small sample project that reproduces the bug?

[NiteLordz]

I have a solution, however, I don't want to publish my connection string, is it possible to send it to you via email?

Reproduction steps are as follows

1. Create new bot application using this
2. Enabled table storage using this
3. Ran the application (note application runs with Bot.Builder SDK 3.12.2.4)
4. Updated all NuGet packages (~20), including Bot.Builder to 3.13.1.
5. Ran application again, and got the following error
   Exception: Bad Request [File of type 'text/plain']

and I used my connection string to my Azure table storage

Bot Application1.zip

[youtoofan]

I fixed it by updating the Microsoft.Azure.DocumentDB package to the latest version (1.20.1)
Prior to this, I deleted the old botstorage table. Not sure if this was necessary though.

Hope it helps.

[teslavitas]

Updating the Microsoft.Azure.DocumentDB to 1.20.1 and deleted table with bot data worked for me.
Just updating the package didn't help, not sure if this step is necessary.

[NiteLordz]

I am not using DocumentDB for storage. I am using Table storage. Also, deleting the table isn't a good solution as ours has been storing for over a year now, and losing the data that is stored is not going to over well.

@EricDahlvang has any progress been made?

[EricDahlvang]

@NiteLordz Sorry, I did not make any progress over the weekend. I'm looking at this today though.

[NiteLordz]

Cool. Let me know if there is anything you want me to try or if any additional information is needed.

[EricDahlvang]

Hey @NiteLordz

Would you please try this:

1. Create new bot application
2. Enabled table storage
3. Updated all NuGet packages (20), including Bot.Builder to 3.13.1

I haven't determined what exactly the issue is, but there seems to be some incompatibility with the latest packages.

[NiteLordz]

@EricDahlvang

Steps taken

1. Created new bot application
2. Configured Table storage
3. Ran bot successfully
4. Updated Microsoft.Bot.Builder to v3.13.1
5. Ran bot successfully.

Continuing to test

1. Updated Autofac to v4.6.2 (Successful run).
2. Updated WindowsAzure.Storage to v9.0.0 (FAILED with Bad Request)

Additional note: downgrading WindowsAzure.Storage to 8.7, 8.6 and 8.5 continues to fail upon running.

Additional note 2: Blew away table, started at WindowsAzure.Storage v8.5, stepped thru updates. All worked fine, until updating to v9.0.0, and got Bad Request error again.

[tiloc]

Maybe this additional info helps with your trouble shooting. I ran into this issue when I finally migrated my bot from the non-production legacy storage to use Azure Table Storage.

My steps were:

1. Have an existing bot application with ALL NuGet packages on latest stable release (Bot Builder SDK 3.13.1). Bot Application is still on legacy state storage at this point
2. Add packages for Azure based state storage. Upgrade again everything to latest stable packages. WindowsAzure.Storage is at 9.0
3. Run bot. Bot fails with the "Buffer cannot be NULL message"
4. Blast the new "botdata" table in the Table Storage
5. Downgrade WindowsAzure.Storage to 8.7
6. Run bot. Bot runs successfully

[NiteLordz]

So at this point, it sounds like the only way to run WindowsAzure.Storage above 8.7 is to start with a clean table. This, unfortunately, is not a viable option for me at this point.

I'm hoping they can resolve the issue in WindowsAzure.Storage 9.1 so that blowing the table away is not the solution.

[tiloc]

@NiteLordz I didn't specifically mention this in my report, but merely blasting the table and not downgrading to 8.7 resulted in the same error again. I think there currently simply is no way to run using WindowsAzure.Storage 9.0 with Azure TableStorage. The reports at the beginning of this thread seem to indicate that the story for DocumentDB might be different.

[NiteLordz]

@tiloc yea, I have a second version for testing purposes, DocumentDB works fine. The problem is, migrating the table data that exists into a DocumentDB (not sure that can be done), or finding a way to preserve the table data, and get it to work with WindowsAzure.Storage 9.0 (and future releases).

[jkolonko]

As far as I can see the problem is related to column names casing. WindowsAzure.Storage 9.0 creates camel cased columns (botId, channelId, conversationId, data, userId) whereas previous versions use Pascal Case (BotId, ChannelId, ConversationId, Data, UserId). I could not get version 9 working with Bot Framework, even after deleting botdata table.

[AndreMantas]

Same problem here.
Could not get it to work with latest version, even after deleting the table.
I've noticed that the first message works but after that it keeps getting the "Buffer cannot be null" error".

Downgrading to version 8.7 and deleting the table seems to solve the problem.

[BelvedereHenrique]

@AndreMantas
Worked. Thanks.

[EricDahlvang]

This is not an issue with the BotBuilder sdk, but with https://github.com/Microsoft/BotBuilder-Azure

I'm closing this issue and I've added one to that GitHub repository. BotBuilder-Azure is open source, so if anyone cares to contribute a fix, please do.

https://github.com/Microsoft/BotBuilder-Azure/issues/58

[sk2andy]

for all of you considering deleting whole tables: just use version 8.7 of windows.azure.storage and delete the affected rows.