Botframework-sdk: [State Service] How to save bot chat history ?

[rjgmail88]

Hello,

I am using node js sdk to write ms bot service. Currently, I am saving everything in data bags (conversationData, userData etc) and using documentDb to store the state. I have also implemented middleware logging to intercept user to Bot and bot to user messages and logging them on console.
There are few concerns with this approach,

1. conversationData is cleared when we call endConversation() in dialog. Thats expected by design but we would like to persist this data for multiple conversation flows with same user (same conversation id.) Now, json document in db gets replaced with new keys on conversationData when user start new intent.
   Ex: schedule a meeting with {name} for {day} at {place}.
   we save conversationData.name , conversationData.day , and conversationData. place.

same user starts over schedule a meeting with {name2} for {day2} at {place2}.
documentDb entry gets replaced with conversationData.name1 , conversationData.day2 , and conversationData. place2

Ideally, we would like to keep everything.

1. Issue 2 is middleware logging is not able to hold session object so I can't save each event.text to any databag? Is this correct approach or there is better way to save chat history ?

const logUserConversation = (event) => {
console.log(' ** BOT/User Message: ' + event.text);
//Want to save event.text somewhere ??
};

//Middleware for logging
bot.use({
receive: function (event, next) {
logUserConversation(event);
next();
},
send: function (event, next) {
logUserConversation(event);
next();
}
});

any help would me much appreciated. Thanks in advance.

[rjgmail88]

Also, I am not sure why ConversationData data bag is cleared even though I set to true as follows for persistence.

bot.set('persistConversationData', true);

[sphanley]

If you want to data that will persist for the lifetime of a user rather than a single conversation, you can save to userData rather than conversationData.

As for overwriting the data - the data bags are essentially just key-value pair dictionaries. If you store data with the same key, you're going to overwrite previous records, so you need to either use different keys each time, or store some sort of data structure such as a list which would let you persist multiple values within one key/value pair.

[rjgmail88]

Thanks Sam.
If I continue to use userData bag instead conversation do you think it will work for multiple users ?Does this bag gets created per user ? In documentDb I could only see a entry 'default-user:userData'

Also, any thoughts/ideas on storing chat history ?

[sphanley]

Yes, userData is per-user. There's three data bags with different uses: userData is per-user and will persist/share across multiple conversations, while conversationData is per-conversation, and in a group conversation, all users will share the same conversationData dataBag. There's also privateConversationData, which is per-conversation but also per-user (i.e. it doesn't persist/share beyond the lifetime of the conversation, but each user within a group conversation will have their own privateConversationData).

I'll have a look back over your initial question again when I've got more time later on, and see if I have any other advice about storing chat history. Hope that helps in the meantime, though!

[rjgmail88]

Appreciate your help Sam.

[rjgmail88]

Currently, I'm taking this approach where I create chatData array in main app.js file, keep filling it will text from bot/user, pass it to schedule.js so Calendar.Add dialog and user to store in userData.

//App.js code

//Middleware for logging(RJ-db,logging)
bot.use({
receive: function (event, next) {
logUserConversation(event);
next();
},
send: function (event, next) {
logUserConversation(event);
next();
}
});

var chatData = []

const logUserConversation = (event) => {
chatData.push(event.text)
console.log('* BOT/User Message: ' + event.text) //+ ', user: ' + event.address.user.name);
};

require('./app/dialogs/schedule.js')(bot,chatData);

// schedule.js code

bot.dialog('Calendar.Add',......
function (session) {
session.userData.chatData = []
session.userData.chatData.push(chatData)
}

But, with this chatData is available for other user to so when I look at session.userData.chatData for user 2, it has chat history of user 1 + its own.

[nwhitmont]

We have a great blog article about creating your own custom state service here:
https://blog.botframework.com/2017/07/21/saving-state-azure-nodejs/

[rjgmail88]

Thank you.

[rjgmail88]

I have gone through the document provided above https://blog.botframework.com/2017/07/21/saving-state-azure-nodejs/
but again it's a same which I had referred earlier. Issues I posted in my thread above still persist.

Just setting DocumentDb isn't helpful as data bags are cleared and also document does not talk able storing chat history.

[EricDahlvang]

@rjgmail88 Which channel are you using? "when I look at session.userData.chatData for user 2, it has chat history of user 1 + its own" <-- user 1 and user 2 must have the same userid for this to occur. How are you initiating the chat?

[rjgmail88]

I'm using Cortana channel but this is happing in bot emulator testing. here is code I have

//in server.js

var chatData = []
const logUserConversation = (event) => {
chatData.push(event.text) // This will push every message in chatData array.
console.log('** BOT/User Message: ' + event.text)
};

//Middleware for logging
bot.use({
receive: function (event, next) {
logUserConversation(event);
next();
},
send: function (event, next) {
logUserConversation(event);
next();
}
});

// In schedule.js
I simply do session.userData.chatData.push(chatData) so it will be logged in cosmosDb

Now if I use 2 clients 1) Cortana and 2) MS teams with 2 different users then when I see session.userData.chatData I see both conversations.

There might be better approach to store chat history. I'am trying to find one.

[lakmals]

I'm facing the same issue.

Is there any way I could use "session.userData.chatData =[]" instead of "var chatData = []"

[EricDahlvang]

@rjgmail88 Based on the code you've shown here, I see why all data is being persisted to every user's .userData. Every message that comes into your bot is being pushed onto chatData. Please review https://blog.botframework.com/2017/07/21/saving-state-azure-nodejs/ as well as https://github.com/Microsoft/BotBuilder-Samples/tree/master/Node/core-CustomState

Add the necessary keys to your .env file. Place var azure = require('botbuilder-azure'); at the top of the .js file. Then, the following:

// Azure DocumentDb State Store
var docDbClient = new azure.DocumentDbClient({
    host: process.env.DOCUMENT_DB_HOST,
    masterKey: process.env.DOCUMENT_DB_MASTER_KEY,
    database: process.env.DOCUMENT_DB_DATABASE,
    collection: process.env.DOCUMENT_DB_COLLECTION
});
var botStorage = new azure.AzureBotStorage({ gzipData: false }, docDbClient);

// Set Custom Store
bot.set('storage', botStorage);

// Enable Conversation Data persistence
bot.set('persistConversationData', true);

This will enable logging of incoming and outgoing messages into the documentdb storage.

[rjgmail88]

Hey EricDahlvang,
I am already doing what you have mentioned above. When I mentioned every user has same copy of chatData I meant it's in DocumentDb. Sorry if I wasn't clear.

[EricDahlvang]

This code will cause every message to be persisted to every user's chatData:

var chatData = []
const logUserConversation = (event) => {
chatData.push(event.text) // This will push every message in chatData array.
console.log('** BOT/User Message: ' + event.text)
};

//Middleware for logging
bot.use({
receive: function (event, next) {
logUserConversation(event);
next();
},
send: function (event, next) {
logUserConversation(event);
next();
}
});

// In schedule.js
I simply do session.userData.chatData.push(chatData) so it will be logged in cosmosDb

When doing session.userData.chatData.push(chatData), you'll need to make sure that you are only adding messages from or to this user.

[lakmals]

Hi rjgmail88,

Found a workaround for this using dynamic variables.

In server.js

var chatHistory= []
const logUserConversation = (event) => {
chatHistory[session.address.conversation.id]) += (event.text);
console.log('** BOT/User Message: ' + event.text);
};

//Middleware for logging
bot.use({
receive: function (event, next) {
logUserConversation(event);
next();
},
send: function (event, next) {
logUserConversation(event);
next();
}

In dialogs

use chatHistory[session.message.address.conversation.id]

[rjgmail88]

Hi lakmals, thanks for sharing this. I wonder how are you using [* ] in dialogs. At what step ?

• chatHistory[session.message.address.conversation.id]

[lakmals]

Your question is not clear. I'm using chatHistory[session.message.address.conversation.id] to trigger a mail with nodemailer.

ex:
bot.dialog('DialogTest', [
function (session) {
session.send(chatHistory[session.message.address.conversation.id]);
}
]);

I would like to know the steps, on how to save this string in a database (like Azure Storage).

[rjgmail88]

If you see in one of my comments I'm doing following

// in app.js
var chatData = []

const logUserConversation = (event) => {
chatData.push(event.text)
console.log('* BOT/User Message: ' + event.text) //+ ', user: ' + event.address.user.name);
};

// in sever.js
bot.dialog('Calendar.Add',......
function (session) {
session.userData.chatData = []
session.userData.chatData.push(chatData)
}

so that how conversation is available in datBag session.userData.chatData. So I was wondering how it's happening in your case for chatHistory.
OR I should say how does it look like in document Db.

[EricDahlvang]

@rjgmail88 Are you still experiencing this problem, or have you found a resolution?

[saifalamstats]

Hi. I do have the above discussed task, at hand too.
After reading the whole thread, I got a pretty good direction to start with.

However, it will really be good if someone helps me understand what is 'Server.js' and 'Schedule.js'.
The bot I am working on, only appears of have an 'index.js' json file, where I can code the body.

I do have limited knowledge in json handling, it will be really great if someone can help me understand the above doubt of mine.

Thanks a lot! :)

[rjgmail88]

Hi ,
'Server.js' and 'Schedule.js' are just custom js files I had created to separate the code doing independent work. In 'Server.js' I was doing initial setup of creating server , bot object and call Schedule.js to delegate the work.

[terabytes]

I didn't know how to save chat bot history in a clean way using the data bags provided out of the box by Microsoft Bot Framework, so I created some custom logging of my own. I hope you find this useful.

First off, Bot Framework has middleware functionality to intercept messages exchanged between the bot and the user. I customized this to save messages to MongoDB.

bot.use({
    receive: function (event, next) {
        // I don't really care about the bot initializing so I don't store that event info
        if(event.type != 'conversationUpdate') {
            logConvo("usr", event, event.user.id);
        }
        next();
    },
    send: function (event, next) {
        // I don't really care about the bot initializing so I don't store that event info
        if(event.type != 'conversationUpdate') {
            logConvo("bot", event, event.address.user.id);
        }
        next();
    }
});

I created a logConvo function that takes 3 arguments:

• identifier: Identifies whether the message is from the usr or the bot
• event: The event information, used to grab the message content
• id: The user's id (the user sending the message or the user the bot is sending a message to)

function logConvo(identifier, event, id) {
    var date = getDate();

    var msg = {
        "usr_id": id,
        "timestamp": date,
        "identifier": identifier,
        "content": event.text
    }

    if(event.text) {
       addMessageToDb(msg);
    }

    // To see messages between bot and usr on the console
    console.log(identifier + ": " + event.text);
    console.log();

};

You can see that if the message contains content, then I add the message to the database. In my case, I decided to use MongoDB.

I do this by calling the addMessageToDb function - which I wrote. See the code below:

function addMessageToDb(msg) {
    function handleConnection(err, client) {
        console.log("Handling connection");
        if(err) {
            console.log("Error connecting to mongodb");
            console.error(err);
        } else {
            console.log("Connected!");
            const DB_NAME = process.env.MONGO_DB_NAME;
            const db = client.db(DB_NAME);
            insertDocument(db, msg, function() {
                client.close();
            });
        }
    }

    var username = process.env.MONGO_USERNAME;
    var password = process.env.MONGO_PW;

    var connection_string = "mongodb://YOUR-HOST-DOMAIN:10255/?ssl=true";

    new mongoClient(connection_string, {
        auth: {
           user: username,
           password: password,
        }
    }).connect(handleConnection);
}

function insertDocument(db, msg, callback) {
    const collection = db.collection('YOUR_COLLECTION_NAME');
    collection.insertOne( {
            "usr_id": msg.usr_id,
            "timestamp": msg.timestamp,
            "identifier": msg.identifier,
            "content": msg.content
        }, function(err, result) {
        assert.equal(err, null);
        console.log("Inserted a document into the msgs collection.");
        callback(result);
    });
};

My MongoDB db is hosted on Azure's CosmosDB account so the connection string uses the syntax I provided above with the port I provided above. You can easily replace my connection string with your own. To understand the insertDocument function I wrote, see MongoDB's Quickstart guide for a similar function.

Let me know if this helps!

[rjgmail88]

I still dont have fixed solution for this as conversation data bag will be cleared. @EricDahlvang , let me know if you have any leads on this. Sorry to follow up on this thread so late but I hope there might be some way available now as this is 2 months over old thread.

[rjgmail88]

@terabytes , I am using CosmosDB (documentDb) in my case. I looked at you awesome implementation with MongoDb but looks like you are creating a document for every message. Do you have any example where I can do similar with CosmosDB ?

[msulaksh]

@terabytes Thanks.
I am working on ChatBot, where I have to save Conversation data in MongoDB, I haven't began storage part, but will soon.
Thanks & Regards,
Sulaksh More

[longsoft-dev]

I can understand by design, the (private)converstionData are cleared in Session.endConversation(). However, there should be a flag/option to let user choose whether to persist the cleared state or just leave as is. Currently there is no work-around, even
session.save(); bot.set(persistConversationData, false)
cannot stop the framework to persist the cleared state data, which is a bit stupid I believe. This will force user to implement their own state management instead of using the out of box one.

[EricDahlvang]

@beatgates .endConversation() means the conversation is over, and all state data related to that conversation should be cleared. ConversationData and PrivateConversationDate are not meant to be persisted forever, they are temporary data bags that exist for the life of the conversation. Use UserData and don't store things in conversationData or privateConvesationData that should not be cleared when the conversation is ended. and/or don't call .endConversation

Consumers of the Bot Builder are free to use whatever storage methods they choose.

I'm closing this, as it is not an Issue, Bug or Feature Request. The Microsoft Bot Framework team prefers that how to questions be submitted on Stack Overflow. The official Bot Framework Github repo  is the preferred platform for submitting bug fixes and feature requests.  

[ajithkumargangrapu]

where the user data is stored in louis and whether it is going to explore to public use.
whether the user data or conversation data is safe or not in louis Framework?

[EricDahlvang]

@ajithkumargangrapu This issue is closed. The team does not actively monitor closed issues. In the future, please ask questions on Stack Overflow. The Bot Framework Team prefers to use the GitHub repository for Bug reports and Feature Requests.

Details of what is 'stored' by LUIS can be found here: https://docs.microsoft.com/en-us/azure/cognitive-services/luis/luis-concept-data-storage

UserData and ConversationData are not related to LUIS, and can be stored whereever you like using one of the Azure Extensions: https://github.com/Microsoft/BotBuilder-Azure

[sagarmodhavaniya]

Hello,

I am using node js sdk to write ms bot service. Currently, I am saving everything in data bags (conversationData, userData etc) and using documentDb to store the state. I have also implemented middleware logging to intercept user to Bot and bot to user messages and logging them on console.
There are few concerns with this approach,

1. conversationData is cleared when we call endConversation() in dialog. Thats expected by design but we would like to persist this data for multiple conversation flows with same user (same conversation id.) Now, json document in db gets replaced with new keys on conversationData when user start new intent.
   Ex: schedule a meeting with {name} for {day} at {place}.
   we save conversationData.name , conversationData.day , and conversationData. place.

same user starts over schedule a meeting with {name2} for {day2} at {place2}.
documentDb entry gets replaced with conversationData.name1 , conversationData.day2 , and conversationData. place2

Ideally, we would like to keep everything.

1. Issue 2 is middleware logging is not able to hold session object so I can't save each event.text to any databag? Is this correct approach or there is better way to save chat history ?

const logUserConversation = (event) => {
console.log(' ** BOT/User Message: ' + event.text);
//Want to save event.text somewhere ??
};

//Middleware for logging
bot.use({
receive: function (event, next) {
logUserConversation(event);
next();
},
send: function (event, next) {
logUserConversation(event);
next();
}
});

any help would me much appreciated. Thanks in advance.

same problem in my case only send function are call received are call

[jwiley84]

This issue is closed. The team does not actively monitor closed issues. In the future, please ask questions on Stack Overflow. The Bot Framework Team prefers to use the GitHub repository for Bug reports and Feature Requests.