Botframework-sdk: Determine events that an application needs to handle

Created on 12 Feb 2019 · 11Comments · Source: microsoft/botframework-sdk

Categorized Activities by Channel

The following tables show what events (Activities on the wire) can come from what Channels.

This is the key for the tables:

Symbol | Meaning
:------------------:|:------------------------------------------------
:white_check_mark: |The Bot should expect to receive this Activity
:x: |The Bot should never expect to receive this Activity
:white_large_square:|Currenly undetermined whether the Bot can receive this

Activities can meaningfully be split into separate categories. For each category we have a table of possible Activities.

Conversational

\ | Cortana | Direct Line | Web Chat | Email | Facebook | GroupMe | Kik | Teams | Slack | Skype | Skype Business | Telegram | Twilio
:---------------------- | :-----: | :----------------: | :--------------------: |:----: | :------: | :-----: | :-----: | :---: | :---: | :---: | :------------: | :------: | :----:
Message | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark:
MessageReaction | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: | :x: | :x:

All Channels send Message Activities.
When using a Dialog, Message Activities should generally always be passed onto the Dialog.
This is probably not true of the MessageReaction although they are very much part of the conversation.
There are logically two types of MessageReaction: Added and Removed

("Message Reactions" are things like a "thumbs up" on a previous comment. They can happen out of order so can be thought of as similar to buttons. This Activity is currently sent by the Teams Channel.)

Welcome

It is common for Channels to send ConversationUpdate Activities.
There are logically two types of MessageReaction: Added and Removed
It is very tempting to assume bot "Welcome" behavior can be simply implemented by wiring up ConversationUpdate.Added and this sometimes works.
However, this is a simplification, in order to produce a reliable "Welcome" behavior the bot implementation may also need to use state.

Application Extensibility

Event Activities are an extensibility mechanism in Direct Line (_aka Direct Line_).
An application that owns both the client and server may chose to tunnel their own events through the service using this Event Activity.

Microsoft Teams

Along with a number of the other typed Activities, Microsoft Teams defines a few Teams specific Invoke Activities.
Invoke Activities are specific to an application and not something a client would define.
There is no general notion of Invoke only specific subtypes of the Activity.
Invoke is currently the only Activity that triggers a request-reply behavior on the bot.

This is very important: if using Dialogs for the OAuth Prompt to work the Invoke.TeamsVerification Activity must be forwarded to the Dialog.

Message Update

Message Update is currently supported by Teams.

OAuth

This is very important: if using Dialogs for the OAuth Prompt to work the Event.TokenResponse Activity must be forwarded to the Dialog.

Uncategorized

Out of Use (includes Payment specific Invoke)

DeleteUserData
Invoke.PaymentRequest
Invoke.Address
Ping

Summary of Activities supported per Channel

Cortana

Message
ConversationUpdate
_Event.TokenResponse_
_EndOfConversation (when the window closes?)_

Direct Line

Message
ConversationUpdate
Event.TokenResponse
Event.*
_Event.CreateConversation_
_Event.ContinueConversation_

Email

Message

Facebook

Message
_Event.TokenResponse_

GroupMe

Message
ConversationUpdate
_Event.TokenResponse_

Kik

Message
ConversationUpdate
_Event.TokenResponse_

Teams

Message
ConversationUpdate
MessageReaction
MessageUpdate
MessageDelete
Invoke.TeamsVerification
Invoke.ComposeResponse

Slack

Message
ConversationUpdate
_Event.TokenResponse_

Skype

Message
ContactRelationUpdate
_Event.TokenResponse_

Skype Business

Message
ContactRelationUpdate
_Event.TokenResponse_

Message
ConversationUpdate
_Event.TokenResponse_

Twilio

Message

Summary Table All Activities to All Channels

\ | Cortana | Direct Line | Web Chat | Email | Facebook | GroupMe | Kik | Teams | Slack | Skype | Skype Business | Telegram | Twilio
:---------------------- | :-----: | :---------: | :--------------------: |:----: | :------: | :-----: | :-----: | :---: | :---: | :---: | :------------: | :------: | :----:
Message | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark:
MessageReaction | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: | :x: | :x:
ConversationUpdate | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :white_large_square: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :x: | :white_check_mark: | :x:
ContactRelationUpdate | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :white_check_mark: | :white_check_mark: | :x: | :x:
Event.* | :white_large_square: | :white_check_mark: | :white_check_mark: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square:
Event.CreateConversation | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square:
Event.ContinueConversation | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square:
Invoke.TeamsVerification | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: | :x: | :x:
Invoke.ComposeResponse | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: | :x: | :x:
MessageUpdate | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :white_check_mark: | :white_large_square: | :x: | :x: | :x: | :x:
MessageDelete | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :white_check_mark: | :white_large_square: | :x: | :x: | :x: | :x:
Event.TokenResponse | :white_large_square: | :white_check_mark: | :white_check_mark: | :x: | :white_large_square: | :white_large_square: | :white_large_square: | :x: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square: | :white_large_square:
EndOfConversation | :x: | :white_check_mark: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x:
InstallationUpdate | :x: | :white_check_mark: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x:
Typing* | :x: | :white_check_mark: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x:
Handoff | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x:

Please note that WebSockets vs REST is another layer of complexity not yet accounted for in these charts. Typing activities only work on WebSockets and do not work on REST.

[edited by @corinagum]

P2 R10

Source

johnataylor

👍8 ❤7 🚀3 🎉3 😄2 👀1 😕1

Most helpful comment

From the cortana side...

Hi; regarding how close conversation is used in Cortana

Desktop

User talks to “client”. Client calls speech to text. Client sends text to agent.
Agent figures out how to route the text. For example, open a third party skill.
Agent lets client know a skill is opening. Client opens a “canvas” in which to have a conversation. We have a session. A “Microsoft.Launch” intent is added and the invocation text is sent to the bot.
Client, agent, and bot now talk with request/response. Responses can include speak in which case text to speech is called by client.
a. When the bot determines its purpose is fulfilled, it asks session to end the conversation. This should end the session and close the canvas. b. when the client closes, right now nothing happens on the bot, but client and agent tear down the conversation. (log fill with errors if any response is sent from the bot. ideally the canvas close should cause the event to flow through agent to the bot to do whatever clean up it needs to do.)

Mobile
Same but different.

There is no canvas to open. However the app now has a context and messages flow from app to agent to bot.
Bot lets agent and client know that conversation/session is done.

So….
App< Hi I am Cortana. What can I do for you?
User> Open SlowChess
App< I’m on it!
[Launch.Intent]
SlowChess User>move Q C6 to C7
SlowChess User>leave
SlowChess [endConversation]
App< Cortana here. What can I do for you?

Hopefully one day we supported nested (stacked conversations with warm hand offs. In that case, endConversation pops the stack.

I have other use cases I’d love to see 😊

For example
As a skill developer, I can set the duration up to a reasonable amount for the user to respond such that I can change the course of a conversation or provide feedback to the user.
As a skill developer, the skill is notified upon the end of rendering such that I can I can reprompt the user or change the flow of a conversation. (For example, if I send an audio card, I am sent an event when it finishes playing.)(Parity with Google Actions.)

Micro-Muncher on 22 Mar 2019

🎉1 👍1

All 11 comments

I think Skype sends ContactRelationUpdate

Stevenic on 13 Feb 2019

👍1

• Bing is no longer a channel
• Kik is a channel
• Slack is a channel

Skype does NOT send ConversationUpdate (I don’t think Facebook does either).

EricDahlvang on 13 Feb 2019

We have updated both C# and JavaScript in master. I'm not closing this, just relabeling, because we would like to keep this open as a doc issue and something to base samples on.

johnataylor on 14 Feb 2019

@EricDahlvang can you take another look.

Anything I wasn't sure about I made _italics_.

johnataylor on 14 Feb 2019

@johnataylor this is one my favorite documents.

EricDahlvang on 15 Feb 2019

😄1

Web Chat will send:
• "message": with "text" and/or "attachments"
• "event": with "name" and "value" (as JSON/string)
• "typing": if the user set an option, namely "sendTypingIndicator"
Web Chat will not send "contactRelationUpdate". And Web Chat do not support "messageReaction", no one explicitly ask us to support this feature.

By default, Web Chat will render:
• "message": will render as either carousel or stacked, depends on the option in the activity
• "typing": will render for 5s and hide it, or until next activity come in
• "conversationUpdate": will hide
• "event": will hide
• Others: will show a warning box (we never see it in production)
Web devs can modify this render pipeline to add/remove/replace any custom render.

Devs can use Web Chat to send any activity type and payload, we don't doc this feature, we don't recommend this feature. We always recommend custom things goes to "event" activity.

johnataylor on 19 Feb 2019

Does this need to be updated with LINE events?

corinagum on 14 Mar 2019

👍1

I'm not sure if the :white_check_mark: is the correct here - the end of conversation docs say that Direct Line passes the activity forward to the client but does not do anything with this information. For Direct Line and Web Chat, should this be ⬜️ instead?

Also please note that WebSocket vs REST is a whole other layer of complexity to this conversation that is not accounted for above. For example, Typing only works on WebSocket and not REST, so Direct Line expecting to receive Typing is only true when using WebSockets.

corinagum on 18 Mar 2019

From the cortana side...

Hi; regarding how close conversation is used in Cortana

Desktop

User talks to “client”. Client calls speech to text. Client sends text to agent.
Agent figures out how to route the text. For example, open a third party skill.
Agent lets client know a skill is opening. Client opens a “canvas” in which to have a conversation. We have a session. A “Microsoft.Launch” intent is added and the invocation text is sent to the bot.
Client, agent, and bot now talk with request/response. Responses can include speak in which case text to speech is called by client.
a. When the bot determines its purpose is fulfilled, it asks session to end the conversation. This should end the session and close the canvas. b. when the client closes, right now nothing happens on the bot, but client and agent tear down the conversation. (log fill with errors if any response is sent from the bot. ideally the canvas close should cause the event to flow through agent to the bot to do whatever clean up it needs to do.)

Mobile
Same but different.

There is no canvas to open. However the app now has a context and messages flow from app to agent to bot.
Bot lets agent and client know that conversation/session is done.

Hopefully one day we supported nested (stacked conversations with warm hand offs. In that case, endConversation pops the stack.

I have other use cases I’d love to see 😊

Micro-Muncher on 22 Mar 2019

🎉1 👍1

Cortana ignores tokenResponse because it handles its own oauth.
I [tried to] put a diagram together here.
https://github.com/bw-kforce-ms/CortanaSkillsWIP/blob/master/Support/OAuth.md#supporting-multiple-channel-oauth

Micro-Muncher on 22 Mar 2019

Does cortana get conversation updates? We implement welcome by watching for a Microsoft.Launch intent.

Micro-Muncher on 22 Mar 2019

Was this page helpful?

0 / 5 - 0 ratings