Botframework-sdk: Handling incoming location

[mikkhait]

We need a way of consistently handling incoming location. I've tested it across mobile apps for _Messenger_ and _Telegram_, I'm sure others will need this too

Given _mobile Messenger app_
When I send my location attachment
Then I get
attachments: [ { contentType: 'location' } ]

Given _mobile Telegram app_
When I send my location attachment
Then I get

attachments: [],
location: 
   { altitude: 0,
     latitude: 32.403613,
     longitude: -131.94175,
     name: '' }

Expected (suggestion)

attachments: [
  { contentType: 'location',
    payload: 
    { altitude: 0,
      latitude: 32.403613,
       longitude: -131.94175,
       name: ''
    }
  }
]

[Stevenic]

Adding @tomlm... I don't think we've tested this across clients super well but it's good feedback. @mikkhait could you share the full attachment you're getting from messenger?

I'm thinking we should probably also add a Prompts.location() prompt that lets you ask a user for their location and waits for it.

[mikkhait]

@Stevenic this is all I get from _Messenger_

created: '2016-05-22T19:25:46.7825221Z',
  text: '',
  attachments: [ { contentType: 'location' } ],
  from:...

[rashoodkhan]

I'm not able to receive any location data from Messenger. The Payload in ChannelData.message is empty and so is attachments. It would be great to have a fix for this.

[tomlm]

the documentation says nothing about an attachment type of location. The only parameter they document is a url parameter which is why you aren't getting location information. They have NO documentation which describes this data.

[Stevenic]

@tomlm is fixing a bug that prevented the url they were passing from being promoted to the attachment so you should see that start coming through shortly. Here's examples of the links:

• ios https://www.facebook.com/l.php?u=https%3A%2F%2Fwww.bing.com%2Fmaps%2Fdefault.aspx%3Fv%3D2%26pc%3DFACEBK%26mid%3D8100%26where1%3D14865%2BNE%2B36th%2BSt%252C%2BRedmond%252C%2BWashington%2B98052%26FORM%3DFBKPL1%26mkt%3Den-US&h=MAQHZ9EyM&s=1&enc=AZOEUjUQUoPW4j0gSoxMEKiKRR8KgcF_lbpvo0LWMLdVGu_B6x8Y77g0mOHNAJ6nIYFKWdxMYsIU_Wcd4JPGU0wtpXtC8V6KpE38WtVcj2sk-g
• android https://www.facebook.com/l.php?u=https%3A%2F%2Fwww.bing.com%2Fmaps%2Fdefault.aspx%3Fv%3D2%26pc%3DFACEBK%26mid%3D8100%26where1%3D47.640465%252C%2B-122.141238%26FORM%3DFBKPL1%26mkt%3Den-US&h=LAQHcE1Ov&s=1&enc=AZN13k_I_4MW1EccKiEaQiRpAA0SW1V4O-FFR9v13McSdW1jfTOnaDM5k-DwRrAAFK-rc9KayLmyAyHaRnecGUdNChdFNAVsfjDhtEHpnOKzuQ

We looked at normalizing this but they don't always send lat+long so it's not easy to normalize :(

• address https://www.bing.com/maps/default.aspx?v=2&pc=FACEBK&mid=8100&where1=14865+NE+36th+St%2C+Redmond%2C+Washington+98052&FORM=FBKPL1&mkt=en-US
• lat+long https://www.bing.com/maps/default.aspx?v=2&pc=FACEBK&mid=8100&where1=47.640465%2C+-122.141238&FORM=FBKPL1&mkt=en-US

[rashoodkhan]

@tomlm, @mikkhait - The documentation does not mention any location attachment, but when I tried out with raw Messenger API's, it returns the latitude and longitude in the payload field as shown below (Shared location from client, and this is the message sent from Facebook):

{
  "object": "page",
  "entry": [
    {
      "id": "XXXXXXXXXXXXX",
      "time": 1464759819251,
      "messaging": [
        {
          "sender": {
            "id": "XXXXXXXXXXXXX"
          },
          "recipient": {
            "id": "XXXXXXXXXXXXXX"
          },
          "timestamp": 1464759819084,
          "message": {
            "mid": "mid.1464759818851:db6a185bfdd1a2fa18",
            "seq": 219,
            "attachments": [
              {
                "title": "Anik's Location",
                "url": "https://www.facebook.com/l.php?u=https%3A%2F%2Fwww.bing.com%2Fmaps%2Fdefault.aspx%3Fv%3D2%26pc%3DFACEBK%26mid%3D8100%26where1%3D12.967736461345%252C%2B77.59405228556%26FORM%3DFBKPL1%26mkt%3Den-US&h=mAQHFlfya&s=1&enc=AZOk_o46RONuMoyJFQQwkgsni5vq9wncDL7A3zbUyBvwrfTZp-bh0bgePvxnzE5er_dPRC-2FQ9Djbz8kbA44boGZlOxeRcXH_1lr4Diz3V9fw",
                "type": "location",
                "payload": {
                  "coordinates": {
                    "lat": 12.967736461345,
                    "long": 77.59405228556
                  }
                }
              }
            ]
          }
        }
      ]
    }
  ]
}

Another example when the location cannot be parsed from the URL, but still returns the latitude and longitude:

{
  "object": "page",
  "entry": [
    {
      "id": "XXXXXXXXXXXXX",
      "time": 1464759898589,
      "messaging": [
        {
          "sender": {
            "id": "XXXXXXXXXXXXXXX"
          },
          "recipient": {
            "id": "XXXXXXXXXXXXXXXXX"
          },
          "timestamp": 1464759898493,
          "message": {
            "mid": "mid.1464759898260:0d0982c55dc8e30194",
            "seq": 220,
            "attachments": [
              {
                "title": "Redmond High School (Washington)",
                "url": "https://www.facebook.com/l.php?u=https%3A%2F%2Fwww.bing.com%2Fmaps%2Fdefault.aspx%3Fv%3D2%26pc%3DFACEBK%26mid%3D8100%26where1%3D17272%2BNE%2B104th%2BSt%252C%2BRedmond%252C%2BWashington%2B98052%26FORM%3DFBKPL1%26mkt%3Den-US&h=RAQFQuxCs&s=1&enc=AZNOwQKUJDqdzz1AxbdSXTLzjhHTi7ABU94GzlVkBP7WVGqcbRMOqQnpum-P9GbR8oCgkKk2abWZJgK5a0NyYtlInD8iLuNeLMXlWhJ0euoejw",
                "type": "location",
                "payload": {
                  "coordinates": {
                    "lat": 47.695277777778,
                    "long": -122.10666666667
                  }
                }
              }
            ]
          }
        }
      ]
    }
  ]
}

Currently for our bot (http://m.me/YellowMssngr), we are using Regex to get latitude and longitude from the URL, but as mentioned in previous comment for some location the lat and long is not sent and for those cases our bot fails. Some locations where we don't receive the Lat & Long in URL - Redmond High School, Brooklyn Bridge. Having this fix is really crucial for our bot to work, I would love to help out where necessary to fix this issue ASAP.

[Stevenic]

I get an empty payload so not sure what's going on.

[rashoodkhan]

Empty payload on raw messenger API or with Botframework? I get empty payload when fetching location from messenger when using Botframework. Can we have a chat about this someplace?

[Stevenic]

@rashoodkhan are you using the messaging webhook directly or something else? Tom is passing through the raw message he's getting from facebook, you can see it in channelData. The attachment is there but without any payload. Are you saying that when you intercept the webhook you get a payload with data?

[rashoodkhan]

@Stevenic - For the above snippet which I have posted I used the messaging web-hook directly and not from https://www.botframework.com.

For the bots which are running on production the messages go through https://botframework.com and do not receive the location in payload. And to answer your question, yes I don't get the payload data when intercepted through the channelData, though ideally it should be there in the payload object in channelData.

[tomlm]

Super helpful, I think I have fixed this.

[rashoodkhan]

Awesome @tomlm, will test this and let you know.

[rashoodkhan]

@tomlm - Just tried it out, still not receiving the location. I don't know if you have pushed the code to prod? Please let me know once it is deployed, I already have made code changes from my end, I will push the new code to prod. Thanks 😄

[tomlm]

I haven’t pushed to prod yet, probably an hour or two

[Stevenic]

This looks to be all fixed now. To go along with the fix Tom made I updated my demo-facebook example to show creating a custom /locationPrompt dialog which will prompt the user to share their current location.

[rashoodkhan]

Thanks a lot @tomlm

[jw84]

@Stevenic Hi Steven! Trying to get a location attachment from Facebook Messenger users and following through the demo-facebook example I'm not seeing the /locationPrompt snippet. Is the code living in a dev branch somewhere?

[Stevenic]

I'm not sure what happened??? It seems to have gotten dropped. I'll try to re-add it but might be a day or so before I get to it.

[Stevenic]

Ok... I see what happened. I'll have to add it back in. I need to do a big update on the facebook bot and will add it back in. For now check out this commit

[scottmcconnell]

Hi @Stevenic, is there a way that I can ask for the location via Facebook quick reply in a waterfall style dialog using a builder prompt?
I have a stackoverflow question here if you need some more information.

Thanks in advance.

Edit; I may have mis-understood how the commit you posted above actually works. Apologies if I'm asking what you've already answered.

Edit 2; I'd also be interested in doing the same in a waterfall dialog but with other facebook messenger send actions such as a list template.

[scottmcconnell]

To answer my question above I have found a solution. It seems it's possible by sending the Facebook sourceevent message with the quick reply and also using the builder.Prompts.text() to continue on the diaog flow.

`bot.dialog('/branchLocator', [
function(session) {

    var msg = new builder.Message(session).sourceEvent({
    facebook: {
        text: "Please send me your postcode or location and I\'ll grab the closest branches to you.",
        quick_replies: [
                {
                    "content_type": "location"
                }
            ]
        }
    });

   builder.Prompts.text(session, 'I can help you with that.');
   session.send(msg);

    // I can find the closest branch to your location, click below to send me your location.
}, function(session, results) {
    console.log('location results: ', results);
    if (session.message.entities.length > 0 && typeof session.message.entities[0].geo !== 'undefined') {
        // Location being sent
        actions.branchLocator(session, null, session.message.entities[0].geo);
    } else if (typeof results.response !== 'undefined') {
        var postcode = results.response.match(/.*(\d{4}).*/);
        // postcode[1] -- This grabs what it matched (i.e. the postcode)
        if (postcode === null) {}
        // TODO; What happens if this is null? How can we handle this
        actions.branchLocator(session, postcode[1], null);
    }
}

]); `

I have also tested this with the list template and all seems to be working quite well.

Thanks,
Scott

N.B. For some reason github doesn't want to format my whole block of code. Sorry!