Botframework-sdk: [Question] Parsing image attachment through skype

[shloak]

A question I had was about how to handle when a user sends images in Skype. In facebook and the emulator, the contentUrl of the image is an accessible URL, however, in Skype, I cannot access this URL. How else can my bot get the contents of the image?

[msft-shahins]

The Skype attachment URLs are secured by JwtToken , you should set the JwtToken of your bot as the authorization header for the GET request your bot initiates to fetch the image. Below is the sample code that temporarily works around this issue and set the JwtToken on the http request. You should be careful when you sent the Bot's JwtToken to a third party server and should always make sure to send it to trusted parties.

For C#:

public static async Task<IEnumerable<byte[]>> GetAttachmentsAsync(this Activity activity)
{
    var attachments = activity?.Attachments?
        .Where(attachment => attachment.ContentUrl != null)
        .Select(c => Tuple.Create(c.ContentType, c.ContentUrl));
    if (attachments != null && attachments.Any())
    {
        var contentBytes = new List<byte[]>();
        using (var connectorClient = new ConnectorClient(new Uri(activity.ServiceUrl)))
        {
            var token = await (connectorClient.Credentials as MicrosoftAppCredentials).GetTokenAsync();
            foreach (var content in attachments)
            {
                var uri = new Uri(content.Item2);
                using (var httpClient = new HttpClient())
                {
                    if (uri.Host.EndsWith("skype.com") && uri.Scheme == "https")
                    {
                        httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
                        httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/octet-stream"));
                    }
                    else
                    {
                        httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue(content.Item1));
                    }
                    contentBytes.Add(await httpClient.GetByteArrayAsync(uri));
                }
            }
        }
        return contentBytes;
    }
    return null; 
}

For node.js:

var async = require('async');
var url = require('url');
var request = require('request');

function downloadAttachments(connector, message, callback) {
    var attachments = [];
    var containsSkypeUrl = false;
    message.attachments.forEach(function (attachment) {
        if (attachment.contentUrl) {
            attachments.push({
                contentType: attachment.contentType,
                contentUrl: attachment.contentUrl
            });
            if (url.parse(attachment.contentUrl).hostname.substr(-"skype.com".length) == "skype.com") {
                containsSkypeUrl = true;
            }
        }
    });
    if (attachments.length > 0) {
        async.waterfall([
            function (cb) {
                if (containsSkypeUrl) {
                    connector.getAccessToken(cb);
                }
                else {
                    cb(null, null);
                }
            }
        ], function (err, token) {
            if (!err) {
                var buffers = [];
                async.forEachOf(attachments, function (item, idx, cb) {
                    var contentUrl = item.contentUrl;
                    var headers = {};
                    if (url.parse(contentUrl).hostname.substr(-"skype.com".length) == "skype.com") {
                        headers['Authorization'] = 'Bearer ' + token;
                        headers['Content-Type'] = 'application/octet-stream';
                    }
                    else {
                        headers['Content-Type'] = item.contentType;
                    }
                    request({
                        url: contentUrl,
                        headers: headers,
                        encoding: null
                    }, function (err, res, body) {
                        if (!err && res.statusCode == 200) {
                            buffers.push(body);
                        }
                        cb(err);
                    });
                }, function (err) {
                    if (callback)
                        callback(err, buffers);
                });
            }
            else {
                if (callback)
                    callback(err, null);
            }
        });
    }
    else {
        if (callback)
            callback(null, null);
    }
}

[letmaik]

@msft-shahins Is getAccessToken() part of the public API or an internal method? I can't find it here.

[HKOLWD]

Code sample worked! Only modified the code to return a string instead of an array, which was easier to use in my case. Finally managed to solve this challenge. Thanks @msft-shahins

[TechWatching]

Thanks for the sample. Is there something planned to integrate this with the dialogs ? From a dialog perspective, I don't have access to activity.

[iassal]

@msft-shahins, Will this ever be part of the BotBuilder? Or is this workaround permanent?

[msft-shahins]

@iassal for now this is a workaround for the skype behavior and no plan to add this to bot builder.

[etiago]

Thanks for the workaround @msft-shahins . It works. Although I do find this an ugly hack 😢 especially considering that intent handlers for LuisDialogs don't even get the serviceUrl passed through... this means that we need to build all kinds of workarounds to save the serviceUrl in order to be able to get the Skype token... Just my 2 cents... it would be nice to have this built into the bot framework itself.

[msft-shahins]

@etiago I believe a fix for this is on Skype's roadmap. But to answer the other concern, you do have access to the incoming activity in the luis intent handlers. You can use the IntentActivityHandler signature instead of IntentHandler and as a result you can access the incoming IMessageActivity: https://github.com/Microsoft/BotBuilder/blob/master/CSharp/Library/Dialogs/LuisDialog.cs#L96

Also a recent change that is still in develop branch made the incoming activity available in the context: https://github.com/Microsoft/BotBuilder/commit/02746b4765b3f8077bd215390b3752065ba2df2b

[vrnmthr]

I actually think there's a simpler way to do this that's baked right into the framework as well:

public async static Task<byte[]> LoadAttachmentAsBytes(Attachment img, string ServiceUrl)
        {
            using (var connectorClient = new ConnectorClient(new Uri(serviceUrl)))
            {
                var content = connectorClient.HttpClient.GetStreamAsync(img.ContentUrl).R‌​esult;
                var memoryStream = new MemoryStream();
                content.CopyTo(memoryStream);
                return memoryStream.ToArray();
            }
        }

The above seems to work perfectly well on Skype and Messenger, and you don't have to deal with the MicrosoftAppCredential stuff :)
The other solution provided above kept giving me 401 unauthorized request errors for some reason...

[crowster]

This work perfect, i only avoid the condition contains, beacuse my url is https://smba.trafficmanager.net/apis/v3/attachments/0-eus-d4-0a1ebe7353c2e0677495f8f127cb7f8c/views/original

my code is:

private async Task<IEnumerable<byte[]>> GetAttachmentsAsync(Activity activity)
    {
       var attachments = activity?.Attachments?
      .Where(attachment => attachment.ContentUrl != null)
      .Select(c => Tuple.Create(c.ContentType, c.ContentUrl));
        if (attachments != null && attachments.Any())
        {
            var contentBytes = new List<byte[]>();
            using (var connectorClient = new ConnectorClient(new Uri(activity.ServiceUrl)))
            {
                var token = await(connectorClient.Credentials as MicrosoftAppCredentials).GetTokenAsync();
                foreach (var content in attachments)
                {
                    var uri = new Uri(content.Item2);
                    using (var httpClient = new HttpClient())
                    {

                            httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
                            httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/octet-stream"));

                        contentBytes.Add(await httpClient.GetByteArrayAsync(uri));
                    }
                }
            }
            return contentBytes;
        }
        return null;
    }

[dchathu]

@vrnmthr's solution works fine on Skype.

[xjose97x]

@TechWatching you do have access to the activities from the dialogs.

The context object has access to the activity.

[ajayshiva]

This doesn't work in v4 since connectorClient.Credentials has null value for AppId & AppPassword

Can someone post an updated code?

[EricDahlvang]

@ajayshiva MicrosoftAppCredentials can be provided as a parameter to ConnectorClient:

var credentials = new MicrosoftAppCredentials("Your_App_ID", "Your_App_Password");
using (var connectorClient = new ConnectorClient(new Uri(activity.ServiceUrl), credentials))
{
    var token = await credentials.GetTokenAsync();
    foreach (var content in attachments)
    {
        var uri = new Uri(content.Item2);
        using (var httpClient = new HttpClient())
        {

            httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
            httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/octet-stream"));

            contentBytes.Add(await httpClient.GetByteArrayAsync(uri));
        }
    }
}

[ajayshiva]

Thanks this works! Is it possible to retrieve the AppID & Password from WaterfallStepContext or from .bot config?

Below code allows me to retrieve the appid & password but it throws error in bot after this step while downloading the image.

var context = stepContext.Context;
using (var connectorClient = (ConnectorClient)context.TurnState.First(x => x.Key.Equals("Microsoft.Bot.Connector.IConnectorClient")).Value)
                    {
                        if (connectorClient?.Credentials != null)
                        {
                            var credentials = (MicrosoftAppCredentials)connectorClient.Credentials;                            
                        }
                    }

[evgeniy-glushin]

Thanks this works! Is it possible to retrieve the AppID & Password from WaterfallStepContext or from .bot config?

Below code allows me to retrieve the appid & password but it throws error in bot after this step while downloading the image.

var context = stepContext.Context;
using (var connectorClient = (ConnectorClient)context.TurnState.First(x => x.Key.Equals("Microsoft.Bot.Connector.IConnectorClient")).Value)
                    {
                        if (connectorClient?.Credentials != null)
                        {
                            var credentials = (MicrosoftAppCredentials)connectorClient.Credentials;                            
                        }
                    }

Hi. Could you please let me know which exact version of v4 you used? I've got 4.5.2 and it's not working. I'm getting 500 without any error message.

[EricDahlvang]

Hi @evgeniy-glushin

This doc should help with debugging the 500 error:

https://docs.microsoft.com/en-us/azure/bot-service/bot-service-troubleshoot-500-errors

Where are you running this code?

[evgeniy-glushin]

Hi @EricDahlvang

I have a skype bot with beck-end running in Azure.

I've got following code

```C#
public async Task {
var credentials = new MicrosoftAppCredentials("appId", "password");
using (var connectorClient = new ConnectorClient(new Uri(serviceUrl), credentials))
{
var token = await credentials.GetTokenAsync();

    var uri = new Uri(att.ContentUrl);
    using (var httpClient = new HttpClient())
    {
        httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
        httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/octet-stream"));

        return await httpClient.GetByteArrayAsync(uri);
    }
}

}
```

When a user sends a video message to the bot you get ContentUrl (https://smba.trafficmanager.net/apis/v3/attachments/0-weu-d11-a7dd2f30f52f28e7c657bbffc81be9e5/views/original) and you are supposed to be able to download the file using this url. But when I try to do it I get 500 without any error message. The interesting part is when I send a video file as an attachment (not a video message) the same code works just fine. It seems like the issue is in this api https://smba.trafficmanager.net/apis/v3/attachments. Here is related question https://social.msdn.microsoft.com/Forums/officeocs/en-US/cf97950f-7af6-4ed4-a053-d3453198c090/500-server-error-while-downloadning-file-from-skype?forum=AzureCognitiveService

[EricDahlvang]

I tried the code above in a Skype connected bot (after replacing the appid and password). I was able to download the bytes of a video recorded and submitted through the skype windows client.

Have you tried debugging locally? https://blog.botframework.com/2017/10/19/debug-channel-locally-using-ngrok/

[evgeniy-glushin]

@EricDahlvang Hi. Thank you so much for looking into this problem. Yes, I've tried debugging it locally. Did you send a video message or a video file as a file attachment? It fails specifically when I try to send a skype video message but it works fine when I send a video file (.mp4) as an attachment.

[EricDahlvang]

I was able to receive both a video message and a video file. The above screenshots are from a video message.

[evgeniy-glushin]

@EricDahlvang That's what I get

I'm using 4.5.2 version of the Bot Framework and the latest version of skype. Which Bot Framework are you using? What do you think could be the reason?

[EricDahlvang]

I'm using the latest sdk. Have you tried creating a new bot registration from scratch?

I see your issue is also being addressed here: https://github.com/microsoft/botbuilder-dotnet/issues/2583 I will follow along there.