Azure-functions-durable-extension: Blob content instead of blob url in the output for the large output

[blackalbum]

Description of the issue
If the output is large enough (60KB) then the function returns in the output the url of the blob in which the actual response is stored. This creates an extra work for the end clients and extra responsibility for them to extract the blob if the expected output is of json.

Expected behavior
It's expected to set the behaviour in the application settings or a config class whether the durable function returns the actual blob url to the end client or the content of the blob by reading from the blob storage. It's clear that the large response cannot be stored in the queue so there was a workaround to store the large response in the blob storage. What is expected is if the output is stored in the blob storage, the durable function needs a setting so as to return the content of the blob rather than the url to reduce the work and avoid a prospect subsequent call from the end client to the blob storage.

Actual behavior
It returns the url of the blob if the response is large enough

Known workarounds
https://github.com/Azure/azure-functions-durable-extension/issues/26

[pmatthews05]

I am getting this issue.

One of my Azure Functions puts some JSON into a .json blob file, I was originally making sure that the file was no larger than 50kb. I have another Azure Function that is called from a SPFX webpart that fetches this Blob at a later stage and returns the file contents as string. Because of this large file (32kb) limit, the content is just being zipped up again as another blob file.

I would really like this to have a parameter to be able to allow my Orchestration to return either contents or link to blob.

[SimonLuckenuik]

This should be tagged as a bug and not an enhancement... this bug forces the caller to understand internal implementation of Durable Functions. Moreover, the way to detect that we are facing that situation is to parse a JToken and figure out it contains an URL, not friendly at all...

Copying my comment from https://github.com/Azure/azure-functions-durable-extension/issues/693#issuecomment-497549591 :

@cgillum, we just faced that specific bug today. The current behavior is not obvious and is not expected as well.

I think Durable Function should be responsible to deserialize and return the content of that file.

Currently, I will need to :
1) deserialize "output" as a JToken
2) check if the content of the JToken is a string starting with a URL, if so it means know I need to get the file
3) check if the extension ends with '.gz', if so, unzip it otherwise read it as is, no unzip.
4) Assuming that is a non-public blob container, so it means I will need to find which Storage account to use so that I can generate a SAS token in order to get the file or extract the relative path and use the Storage SDK to get the file directly...
5) Deserialize the file content and use it.

It also means that I need to make sure that my output is never a single string and rather a complex structure to make sur that by mistake the start of the string returned is a URL...

[grahamrb]

Has anyone coded up a workaround for this? I guess a function in front of the durable function that detects the blob URL and pulls the content back to the consumer?

[SimonLuckenuik]

@grahamrb, exactly. I never expose Durable Function HTTP APIs to my clients, simply because that would leak my internal implementation, I don’t want to be coupled with Durable Function to make sure I can remove it if I am not satisfied with the direction/responsiveness of the product or at least workaround issues like this one.

[cgillum]

Thanks for the feedback. I think it has been made clear that this behavior can be pretty painful to deal with. @ConnorMcMahon is currently looking at changing this behavior for an upcoming release.

[SimonLuckenuik]

Excellent! Thanks Chris!

[larryeby]

Any news on the ETA for this? Really hoping to use the newest package since there's quite a few improvements, but this is currently a big blocker for my team.

[cgillum]

This will be included in our 1.8.3 release, which should be ready sometime next week if all goes according to plan.

[CastleArg]

Will this fix handle the same problem for large inputs?
After learning about this bug I refactored everything to have smaller outputs,
but my code is still dying at a certain scale, presumably because of large object input to orchestrator function

[cgillum]

@CastleArg the problem you're encountering appears to be specific to the JavaScript library. I believe that will require a separate fix.

[grahamrb]

@cgillum - any updates on the 1.8.3 release? Is it still scheduled for this week? Do you have a date planned for this? Thanks

[cgillum]

There is a PR for fixing this issue, but it's still currently being worked on. We're still hoping to have a release out by the end of this week: https://github.com/Azure/durabletask/pull/303.

[grahamrb]

Is there an anticipated release date for this fix? We've just retested and it isn't working as yet within our subscription.

[cgillum]

Have your upgraded to the 1.8.3 nuget package version? The fix was included as part of that release.

https://github.com/Azure/azure-functions-durable-extension/releases/tag/v1.8.3

[wisesimpson]

I met the same issue today but I'm writing duration functions with JavaScript. Does v1.8.3 only solve the problem for C#?

[cgillum]

Hi @wisesimpson, v1.8.3 should solve this for both C# and JavaScript.

[wisesimpson]

Hi @wisesimpson, v1.8.3 should solve this for both C# and JavaScript.

@cgillum Thanks for your reply. I got a very silly question:
The v1.8.3 is a nuget package so the use case is only for Visual Studio on Windows?
I'm using Visual Studio Code on MacOS. By following the documentation I installed Azure Functions Core Tools (2.7.1724) which included Function Runtime Version: 2.0.12763.0.
Does Azure Function Core Tools contain Function Runtime and Function Runtime contains "azure function durable extension"?
I found this thread because I'm still getting a blob storage link when I query from Cosmos DB and the result is large.

[ConnorMcMahon]

@wisesimpson,

You can update the Durable Task extension using the Functions CLI tool with the following command

func extensions install -p Microsoft.Azure.WebJobs.Extensions.DurableTask -v 1.8.3

Note that you will need .NET Core installed on your dev device to run this command. This is tracked here.

[wisesimpson]

@ConnorMcMahon You just made my day! Thank you so much

[ltwlf]

func extensions install -p Microsoft.Azure.WebJobs.Extensions.DurableTask -v 1.8.3
Fixed is for me (JavaScript)!
Thanks!