Azure-functions-host: Getting only 404 after runtime update to 2.0.11776-alpha

[forki]

The beta runtime was updated to 11776 and now I only get 404 if I try to run my functions in the portal. All functions used to work and now the are broken.

Investigative information

Please provide the following:

• Function App version (1.0 or 2.0-beta): beta
• Function App name: ecarbeta
• Function name: e.g. CheckTransactions
• Region: Europe West

Repro steps

go to portal an run function CheckTransactions
--> 404

Expected behavior

functions executes and returns without error

Actual behavior

getting 404 and function is not executed at all

Known workarounds

None. It's completely broken for me

/cc @fabiocav

[forki]

Created support request: 118052218231390

[forki]

I deleted that functionapp and created a new one called "ecarjobs". but same thing. only 404

[forki]

From logs:
Warning: Only got partial types from assembly: CheckTransactions, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null
Exception message: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
at System.Reflection.RuntimeModule.GetTypes()
at System.Reflection.Assembly.GetTypes()
at Microsoft.Azure.WebJobs.Host.Indexers.DefaultTypeLocator.FindTypes(Assembly assembly, IEnumerable`1 extensionAssemblies) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Indexers\DefaultTypeLocator.cs:line 120

[cyky]

Version 11776 seems to have some breaking changes. See details: Azure Functions Runtime 2.0 Preview Breaking Change Notice

You can rollback to earlier version and that might fix the problem for now.

[forki]

@cyky tbh I actually want to have the new version resolution changes. I'd appreciate someone from the team or support takes a deeper look. At least the error reporting needs to be improve here.

[cyky]

@forki I understand. Just wanted to let you know that there is "workaround" as I stumbled here due to a similar problem.

[pragnagopa]

Added this to breaking changes investigations issue.

[fabiocav]

@forki we'll be looking at the details to see what is going on with your app. Support might be unable to help since this is with the preview bits.

[forki]

Thanks @fabiocav.

[paulbatum]

@fabiocav I think the underlying assembly load failure is being hidden because that exception handler does not inspect the LoaderExceptions

Its also interesting that this code path has now become a hotspot for assembly load failures. Do you know why this is the case? All sorts of assembly load failures were possible before the loadcontext changes but they never manifested in this code path as best as I can recall...

[fabiocav]

@paulbatum yes, that code has to be enhanced to properly report the error.

We've seen failures with that code path before the changes (we looked at a few last week), but it all depends on when those assemblies are requested. In the functions scenarios, we'd typically see that early as we probe for extensions.

[paulbatum]

Last week doesn't count - LoadContext changes were already in at that point! I'm talking about the many months previous.

[fabiocav]

Those issues weren't new, though. I pretty sure some of them were quite old, and that was my point; we've seen this before the changes were in place.

[fabiocav]

There's definitely an unrelated issue here being uncovered, as the DefaultTypeLocator shouldn't be in play at all. Looking at that now

[fabiocav]

Indeed, looks like an issue with the implementation of the default ExtensionTypeLocator. Not directly related to the recent assembly load changes, but the issue there is triggered by some of the behavior changes.

I'm looking at a feasible workaround, but will aim to have this fixed tomorrow to release with the next drop (which we're planning for this week).

[forki]

I can confirm that downgrade to 2.0.11651-alpha solves my issues. @fabiocav can you please notify me when changes are ready and I can switch backt to beta

[fabiocav]

Absolutely! I'll update this issue when this is resolved.

[JeryLu2020]

@fabiocav any update on this issue?

[forki]

Ping

[fabiocav]

The changes have been made. The deployment is being prepared and slated to go out this week. Waiting for some additional changes going in with this payload.

[forki]

does it mean my function app will work again or doees it mean it will show better error and I need to do something?

[fabiocav]

For pure WebJobs SDK scenarios, there will be a better, detailed and actionable message.

In functions, this code is no longer in play, so your Function App should just start working with those bits.

[JeryLu2020]

@fabiocav can you kindly share the release date to us? thanks a lot

[davidebbo]

@13127755587 The deployment of the new build (2.0.11857-alpha) is now complete. Can you verify that it works for you?

[forki]

Just restarted the function app and it shows the new version. But still
getting 404.

David Ebbo [e-mail] schrieb am Di., 12. Juni 2018, 17:10:

@13127755587 https://github.com/13127755587 The deployment of the new
build (2.0.11857-alpha) is now complete. Can you verify that it works for
you?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Azure/azure-functions-host/issues/2881#issuecomment-396624944,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADgNIW2e1JmW8hOxUwYa2Ryx6YJQhw_ks5t79nOgaJpZM4UIPi-
.

[forki]

Let's consider the following function app:

Subscription ID: 3363a07f-9c2b-4ed0-9624-d692d2038939
FunctionAppName: chargingpointimport
Function: BNetzAImport
Europe West

Could not load file or assembly 'TaskBuilder.fs, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.

But I think the file is there - it worked with 2.0.11651-alpha

[davidebbo]

@forki I checked and you're definitely running the new build. I'll let @fabiocav comment further on why it may still not be working.

[forki]

Thank you!

David Ebbo [e-mail] schrieb am Di., 12. Juni 2018, 18:14:

@forki https://github.com/forki I checked and you're definitely running
the new build. I'll let @fabiocav https://github.com/fabiocav comment
further on why it may still not be working.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Azure/azure-functions-host/issues/2881#issuecomment-396647751,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADgNBFxV7Tq1Oue8GvryzsCls_qp5zSks5t7-jZgaJpZM4UIPi-
.

[fabiocav]

Will need to take a look at your function app to see what is going on there. Does this happen locally as well?

[forki]

Dunno. I can't test locally right now. Tbh we only test with the online
version for various reasons.

Fabio Cavalcante [e-mail] schrieb am Di., 12. Juni 2018,
19:31:

Will need to take a look at your function app to see what is going on
there. Does this happen locally as well?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Azure/azure-functions-host/issues/2881#issuecomment-396671108,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADgNMQOf2iwObCmW-aTyomXslWeaegvks5t7_rXgaJpZM4UIPi-
.

[fabiocav]

Sounds good. Will take a look and update with what I find.

[forki]

Any news? I'm getting really worried since this is second release after the one that works and I assume you guys will deprecate the working one soon.

[fabiocav]

We'll keep that around until we get to the bottom of this (/cc @davidebbo)

Haven't been able to investigate this yet, blocking the time to do this today. To speed up the cycle, I'll also see if I can make updated releases available out of band so we can validate that we won't hit any other issues as we move forward.

[forki]

Very cool. I'm currently at vacation, but I'm definitely able to test other
runtime versions if needed. Thank you

Fabio Cavalcante [e-mail] schrieb am Mi., 13. Juni 2018,
19:46:

We'll keep that around until we get to the bottom of this (/cc @davidebbo
https://github.com/davidebbo)

Haven't been able to investigate this yet, blocking the time to do this
today. To speed up the cycle, I'll also see if I can make updated releases
available out of band so we can validate that we won't hit any other issues
as we move forward.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Azure/azure-functions-host/issues/2881#issuecomment-397025771,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADgNCLwOxsKzbq1v_YgOCQSTswV2Nywks5t8U_4gaJpZM4UIPi-
.

[forki]

ping

[tforkmann]

I have the same issue.
I'm not able to successfully deploy and then run a azure function.

[tforkmann]

I downgraded to 2.0.11651-alpha and FSharp.Core 4.2.3.
Now the Azure Functions are successfully executing again.

[fabiocav]

Making sure I have a good repro and running through some validation. Will continue to update this.

[forki]

Btw you can run my function as often as you want. It's a good repro and currently just fails. If it succeeds then nothing bad happens. So don't worry if you want to use it

[forki]

Any news?

[fabiocav]

Yes. Looked at your current setup, and your functions are using the old style pre-compiled model (non-attributed model), and that model seems to have a bug that fails to load your assembly locations into the probing paths.

The fix is pretty simple and I'll have that in place soon, but the good news is that you can make a small change to your deployment/directory layout to workaround the issue in the meantime.

If you move all your binaries into a bin folder (this is the default/fallback probing location) and just leave the function directories with a function.json file in them (updated to point to ../bin/<assembly_file> for the scriptFile reference), that should work as expected. One of the benefits of this is that you reduce the number of duplicate files.

[forki]

Ok. Thanks. I will try that.
But this was created by the function cli / dotnet publish.
I didn't change to the attribute form yet because this created all kinds of
different issues. For example you can't return a Task in the new
form. I didn't report that yet.

Fabio Cavalcante [e-mail] schrieb am Fr., 22. Juni 2018,
20:11:

Yes. Looked at your current setup, and your functions are using the old
style pre-compiled model (non-attributed model), and that model seems to
have a bug that fails to load your assembly locations into the probing
paths.

The fix is pretty simple and I'll have that in place soon, but the good
news is that you can make a small change to your deployment/directory
layout to workaround the issue in the meantime.

If you move all your binaries into a bin folder (this is the
default/fallback probing location) and just leave the function directories
with a function.json file in them (updated to point to
../bin/ for the scriptFile reference), that should work as
expected. One of the benefits of this is that you reduce the number of
duplicate files.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Azure/azure-functions-host/issues/2881#issuecomment-399531348,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADgNHuGKXu3uzdWle2D1ywd0_BgpVRMks5t_TNigaJpZM4UIPi-
.

[forki]

@fabiocav, so @tforkmann and myself just tried to move everything to bin folder and seems to do the trick. So that's good news.

After that we tried to upgrade the FSharp.Core to 4.5 (which is the current release) and now we get an loading issue again. I thought this was fixed and we could use latest?

Test is at
Europe West
Subscription ID: 65a46f89-725f-4b7b-89b0-75fe5425980c
FunctionApp: dptestaf
Function: proxy

2018-06-24T11:29:50.921 [Error] The following 1 functions are in error:

Proxy: Could not load file or assembly 'FSharp.Core, Version=4.5.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified. Could not load the specified file.

[forki]

@fabiocav any ideas what we can try?

[fabiocav]

Glad to hear that moving things to bin helped.

The FSharp.Core issue is caused by a runtime restriction that adds that assembly to the list of "shared runtime dependencies", which is a bit aggressive right now.

I had a task to remove that restriction for F# dependencies and a couple of others that still remain before the last release (that was coupled with the work to update our own F# dependencies for the compilation services), but that did slip. I'll have that done and can make those updates available for local testing. I'll also see if we can push that as quick update to 2.0.11888 to make it available on Azure.

[forki]

That would be awesome.

Fabio Cavalcante [e-mail] schrieb am Di., 26. Juni 2018,
19:34:

Glad to hear that moving things to bin helped.

The FSharp.Core issue is caused by a runtime restriction that adds that
assembly to the list of "shared runtime dependencies", which is a bit
aggressive right now.

I had a task to remove that restriction for F# dependencies and a couple
of others that still remain before the last release (that was coupled with
the work to update our own F# dependencies for the compilation services),
but that did slip. I'll have that done and can make those updates available
for local testing. I'll also see if we can push that as quick update to
2.0.11888 to make it available on Azure.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Azure/azure-functions-host/issues/2881#issuecomment-400399980,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADgNL0WK4NgI9VWkgk9wUztLOZmL7JMks5uAnC1gaJpZM4UIPi-
.

[JeryLu2020]

@fabiocav thank you Fabio for the continous help!!

[forki]

@fabiocav any news about this?

[forki]

ping

[davidebbo]

We are planning a 2.0 update next week (primarily to fix the http response issue mentioned here). @fabiocav, can you confirm whether the change you referred to above will also be part of that release?

[fabiocav]

Indeed, it will. Working to have that out ASAP. Will again update this thread once that process begins.

[forki]

Hugs!

Fabio Cavalcante [e-mail] schrieb am Do., 5. Juli 2018,
18:39:

Indeed, it will. Working to have that out ASAP. Will again update this
thread once that process begins.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Azure/azure-functions-host/issues/2881#issuecomment-402782704,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADgNA-1WN7kCoLWqbUuLi5Ydbt7hwUrks5uDkFUgaJpZM4UIPi-
.

[forki]

ping. any news here?

[fabiocav]

Apologies for the delay here. Those bits have been ready for a while and blocked on deployment activities. This is finally moving forward and we have the announcement out here: https://github.com/Azure/app-service-announcements/issues/124

[forki]

awesome thank you. Waiting for bits to show up on my function apps now ;-)

[forki]

@davidebbo how long does it usally take until the bits land in Europe West?

[fabiocav]

David is unavailable for the next few days, but the deployment is completing today.

[TurtleBeach]

With respect, which deployment?? There are at least 3 reasonably current runtimes in various threads, and it is increasingly hard to identify what MS considers "THE" current runtime, and which one is currently being deployed, to where. To wit: 2.0.11857 was a fallback I had to use because 2.0.11888 broke HTTP returns; 2.0.11888 seems to sometimes be referred to as 1.0.11888; in any case it is deployed in US EAST and at least partially works, though I reported an issue today with 2.0.11888 sometimes not reading JAVA_OPTS correctly; 2.0.11933 supposedly fixes the HTTP return issue but the gitHub issue was closed a few minutes ago and we have been unable to build a Java function to use the 2.0.11933 runtime locally (we keep getting 2.0.11888, even after following MS instructions).
It would be helpful to have some consistent and clear statements about what specific runtime is deployed where.
Thanks.

[pragnagopa]

I have update the title to include full version for the runtime. This issue is now fixed as part the 2.0.11933-alpha release

@TurtleBeach - Breaking changes for Java functions that require users to update functions that use HttpRequest/Response messages will be part of the next release. Annoucement https://github.com/Azure/app-service-announcements/issues/122 has been posted early as a heads up on the breaking changes.

[TurtleBeach]

@pragnagopa I understand and appreciate the heads-up notice. But we tried following your instructions from another thread to get the correct maven tools to create a simple HTTP triggered function using the new runtime. That is what is not working - the local environment still shows we are using the 2.0.11888 runtime. Did I miss something? (i.e., is it actually not possible to create / run a function locally using the new 2.0.11933 right now, even using tools from the sona..... (I forget the name) path you specified?

[pragnagopa]

@TurtleBeach - To test locally with runtime 2.0.11933, please get the latest Azure functions CLI version 2.0.1-beta.32. We just published this one.

You will not be able to test java breaking changes with runtime 2.0.11933.

[TurtleBeach]

Thanks - and now understood about the breaking changes.

[TurtleBeach]

@pragnagopa Now I am thoroughly confused and disheartened. A queue-trigger function that runs at the end of potentially long running jobs running on a separate VM just crashed because the runtime failed to read a valid string property defined in JAVA_OPTS (a property - storage account name) that has been in JAVA_OPTS and in use in all my Java functions for at least 6 months. I earlier today reported this happening with an HTTP triggered function running on 2.0.11857 (due to the issue with HTTP function returns in 2.0.11888). When the queue function was deployed, the runtime in EAST US was 2.0.11888 (I specifically wanted to see the effect of WEBSITE_USE_PLACEHOLDER = 1 on cold starts) and the function has been working. In reviewing the environment when the crash occurred this evening, I was surprised to see that 2.0.11933 has been deployed in EAST US (according to the portal). You told me just a few hours ago that the necessary tools to build a function for 2.0.11933 had just been deployed. In any case, the function handles bookkeeping at the end of a job, so 21 hours of modeling building went down the drain (more to the point, the information needed to close out the job). This does not make me happy. Despite my significant investment in time and dollars in the Azure platform, I have to now question that. What is going on??

[pragnagopa]

@TurtleBeach - Opened https://github.com/Azure/azure-functions-java-worker/issues/146 to figure out issue with JAVA_OPTS.

[forki]

@fabiocav thank you for all the hard work on this. This is a nice birthday present. I can confirm that runtime version 2.0.11933.0 (beta) works for me with FSharp.Core 4.5.1 and latest TaskBuilder. This is really really awesome and allows me to update my libs in my SAFE-Stack app - and that even without binding redirects or crazy assembly loading hacks. Thank you so much.

[tforkmann]

Hi there!

I can confirm as well that the runtime version 2.0.11933.0 (beta) is working for me like a charm!

@fabiocav thank you very much for you help on that!

@forki happy birthday bro!!

[fabiocav]

That's great to hear! Happy birthday! :)