Azure-functions-host: Locking issue with DLLs in Functions

[richardagreene]

I'm having issues of files locking within a Functions App, that is preventing Continuous Delivery. The issue presents itself when referencing a .NET DLL within the function, running a test, then attempting to deploy a new version of the referenced .NET DLL.

Here are the steps needed to replicate the issue:
1) Create a simple class library in Visual Studio called "FunctionsTest.Calculation" and build.

``` c#
using System;
using System.Text;
namespace FunctionsTest.Calculation
{
public class SomeWork
{
public string Execute(string inputTest)
{
return String.Format("result {0}", inputTest);
}
}
}

2) Create a new Functions App
3) Create a new Manually triggered C# Function called "ManualTriggerCSharp1", with the following code

``` c#
#r "FunctionsTest.Calculation.dll"
using System;
using FunctionsTest.Calculation;

public static void Run(string input, TraceWriter log)
{
    log.Info($"C# manually triggered function called with input: {input}");
    var somework = new SomeWork();
    var result = somework.Execute(input);
    log.Info("result:" + result );
}

4) Upload a copy of the generated DLL to the BIN directory of the Function App using FTP
5) Run the function .. all should execute correctly.
6) Change the return parameter type of the method "Execute" within the local class library and rebuild the DLL.

``` c#
using System;
using System.Text;

namespace FunctionsTest.Calculation
{
public class SomeWork
{
public double Execute(string inputTest)
{
return 1f;
}
}
}

7) Attempting to FTP the new DLL to Azure returns:
     Command: STOR FunctionsTest.Calculation.dll
     Response: 550 The process cannot access the file because it is being used by another process.
Wait a while, then try again ... wait again.. do this a few few times and you should get the same error.
8) Eventually, when you get tired of waiting, rename your locally generated DLL file to something else for example "FunctionsTest.Calculation1.dll" and FTP that to the server.  This should work fine, so you now have two DLLs in the BIN directory.
9) Update the reference in the Function script to point to the _new_ DLL (note line: **#r "FunctionsTest.Calculation1.dll" *\* but continue to use _old_ namespace **using FunctionsTest.Calculation**).

``` c#
#r "FunctionsTest.Calculation1.dll"
using System;
using FunctionsTest.Calculation;

public static void Run(string input, TraceWriter log)
{
    log.Info($"C# manually triggered function called with input: {input}");
    var somework = new SomeWork();
    var result = somework.Execute(input);
    log.Info("result:" + result );
}

10) Run the script again and you get the error, showing the function is still working against the _old_ DLL reference, even after modifying the code:

Exception while executing function: Functions.ManualTriggerCSharp1.
mscorlib: Exception has been thrown by the target of an invocation.
Method not found: 'Double
FunctionsTest.Calculation.SomeWork.Execute(System.String)'.

11) Change the namespace in Visual Studio of the local DLL to "FunctionsTest.Calculation1", rebuild and FTP to the server under the filename "FunctionsTest.Calculation1.dll".
12) Change the Function to use the _new_ namespace (note line: using FunctionsTest.Calculation1) and success, we're up and running again.

``` c#

r "FunctionsTest.Calculation1.dll"

using System;
using FunctionsTest.Calculation1;
public static void Run(string input, TraceWriter log)
{
log.Info($"C# manually triggered function called with input: {input}");
var somework = new SomeWork();
var result = somework.Execute(input);
log.Info("result:" + result );
}
```

This is perhaps a edge case; however it does seem to indicate that Functions are holding on to references to external resources, even after a shutdown/restart or recompile. With CI we are updating these DLLs continually and it's a drawback that we can't simply update the files.

[fabiocav]

Currently, those assemblies are not shadow copied, which leads to this locking behavior. You mentioned you ran into this issue after a shutdown, which should indeed release all of those assemblies. Did you perform an explicit restart?

[richardagreene]

Thanks for the quick response. I can confirm this occurred following a manual restart; however this was originally reported back in early May so things may have changed since then. You should run through the full set of steps to confirm with the latest build.

The core DLL locking issue is however is not the point of me raising the issue, it's more about the Continuous Integration/Deployment customer story. A manual restart, even if it worked, would not solve the underlying issue in that external DLLs are locked while the function is "in flight". In a test environment the DLLs would be updated 10-20 times a day. We can't shut down every function every time an update takes place. Also the REST API does not seem to allow for automated shutdown/restarts; however that may have changed since I last reviewed the rather limited documentation. More seriously, changing the file reference in the function code did not force a reload of the external DLL (i.e. step 9); so without the error reported, we could have some serious consequences in processing data. The developer believes they are running file "X.dll", the function code say's it's running "X.dll", but the actual Function is running an old copy of the code.

[paulbatum]

One thought - recycling the app domain would address this issue, the problem we have is that Edge.js throws an access violation in this case. If we made our Edge initialization smarter to only occur in the presence of JavaScript functions, we could then proceed with the app domain recycle for function apps that do not have JavaScript based functions. That should address the issue for users that use C#/F# exclusively.

[ashelley]

Just got burned by this myself.

Using wawsdeploy, kudu drag and drop / unzip, view files tab in the function editor cannot be used to deploy if you have a #r referencing another dll due to this locking issue. Haven't tried git deployment yet.

Example output from file locking:

C:\inetpub\wwwroot\azure-func-deploy-test> wawsdeploy TestQueueFunction.zip publish.PublishSettings /d
Starting deployment...
Deployment failed: Web Deploy cannot modify the file 'AzureFunctionDependencyTest.dll' on the destination because it is locked by an external process.  In order to allow the publish operation to succeed, you may need to either restart your application to release the lock, or use the AppOffline rule handler for .Net applications on your next publish attempt.  Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_FILE_IN_USE.

Perhaps a note of this caveat should be added here.

https://docs.microsoft.com/en-us/azure/azure-functions/functions-reference#a-idfileupdatea-how-to-update-function-app-files

Shutting down the app service doesn't seem like a good solution especially if you are just trying to update code in one function.

Anyone else have a better work around for this?

[davidebbo]

When using msdeploy, can you try setting MSDEPLOY_RENAME_LOCKED_FILES=1 in your Azure Appsettings? This enables a mode that should allow it to deal with loaded assemblies.

[ashelley]

@davidebbo thank you! this does seem to work!

[ashelley]

@davidebbo i actually lied about this being successful... it does let me update the files.. however... to get the assembly to reload i think i have to touch function.json

[davidebbo]

Yes, I'm not surprised. This helps avoid deployment failures, but the runtime is still using the old dll.

Did you check if touching function.json actually works? If not, then the foolproof solution is to restart the Function App, though that's of course heavy weight.

[ashelley]

yeah it does seem that touching function.json does work but i just did limited testing on it.

[akrock]

I ran into this issue as well. We are attempting to setup function deployments to where we can easily include business logic to prevent having to duplicate it all in CSX.

Adding MSDEPLOY_RENAME_LOCKED_FILES=1 does allow deployments to succeed (in our case MsDeploy via Octopus Deploy) however the runtime will not pick up on the changed DLL. After the DLL was updated I also tested updating the CSX, function.json and project.json all caused the function to be recompiled but still use the old referenced DLL version. This seems to be the same issue as reported in #953.

If we force a restart by killing the process in kudu or if there is no traffic on the function app and it shuts down it will eventually consume the new DLL.

[Edit]

I ended up working around this by adding a heavy handed Restart-AzureWebsite powershell call after deployment. This results in the new DLLs being loaded with the side effect of about 10 seconds of throwing 50X errors.

To ensure that we don't miss any calls to the web hooks while the functions app is restarting I put the calls behind Azure API Management with a retry policy setup to retry any requests with 50X errors for 30 seconds.

Certainly not graceful but it will work until the team can come up with a more permanent solution.

[fabiocav]

@richardagreene for the issue above, the fact that your assembly had the same name (just a different file name) is what causes the runtime to use the existing assembly, if loaded. Changing the assembly name or version should result in proper lookup to find the new dependency as the loaded one won't be a match.

For the issue where a deployment succeeds with but DLLs are not reloaded, we have some work planned to for a fix that will eliminate the need to manually restart the site. You can track that here: #1023

Closing this as will be updating the referenced issue for moving forward.

[paulbatum]

Just a short note for anyone running into problems discussed in this issue. In Functions V2 there is no shadowcopy behavior (because it does not exist in .NET core) and so running into locking issues is more likely. However we have implemented support for the "take app offline" feature that is provided through msdeploy. To use this feature from VS, when you're publishing look for the "take app offline" checkbox. If you can't see it, you can specify the setting manually by going to your .pubxml file for your publish profile and make sure you have this setting:

<EnableMsDeployAppOffline>True</EnableMsDeployAppOffline>

[4c74356b41]

we tried everything in this thread and even stopping the application completely. it doesnt help.

[fabiocav]

@4c74356b41 can you please open a new issue with details (function app version and the other requested information) so we can follow up?