Azure-functions-host: Azure Functions Host appears to be looking for dll's that don't exist

[TehWardy]

I initially hit this https://github.com/dotnet/core/issues/4390 which in turn led me to https://github.com/dotnet/roslyn/issues/42445 because i noticed this only happened in my functions project and not the rest of my stack.

The roslyn guys have suggested "quote" ...

Microsoft.CodeAnalysis is a metapackage that contains no dlls at all. instead it references all other roslyn packages. It exists as an easy way to take a dependency on a specific version of all roslyn assemblies.

Microsoft.CodeAnalysis.Common is a set of common libraries that all other roslyn packages have a dependency on. It is unlikely you ever need to reference this package directly.

From reading through the issues you posted elsewhere @TehWardy this may be an issue with the Azure Functions and which set of runtimes/dependencies it supports. I would file an issue on https://github.com/Azure/azure-functions-host

... so here I am.
Any help with this would be really appreciated as I have an urgent need to get a deployment through for an important project.

[TehWardy]

Ok i've managed to pin it down to a way smaller sample of code.
I don't know if the issue is something that belongs here or in https://github.com/dotnet/roslyn/issues/42445 but here's how to reproduce it ...

Install the .Net Core SDK v3.1 from the current download location on top of VS 2019.
Create a new Azure functions project targeting Core 3.0 and add a reference to the package Microsoft.CodeAnalysis.CSharp.Scripting

... I think this may happen with other stuff too since the project in my main solutoin isn't particularly complex.

but anyway I opened up the project file and replaced the reference from 3.0 to 3.1 (since for some reason the default was 3.0) ...

  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <AzureFunctionsVersion>v3</AzureFunctionsVersion>
    <LangVersion>latest</LangVersion>
  </PropertyGroup>

... adding the langversion node is something of a default for me too lately (apparently VS is happier in general when I do this.

I then updated the program.cs file to this ...

using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.CodeAnalysis.CSharp.Scripting;
using Microsoft.CodeAnalysis.Scripting;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Reflection;
using System.Threading.Tasks;

namespace FunctionApp1
{
    public static class Function1
    {
        [FunctionName("Function1")]
        public static async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequest req,
            ILogger log)
        {
            log.LogInformation("C# HTTP trigger function processed a request.");

            string name = req.Query["name"];

            string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
            dynamic data = JsonConvert.DeserializeObject(requestBody);
            name = name ?? data?.name;

            var result = await new ScriptRunner().Run<string>("\"Hello \" + Name", new[] { "System" }, new Args { Name = name });

            return name != null
                ? (ActionResult)new OkObjectResult(result)
                : new BadRequestObjectResult("Please pass a name on the query string or in the request body");
        }
    }

    public class Args
    {
        public string Name { get; set; }
    }

    internal class ScriptRunner
    {
        internal static Assembly[] references;

        static ScriptRunner()
        {
            // some of the stack might not have been loaded, lets make sure we load everything so we can give a complete response
            // first grab what's loaded
            var loadedAlready = AppDomain.CurrentDomain
                    .GetAssemblies()
                    .Where(a => !a.IsDynamic)
                    .ToList();

            // then grab the bin directory
            var binDir = Assembly.GetExecutingAssembly().CodeBase
                .Replace(Assembly.GetExecutingAssembly().CodeBase.Split('/').Last(), "")
                .Replace("file:///", "")
                .Replace("/", "\\");

            // from the bin, grab our core dll files
            var stackDlls = Directory.GetFiles(binDir)
                .Select(i => i.ToLowerInvariant())
                .Where(f => f.EndsWith("dll"))
                .ToList();

            // load the missing ones

            var stackPaths = stackDlls.Where(assemblyPath => loadedAlready.All(a => a.CodeBase.ToLowerInvariant() != assemblyPath.ToLowerInvariant()));
            foreach (var assemblyPath in stackPaths)
                loadedAlready.Add(Assembly.LoadFile(assemblyPath));

            references = loadedAlready.ToArray();
        }

        public async Task<T> Run<T>(string code, string[] imports, object args)
        {
            try
            {
                var referencesNeeded = references.Where(r => r.GetExportedTypes().Any(t => imports.Contains(t.Namespace)));
                var options = ScriptOptions.Default
                    .AddReferences(referencesNeeded)
                    .WithImports(imports);

                return await CSharpScript.EvaluateAsync<T>(code, options, args, args.GetType());
            }
            catch (NullReferenceException ex)
            {
                var typeAndCall = $"(({ex.TargetSite.DeclaringType.Name})object).{ex.TargetSite.Name}";
                var data = new List<string>();

                foreach (var k in ex.Data.Keys) data.Add($"{k}: {ex.Data[k]}");

                throw new Exception(ex.Message + $"\nContext: {ex.Source}\nTarget: {typeAndCall}\n{string.Join("\n", data)}");
            }
            catch (CompilationErrorException ex)
            {
                throw new Exception($"Compilation failed:\n{ex.Message}\n{string.Join(Environment.NewLine, ex.Diagnostics)}");
            }
        }

        public Task Run(string code, string[] imports, object args) => Run<bool>(code + ";return true;", imports, args);
    }
}

... build and run this.
You should get (assuming i'm not in some weird system specific situation here) ,,, and exception that reads ...

Could not load file or assembly 'Microsoft.CodeAnalysis, Version=3.4.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. The system cannot find the file specified.

... looking in the build output folder I can see ...

[TehWardy]

Yeh ok it definitely appears to be a functions framework host issue, here's the same code in a console app that works ...

using Microsoft.CodeAnalysis.CSharp.Scripting;
using Microsoft.CodeAnalysis.Scripting;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Reflection;
using System.Threading.Tasks;

namespace WorkflowConsole
{
    class Program
    {
        static void Main(string[] args)
        {
            var name = args[0];
            var result = new ScriptRunner().Run<string>("\"Hello \" + Name", new[] { "System" }, new Args { Name = name }).Result;

            Console.WriteLine(
                name != null
                    ? result
                    : "Please pass a name on the query string or in the request body"
            );

            _ = Console.ReadKey();
        }
    }

    public class Args
    { 
        public string Name { get; set; }
    }

    public class ScriptRunner
    {
        internal static Assembly[] references;

        static ScriptRunner()
        {
            // some of the stack might not have been loaded, lets make sure we load everything so we can give a complete response
            // first grab what's loaded
            var loadedAlready = AppDomain.CurrentDomain
                    .GetAssemblies()
                    .Where(a => !a.IsDynamic)
                    .ToList();

            // then grab the bin directory
            var binDir = Assembly.GetExecutingAssembly().CodeBase
                .Replace(Assembly.GetExecutingAssembly().CodeBase.Split('/').Last(), "")
                .Replace("file:///", "")
                .Replace("/", "\\");

            // from the bin, grab our core dll files
            var stackDlls = Directory.GetFiles(binDir)
                .Select(i => i.ToLowerInvariant())
                .Where(f => f.EndsWith("dll"))
                .ToList();

            // load the missing ones

            var stackPaths = stackDlls.Where(assemblyPath => loadedAlready.All(a => a.CodeBase.ToLowerInvariant() != assemblyPath.ToLowerInvariant()));
            foreach (var assemblyPath in stackPaths)
                loadedAlready.Add(Assembly.LoadFile(assemblyPath));

            references = loadedAlready.ToArray();
        }

        public async Task<T> Run<T>(string code, string[] imports, object args)
        {
            try
            {
                var referencesNeeded = references.Where(r => r.GetExportedTypes().Any(t => imports.Contains(t.Namespace)));
                var options = ScriptOptions.Default
                    .AddReferences(referencesNeeded)
                    .WithImports(imports);

                return await CSharpScript.EvaluateAsync<T>(code, options, args, args.GetType());
            }
            catch (NullReferenceException ex)
            {
                var typeAndCall = $"(({ex.TargetSite.DeclaringType.Name})object).{ex.TargetSite.Name}";
                var data = new List<string>();

                foreach (var k in ex.Data.Keys) data.Add($"{k}: {ex.Data[k]}");

                throw new Exception(ex.Message + $"\nContext: {ex.Source}\nTarget: {typeAndCall}\n{string.Join("\n", data)}");
            }
            catch (CompilationErrorException ex)
            {
                throw new Exception($"Compilation failed:\n{ex.Message}\n{string.Join(Environment.NewLine, ex.Diagnostics)}");
            }
        }

        public Task Run(string code, string[] imports, object args) => Run<bool>(code + ";return true;", imports, args);
    }
}

[andrew-vdb]

Got this issue too

What i did simply updating these dll

    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.DurableTask" Version="2.2.1" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage" Version="3.0.11" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.6" />

from

  <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.DurableTask" Version="2.1.0" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage" Version="3.0.10" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.2" />

Doing a rollback now and see if the issue is gone

[andrew-vdb]

it works again after rolling back, i think Microsoft.NET.Sdk.Functions somewhere between 3.03 - 3.0.6 is broken

[pseabury]

Same issue after upgrading the Functions SDK beyond 3.0.2. Rolling back to 3.0.2 allows the Functions app to run properly again.

[pedromvgomes]

I confirm the same. I've bumped into this issue while using version 3.0.9 of Microsoft.NET.Sdk.Functions. Changing to 3.0.2 get's it fixed.

Wondering when or if it will be fixed.

[DenisBalan]

Its important which version you have referenced in csproj.

For this, please have a look on tools folder from azure-functions-core-tools-3 (somewhere on your drive)
and extract file version

[System.Diagnostics.FileVersionInfo]::GetVersionInfo("Microsoft.CodeAnalysis.dll")

after that reference in your csproj expected version

And module gets referenced in runtime.

used this with codedom and cs-script (csscript for function app)

ping for @alrod to update tags.

[DenisBalan]

@TehWardy the files you see in your bin / obj directories are not the dll which will be used in runtime binding.

Azure function host decide on their own half based on version, and may override out-of-the-box libraries.

Thats why this is happening, you version is being replaced by azure function host.

[TehWardy]

General .Net etiquette

If I'm dependent on an assembly directly then I should be able to deploy that assembly with my application code and force usage of that particular version.
If the runtime wants to replace something then breaks it, that's on the runtime.

The complications as I see them

Forcing me to use specific versions of assemblies is against everything .Net is built on.
My understanding from my conversation with @brettsam over on https://github.com/Azure/azure-functions-host/issues/6512#issuecomment-679348093 is that The functions framework manages multiple Assembly load contexts and is therefore able to run code in each which should allow for multiple version if the framework is building everything correctly.

That said, as per that same issue I ended up over on https://github.com/dotnet/aspnetcore/issues/25305 which highlighted that some packages may not report their dependencies correctly further complicating the process.

Clearly there's a lot of different things at play here but my understanding from "general advice" I receive on most tickets is that to resolve any reference issues directly reference the version of the problem assembly directly in my project and the build process should honour that.

So @DenisBalan: the bottom line

We presumably need some way that we as consumers of the Azure functions framework can reliably control or derive the versions needed explicitly by the framework to enforce the same versioning constraints on our deployments?

Or am I missing something here?

[DenisBalan]

@TehWardy this is.
Whether you accept framework’s rules and bottleneck and just do the job, or trying to knock on the ecosystem and pass their checks .

The whole process of building function involves a lot of msbuild target files, where, if you want, you could replace libraries that came out of the box with your own ones, but there is no compatibility guarantee.

There is a lot to say on what should and should not allow the host to do, but starting from the point that there is a way to deal with those issues, you just have to comply with those. IMHO.

[TehWardy]

I'm not sure how that helps / what you're trying to say exactly but my take from it is ...
I'm not asking for any rules to be broken just for the tooling to enable me to clearly see what they are going to be when my code stack is deployed with it's dependencies to the cloud so I can correctly follow them or if need be make a decision that the tooling won't be informed enough to make to fix known issues in Microsoft's dependency stacking.

In some cases like my SignalR case above I can clearly do everything right and still not have it all work.
The relevant rules of course being ...

• only 1 version of an assembly can be loaded in to an app domain
• all dependencies must agree to a version to use".

In addition to that you're saying the rules that Azure Functions adds are something like ...

• It'll use whatever version of it's internal stuff it wants (and I have no control over that)
• If it and I share a dependency then we must agree on the version of that dep.

I'm simply asking if there's a means to allow for some ability to "manually resolve" or "advise" azure functions that I "prefer" a given version of something because other parts of .Net "require" that version (which I also have no control over).
When all the rules being followed clearly doesn't work as intended there has to be a fix or I end up waiting (as shown here) for months to get a working build because you guys have to change something in the framework on my behalf.

I've been told in other tickets that all .Net Core 3.x assemblies that are part of the Microsoft ecosystem _should be compatible_.
So if I see a package that incorrectly wants to depend on a 2.x assembly I should be able force it to use a 3.x equiv.
The build process and everything that then gets consumed on deployment should also follow that "rule".

I don't see how I'm breaking any rules here by trying to force consistency in my assembly stack when some of it isn't correctly linked on your (Microsofts) end.

I also hit issues like:

• https://github.com/Azure/azure-functions-host/issues/4049#issuecomment-479844510
• https://github.com/dotnet/aspnetcore/issues/9022

If Microsoft wants to have a framework dependency "recommendation" based on what is known to be tested and working or a "minimum version" requirement that's fine with me.
If I want to override that though I should be able to and know that the tooling will honour that because the assumption from the tool should be that I as a developer know better for some reason.

My reasoning is simple
You guys are human too, you make mistakes just as I do, packages "report incorrect information" or have inconsistencies and incorrect references. I should be able to correct that (and report that I have had to for you guys to fix in your own time) instead of having to wait weeks (or in some cases years) for you guys to fix things.

Alternative suggestion
Allow the functions framework to "do it's own thing" and "host" our code which can then have entirely it's own stack not requiring the Functions Framework to break our code as is being asked for over at https://github.com/Azure/azure-functions-host/issues/4543

[TehWardy]

Ping ... to prevent bot closes¬

[edgett]

This is still a problem in Microsoft.NET.Sdk.Functions 3.0.11

Microsoft.CodeAnalysis 3.8.0 works with Microsoft.NET.Sdk.Functions 3.0.2

The problem: 'Could not load file or assembly 'Microsoft.CodeAnalysis, Version=3.8.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'.

Can be resolved in Azure Functions by referencing: Microsoft.NET.Sdk.Functions 3.0.2 instead of 3.0.>=3

Code to demo problem, this will work in Functions 3.0.2 but fail between 3.0.3 to 3.0.11, latest at time of writing:

using System;
using System.Collections.Generic;
using System.IO;
using System.Reflection;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using Microsoft.CodeAnalysis.Emit;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;

namespace MMIT.DiffResearch.DiffTestRunner
{
    public static class Function1
    {
        [FunctionName("Function1")]
        public static void Run([TimerTrigger("0 0 29 2 1", RunOnStartup = true)] TimerInfo timerInfo, ILogger log)
        {
            var bc = new BasicCSharpCompile();
            var code = bc.GetDiffCodeFromGithub();
            var assembly = bc.CompileCodeAndCreateAssembly(code);

            var diffAlgorithmClassInfo = assembly.GetType("DiffMatchPatch.diff_match_patch");
            //Create an instance of diff_match_patch.
            var instanceOf_diff_match_patch = assembly.CreateInstance(diffAlgorithmClassInfo.FullName);
            //Invoke the diff_main function on the above instance of diff_match_patch.
            var diffs = instanceOf_diff_match_patch.GetType().InvokeMember("diff_main", BindingFlags.InvokeMethod, null, instanceOf_diff_match_patch, new[] { "This is code.", "This is not code." });


            Console.WriteLine(JsonConvert.SerializeObject(diffs));
        }

        public class BasicCSharpCompile
        {

            public Assembly CompileCodeAndCreateAssembly(string code)
            {

                var parsedCSharpCode = CSharpSyntaxTree.ParseText(code);
                var dependencyInfoForCSharpCode = GetNecessaryDllsToRunCode();
                var compileOptions = new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary).WithOverflowChecks(true).WithOptimizationLevel(OptimizationLevel.Release);
                var compileOperation = CSharpCompilation.Create("DiffPatch", new List<SyntaxTree>() { parsedCSharpCode }, references: dependencyInfoForCSharpCode, compileOptions);

                using (var compiledCodeStream = new MemoryStream())
                {

                    EmitResult compileResult = compileOperation.Emit(compiledCodeStream);

                    if (compileResult.Success)
                    {
                        compiledCodeStream.Position = 0;
                        return Assembly.Load(compiledCodeStream.ToArray());
                    }
                    else
                    {
                        throw new Exception("Error compiling code file.");
                    }
                }
            }

            //Get nesessary dependencies to run code
            IEnumerable<MetadataReference> GetNecessaryDllsToRunCode()
            {
                var assemblyPath = Path.GetDirectoryName(typeof(object).Assembly.Location);

                IEnumerable<MetadataReference> defaultReferences = new[]
                    {
            MetadataReference.CreateFromFile(Path.Combine(assemblyPath, "System.dll")),
            MetadataReference.CreateFromFile(Path.Combine(assemblyPath, "System.Core.dll")),
            MetadataReference.CreateFromFile(Path.Combine(assemblyPath, "System.Runtime.dll")),
            MetadataReference.CreateFromFile(Path.Combine(assemblyPath, "System.Text.RegularExpressions.dll")),
            MetadataReference.CreateFromFile(Path.Combine(assemblyPath, "System.Collections.dll")),
            MetadataReference.CreateFromFile(Path.Combine(assemblyPath, "System.Web.HttpUtility.dll")),
            MetadataReference.CreateFromFile(Path.Combine(assemblyPath, "System.Linq.dll")),
            //When you have problems for no reason add this:
            MetadataReference.CreateFromFile(Path.Combine(assemblyPath, "System.Private.CoreLib.dll")),
        };

                return defaultReferences;
            }


            public string GetDiffCodeFromGithub()
            {
                var wc = new System.Net.WebClient();
                return wc.DownloadString("https://raw.githubusercontent.com/google/diff-match-patch/master/csharp/DiffMatchPatch.cs");
            }

        }

    }
}