Kudu: Update MSBuild in App Services

[jt000]

I'm seeing an issue when building in my deploy.cmd, which appears to be fixed in Microsoft Build Tools 2015 Update 3 (msbuild version 14.0.25420.1) according to dotnet/corefx#14050. I've been able to verify that I do not repro this issue on my work machine after upgrading, but still see the issue in my deploy.cmd likely due to the 14.0.23107.0 of msbuild in my app service.

Is there a way to upgrade to 14.0.25420.1 in my app service instances? If not, who should I contact to get this resolved?

[davidebbo]

Please share a minimal repo that we can repro your issue with, per this page. Thanks!

[davidebbo]

Note that when building Core projects, the default Kudu script uses dotnet.exe and not msbuild. But anyway, we'll take a look at what you have.

[jt000]

Thanks @davidebbo. For a _really_ minimal repro, just log into the console on SCM and run msbuild /version. You'll see that the installed version is 14.0.23107.0 and the latest version is 14.0.25420.1 which is installed by Microsoft Build Tools 2015 Update 3.

If you'd like a repro of the original problem, then the repro @michael-x gave works pretty well:

A simpler set of steps to reproduce:

1. On machine A: Create a new .NET 4.6 Class Library project in Visual Studio 2015
2. On machine B: Build using MSBuild 14 --> Works
3. On machine A: Add NuGet package System.Text.Encoding.CodePages 4.3.0
4. On machine B: Build using MSBuild 14 --> Works
5. On machine A: Add NuGet package System.Xml.ReaderWriter 4.3.0
6. On machine B: Build using MSBuild 14 --> Error
   CSC : error CS1703: Multiple assemblies with equivalent identity have been imported: 'D:\Source\Test\ClassLibrary2\packages\System.Xml.ReaderWriter.4.3.0\lib\net46\System.Xml.ReaderWriter.dll' and 'C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework.NETFramework\v4.6\Facades\System.Xml.ReaderWriter.dll'. Remove one of the duplicate references.

Machine A has got Visual Studio 2015 installed, machine B has got the .NET Framework 4.6.2 SDK installed, no Visual Studio (build machine).

Note that "machine B" in this case is the App Services instance. If the latter is what you're looking for, then I can create a separate git repro for it as mentioned in the link you provided. Repro below...

[jt000]

Repro for the underlying issue created at https://github.com/jt000/repro-kudu-2336

Just create an app service & direct to this repo as a deployment source and you'll see the following error during deployment:

Command: deploy.cmd
::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: Setup
:: ---------------------------

::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: Deployment
:: ---------------------------

:: "D:\Program Files (x86)\SiteExtensions\Kudu\59.60207.2684\bin\Scripts\nuget.exe" restore "D:\home\site\repository\MyProj\MyProj.sln" -Verbosity detailed -NonInteractive
NuGet Version: 3.5.0.1938
MSBuild auto-detection: using msbuild version '14.0' from 'D:\Program Files (x86)\MSBuild\14.0\bin'. Use option -MSBuildVersion to force nuget to use a specific version of MSBuild.
Restoring NuGet package System.Text.Encoding.CodePages.4.3.0.
Restoring NuGet package System.Xml.ReaderWriter.4.3.0.
  GET https://api.nuget.org/v3-flatcontainer/system.text.encoding.codepages/4.3.0/system.text.encoding.codepages.4.3.0.nupkg
  GET https://api.nuget.org/v3-flatcontainer/system.xml.readerwriter/4.3.0/system.xml.readerwriter.4.3.0.nupkg
  OK https://api.nuget.org/v3-flatcontainer/system.text.encoding.codepages/4.3.0/system.text.encoding.codepages.4.3.0.nupkg 35ms
Acquiring lock for the installation of System.Text.Encoding.CodePages 4.3.0
Acquired lock for the installation of System.Text.Encoding.CodePages 4.3.0
Installing System.Text.Encoding.CodePages 4.3.0.
  OK https://api.nuget.org/v3-flatcontainer/system.xml.readerwriter/4.3.0/system.xml.readerwriter.4.3.0.nupkg 179ms
Acquiring lock for the installation of System.Xml.ReaderWriter 4.3.0
Acquired lock for the installation of System.Xml.ReaderWriter 4.3.0
Installing System.Xml.ReaderWriter 4.3.0.
Completed installation of System.Text.Encoding.CodePages 4.3.0
Adding package 'System.Text.Encoding.CodePages.4.3.0' to folder 'D:\home\site\repository\MyProj\packages'
Completed installation of System.Xml.ReaderWriter 4.3.0
Adding package 'System.Xml.ReaderWriter.4.3.0' to folder 'D:\home\site\repository\MyProj\packages'
Added package 'System.Text.Encoding.CodePages.4.3.0' to folder 'D:\home\site\repository\MyProj\packages'
Added package 'System.Xml.ReaderWriter.4.3.0' to folder 'D:\home\site\repository\MyProj\packages'

NuGet Config files used:
    C:\DWASFiles\Sites\#1repro-kudu-2336\AppData\NuGet\NuGet.Config

Feeds used:
    https://api.nuget.org/v3/index.json

Installed:
    2 package(s) to packages.config projects

:: "D:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exe" "D:\home\site\repository\MyProj\MyProj.sln" /verbosity:m /nologo /p:ResolveNuGetPackages=false /p:Configuration=Release
CSC : error CS1703: Multiple assemblies with equivalent identity have been imported: 'D:\home\site\repository\MyProj\packages\System.Xml.ReaderWriter.4.3.0\lib\net46\System.Xml.ReaderWriter.dll' and 'D:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Xml.ReaderWriter.dll'. Remove one of the duplicate references. [D:\home\site\repository\MyProj\MyProj.csproj]
Failed exitCode=1, command="D:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exe" "D:\home\site\repository\MyProj\MyProj.sln" /verbosity:m /nologo /p:ResolveNuGetPackages=false /p:Configuration=Release

An error has occurred during web site deployment.
\r\nD:\Program Files (x86)\SiteExtensions\Kudu\59.60207.2684\bin\Scripts\starter.cmd deploy.cmd

[davidebbo]

Looking at your repo, I note a couple things:

• you have a custom deployment script, so you're not letting Kudu use its default handling. As mentioned above, for Core, Kudu uses dotnet.exe, not msbuild. If you need to change the script, start with the default script and modify from there,
• your repo has a library, but has no web project. Kudu can only deploy web apps.

[jt000]

Thanks for taking a look @davidebbo. A couple things on your couple things :)

• Yes, it is a custom deployment script that was based on the original from a call to azure site deployscripts that I trimmed down for simplicity purposes. I think at some point people need to modify the original to do things that are specific to their application. Are you suggesting that people shouldn't have to use msbuild in a deploy.cmd? AFAIK, dotnet.exe isn't capable of building csproj projects. Or maybe I'm misunderstanding what you mean by "Kudu's default handling". Could I add something to this script to allow MSBuild to successfully build this project?
• Yes, this is a simplified repro. The part where it would be deployed (i.e. in a call to kudusync) would be after the msbuild call, which in this repro it can't get to so the fact that it isn't a web app is moot. If converting this to a web project would prove something, then I can do that as well but that seems unnecessary to prove that there is an issue.

Please consider reopening this. I am currently unable to deploy due to this issue.

[davidebbo]

Yes, dotnet.exe is absolutely able to build and deploy csproj based project. In fact, it the new Core world, csproj is all there is (project,json/xproj no longer exists).

But I'm still confused about your repo. msbuild or not, you can't deploy a class library as a web site. It's just not something that makes sense. What do you expect it to do?

Also, note that VS 2015 does not support .NET Core. It only supports the old non-csproj, so you need to use 2017 (or dotnet new). Yes, this can be confusing. See this post which explains some of it (it's slightly outdated).

[davidebbo]

And to add more specific things you should do if you want to deploy a Core project:

• Create a fully valid Core Web project using either latest VS2017 or dotnew new web.
• Try pushing it to Kudu as is
• If the default script doesn't do what you want, start for that script and modify what you need

[jt000]

In the repro it fails on the call to MSBuild w/ the above error, so there is no DLL to be deployed. With a newer version of MSBuild.exe this issue is resolved and the project builds fine. I wouldn't expect it to magically turn into a website, but this issue is only about MSBuild in Kudu so I trimmed down the repro do just the build part. Sorry for the confusion. I probably should've been a little clearer.

[jt000]

Just to be clear, the project I'm working on _(obviously not the one in the repro)_ has been in progress for the past year & may not easily be portable to Core. This issue just started introducing itself after recently taking a dependency on SendGrid.

I'm also not sure what issues might arise from converting to a Core project. From what I recall OData wasn't available in Core, so I wasn't able to do that at the time when this project was first started. I can certainly look into porting the site to Core, but I'm not sure I understand why that should be necessary here unless non-Core projects aren't supported in Azure (which would be unfortunate). Shouldn't msbuild.exe still be supported as a build step?

I updated the repro with a deploy.cmd created directly from azure site deploymentscript --aspWAP MyProj\MyProj.csproj --solutionFile MyProj\MyProj.sln and the issue still repros. The only difference between this one is that it does the nuget restore as a part of the msbuild step.

[xt0rted]

@jt000 are you using v9 of SendGrid? There was a bad .NET core dependency on it which was fixed in v9.0.10 which may be related to your problem since you said this started after adding that.

[davidebbo]

So you are not using Core after all? I think we're really talking past each other because we don't have a good repro to look at here. It doesn't have to be your full complex web app, but it does need to be a web app, because that's the only thing that Kudu is designed to build and deploy.

[jt000]

@davidebbo - Nope, not a Core project. The linked bug was (i think) regarding referencing a Core project that takes a dependency on Xml.ReaderWriter (which coincides with what @xt0rted is referring to), but the web app itself isn't a Core project. I'll convert the repro to a web app when I have a chance (likely tomorrow), but I am also building class libraries along with my web app in my production application. I'm surprised those project types aren't supported since they're both essentially the same project type as far as CSC.exe is concerned... By the way, thanks a ton for the help on a Saturday. I really appreciate it.

@xt0rted - Yep, I think that was exactly what caused this in the first place. I've stepped back to v8 of SendGrid for other reasons, but I probably still have that Core dependency in my packages.config. I remember hitting this a while back w/ a different package & having to step back to undo it, so I imagine it'll come up again even after fixing the SendGrid issue. I'll take a look at that to see if I can workaround this in the meantime...

[davidebbo]

Keeping active till this is better understood. Note that we'll soon have the VS 2017 msbuild, which hopefully will work for all scenarios.

[jt000]

I updated the project to be a WebApp and have it repro'ing, however in doing so I noticed a couple of things. _(hopefully, this doesn't make the scenario more confusing)_

When I created a barebones web app and followed the repro above I got the same error in VS as I was getting on Azure, which is definitely not what I expected. After a little investigation, I noticed that the following packages were installed:

  <package id="Microsoft.CodeDom.Providers.DotNetCompilerPlatform" version="1.0.0" targetFramework="net461" />
  <package id="Microsoft.Net.Compilers" version="1.0.0" targetFramework="net461" developmentDependency="true" />

After upgrading Microsoft.Net.Compilers to latest, I no longer saw the error in VS.

However, in looking back on my original scenario I do not take a reference to these packages, so I then removed these packages entirely from the repro (since that is my original scenario) and I am back to the the same repro where I do not see the error in VS, but I do see the error in Kudu.

In Summary:

1. MSBuild 14.0.23107.0 has a bug
2. Microsoft.Net.Compilers 1.0.0 also has the same bug
3. Microsoft.Net.Compilers 1.3.2 has the bug fixed
4. MSBuild 14.0.25420.1 also has the bug fixed

For projects using Microsoft.Net.Compilers, the package just needs to be updated. For projects not using Microsoft.Net.Compilers (i.e. class libraries & console applications by default) the MSBuild library needs to be updated in Azure. However, adding the DotNetCompilerPlatform package (which adds the Microsoft.Net.Compilers) to the class libraries & console applications would also fix this (I presume by having their own targets\props rather than the ones installed by MSBuild). I've updated a workarounds branch to look into this.

In Conclusion:
There's a bug in MSBuild 14.0.23107.0, which can be fixed by

• Kudu upgrading the instances to use a newer version, which it sounds like is coming w/ the 2017
• or by packages adding Microsoft.CodeDom.Providers.DotNetCompilerPlatform

I can imagine someone else might get discouraged when using the azure site deploymentscript --dotNetConsole <proj> template. I don't know if you want to keep this bug open to possibly help some other person who might be hitting this w/ their webjob while waiting for the VS 2017 build, but I think I'm all set by adding the DotNetCompilerPlatform package. Thanks for your help @davidebbo!

[davidebbo]

Is this latest error something you see locally, or on Azure? And when you say 'Updating the compiler', what specifically are you referring to?

[davidebbo]

Also, I suggest removing the custom deployment script to further isolate, unless you have a specific reason that you need it.

[jt000]

_Sorry, ignore that comment about it not working at runtime. I had a typo in my csproj & overreacted since coincidentally the typo was regarding the Xml.ReaderWriter.dll. That comment was deleted. My issue appears to currently be resolved. Thanks!_

[davidebbo]

For proper closure, can you clarify what the specific issue was, e.g. by pushing a change to your test repo that corrects it?

[jt000]

Sure thing. I also updated the repro to be a little more of a realistic scenario using a WebJob instead of a class library or web app with DotNetCompilerPlatform removed. If you look at the last 2 commits it has the issue & the resolution

• commit 8d25a40 - Broken - A webjob that uses the MSBuild 14.0.23107.0 build tasks on Kudu
• commit 35f7c3b - Fixed - Updated to use Microsoft.Net.Compilers 1.3.2 (via Microsoft.CodeDom.Providers.DotNetCompilerPlatform) which includes updated build tasks

Thanks again for your help!

[davidebbo]

Great, thanks for closing the loop! This might end up helping someone else at some time.