Kudu: Zip Push Deploy API returning Decoding Errors

[jgavin-zz]

I use the push deploy API to deploy my my .net core 2.0 function app project. This process has been working fine for months, and suddenly today it stopped working with no changes to our deployment scripts that broker the upload process. We are seeing this error across multiple projects with no source code changes. I can't share the source code, as it is stored in a private source control server, but here is the full output we are seeing from the API:

{"Message":"An error has occurred.","ExceptionMessage":"Unable to translate bytes [A1] at 
index 16 from specified code page to Unicode.","ExceptionType":"System.Text.DecoderFallbackException","StackTrace":"   at 
System.Text.DecoderExceptionFallbackBuffer.Throw(Byte[] bytesUnknown, Int32 index)\r\n   at 
System.Text.DecoderExceptionFallbackBuffer.Fallback(Byte[] bytesUnknown, Int32 index)\r\n   at 
System.Text.DecoderFallbackBuffer.InternalFallback(Byte[] bytes, Byte* pBytes, Char*& chars)\r\n   at System.Text.UTF8Encoding.GetChars(Byte* bytes, 
Int32 byteCount, Char* chars, Int32 charCount, DecoderNLS baseDecoder)\r\n   at System.Text.DecoderNLS.GetChars(Byte* bytes, Int32 byteCount, Char* 
chars, Int32 charCount, Boolean flush)\r\n   at System.Text.DecoderNLS.GetChars(Byte[] bytes, Int32 byteIndex, Int32 byteCount, Char[] chars, Int32 
charIndex, Boolean flush)\r\n   at System.Text.DecoderNLS.GetChars(Byte[] bytes, Int32 byteIndex, Int32 byteCount, Char[] chars, Int32 charIndex)\r\n 
  at System.IO.StreamReader.ReadBuffer(Char[] userBuffer, Int32 userOffset, Int32 desiredChars, Boolean& readToUserBuffer)\r\n   at 
System.IO.StreamReader.Read(Char[] buffer, Int32 index, Int32 count)\r\n   at Newtonsoft.Json.JsonTextReader.ReadData(Boolean append, Int32 
charsRequired)\r\n   at Newtonsoft.Json.JsonTextReader.ParseValue()\r\n   at Newtonsoft.Json.JsonTextReader.Read()\r\n   at 
Newtonsoft.Json.JsonReader.ReadAndMoveToContent()\r\n   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadForType(JsonReader reader, 
JsonContract contract, Boolean hasConverter)\r\n   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type 
objectType, Boolean checkAdditionalContent)\r\n   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)\r\n   at 
System.Net.Http.Formatting.BaseJsonMediaTypeFormatter.ReadFromStream(Type type, Stream readStream, Encoding effectiveEncoding, IFormatterLogger 
formatterLogger)\r\n   at System.Net.Http.Formatting.JsonMediaTypeFormatter.ReadFromStream(Type type, Stream readStream, Encoding effectiveEncoding, 
IFormatterLogger formatterLogger)\r\n   at System.Net.Http.Formatting.BaseJsonMediaTypeFormatter.ReadFromStream(Type type, Stream readStream, 
HttpContent content, IFormatterLogger formatterLogger)\r\n   at System.Net.Http.Formatting.BaseJsonMediaTypeFormatter.ReadFromStreamAsync(Type type, 
Stream readStream, HttpContent content, IFormatterLogger formatterLogger)\r\n--- End of stack trace from previous location where exception was thrown 
---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at 
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at 
System.Net.Http.HttpContentExtensions.d__0`1.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at 
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at 
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at 
Kudu.Services.Deployment.PushDeploymentController.d__10.MoveNext() in C:\\Kudu 
Files\\Private\\src\\master\\Kudu.Services\\Deployment\\PushDeploymentController.cs:line 136\r\n--- End of stack trace from previous location where 
exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at 
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at 
Kudu.Services.Deployment.PushDeploymentController.d__11.MoveNext() in C:\\Kudu 
Files\\Private\\src\\master\\Kudu.Services\\Deployment\\PushDeploymentController.cs:line 164"}

[davidebbo]

@jgavin can you share the name of any affected Web App so we can investigate?

[jgavin-zz]

@davidebbo wheels-centralus-devl-tp-transform-ft-app is one of our apps that's being affected

[davidebbo]

@jgavin any chance you are setting the content type to application/json when sending the zipdeploy request? If so, this might be related to a recent change. When calling zipdeploy, the content you are sending is binary and definitely not json, and I believe that is confusing the code.

/cc @ankitkumarr

[jgavin-zz]

@davidebbo That was exactly right. I changed the content type to application/octet-stream and it worked just fine. Thanks.

[davidebbo]

Great. So technically, this is by design, but for backward compat, we may be able to change the code so this continues to work. I will let @ankitkumarr take it from there.

[davidebbo]

This also came up on https://developercommunity.visualstudio.com/content/problem/363405/azure-app-service-deploy-task-version-437-fails-wh.html (at least I assume it's the same issue). + @ahmelsayed to help sort this out.

[pjirsa]

Curious. We are able to manually publish the zip by browsing the kudu site /zipdeployUI page. But, it fails when using the push api (from our build pipeline action).

[davidebbo]

@pjirsa it's likely because they incorrectly set the mime type to application/json, per above. This used to not matter but now it does. They'll need to fix that in the build pipeline action.

Can you try setting a KUDU_EXTENSION_VERSION Azure app setting to 78.11002.3584 and see if that helps (going back to previous version of Kudu)?

[PeterOrneholm]

I have the same scenario (ZIP deploy of .NET core to (in my case) Azure Web Apps), but different exception. Also started today.

Debug output from Azure DevOps:

{"statusCode":400,"statusMessage":"Bad Request","headers":{"cache-control":"no-cache","pragma":"no-cache","content-length":"836","content-type":"application/json; charset=utf-8","expires":"-1","server":"Microsoft-IIS/10.0","x-ms-request-id":"XXXX","x-aspnet-version":"4.0.30319","x-powered-by":"ASP.NET","set-cookie":["ARRAffinity=XXXX;Path=/;HttpOnly;Domain=al-samples-mvcclient.scm.azurewebsites.net"],"date":"Mon, 22 Oct 2018 20:05:28 GMT","connection":"close"},"body":{"Message":"An error has occurred.","ExceptionMessage":"Object reference not set to an instance of an object.","ExceptionType":"System.NullReferenceException","StackTrace":"   at Kudu.Services.Deployment.PushDeploymentController.<GetZipURLFromJSON>d__10.MoveNext() in C:\\Kudu Files\\Private\\src\\master\\Kudu.Services\\Deployment\\PushDeploymentController.cs:line 150\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Kudu.Services.Deployment.PushDeploymentController.<PushDeployAsync>d__11.MoveNext() in C:\\Kudu Files\\Private\\src\\master\\Kudu.Services\\Deployment\\PushDeploymentController.cs:line 164"}}

Can you try setting a KUDU_EXTENSION_VERSION Azure app setting to 78.11002.3584 and see if that helps (going back to previous version of Kudu)?

Did not work for me unfortunately.

[pjirsa]

we were able to get ours working by adding this to our slot's appsettings:
KUDU_EXTENSION_VERSION - 78.11002.3584

[davidebbo]

@PeterOrneholm the KUDU_EXTENSION_VERSION workaround should definitely work, as it did for @pjirsa. Maybe you didn't set it quite right? Of course, this is only meant as a temporary workaround while we figure this out.

[PeterOrneholm]

@PeterOrneholm the KUDU_EXTENSION_VERSION workaround should definitely work, as it did for @pjirsa. Maybe you didn't set it quite right? Of course, this is only meant as a temporary workaround while we figure this out.

Ahh, sorry. I did just try to add it manually in the Azure portal but just realized I run my ARM template before deploying my webapp so that config was lost... I'll add it to the ARM template and that should work.

[craig-wagner]

We ran into the same problem today deploying our Azure Functions project. Adding the KUDU_EXTENSION_VERSION to our ARM template fixed it and we can deploy again.

FWIW, we're using v4 (preview) of the Azure App Service Deploy task.

[ankitkumarr]

As an update, this issue should be fixed and KUDU_EXTENSION_VERSION should not be necessary anymore.

I am closing this issue, but please feel free to re-open if you still see the issue.

[fschneider1509]

Error still exists here with the Azure App Service Deploy stable task.
Setting the KUDU_EXTENSION_VERSION didn't fix the problem.

[vit-svoboda]

I'm sending {"packageUri":"path-to-the-zip-archive"} in the request body, what seems to me like a normal JSON. What do you mean with binary definitely not json content, @davidebbo?

[davidebbo]

@vit-svoboda my comment above was related to what someone else was doing more than two years ago. Your case may be different.

I suggest starting a new issue and describe your scenario. Also, I'm no longer working on this project, so I'll let others help out.