Kudu: Linux: zipdeploy API times out in synchronous mode

Created on 25 May 2018 · 11Comments · Source: projectkudu/kudu

Due to some pretty severe issues with VSTS deployments and performance we've using CircleCI to assemble our application, compress to a zip and deploy via the REST API. Unfortunately due to this we're having to deploy with all the node modules bundled inside and we've bypassed the npm install phase of the deployment script. This results in deploying a 50Mb zip file.

This all works, we have our application deploying and running. We're using the non-asynchronous API because we can use that detect when the application is deployed and due to fan-out out build pipeline isn't getting choked, the region deployments happen in parallel.

However the API lies, it's doing weird things instead of holding the connection until deployment is complete.

At a little over 4 minutes we've had this.

#!/bin/bash -eo pipefail
if [ "${CIRCLE_BRANCH}" == "master" ]; then
  curl -i -X POST -u SNIP --data-binary @"/tmp/drop.zip" https://SNIP.scm.azurewebsites.net/api/zipdeploy
fi
HTTP/1.1 100 Continue



HTTP/1.1 502 Bad Gateway

Content-Type: text/html

Server: Microsoft-IIS/10.0

Date: Thu, 24 May 2018 08:18:08 GMT

Content-Length: 1477



<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">

<head>

<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/>

<title>502 - Web server received an invalid response while acting as a gateway or proxy server.</title>

<style type="text/css">

<!--

body{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-serif;background:#EEEEEE;}

fieldset{padding:0 15px 10px 15px;} 

h1{font-size:2.4em;margin:0;color:#FFF;}

h2{font-size:1.7em;margin:0;color:#CC0000;} 

h3{font-size:1.2em;margin:10px 0 0 0;color:#000000;} 

#header{width:96%;margin:0 0 0 0;padding:6px 2% 6px 2%;font-family:"trebuchet MS", Verdana, sans-serif;color:#FFF;

background-color:#555555;}

#content{margin:0 0 0 2%;position:relative;}

.content-container{background:#FFF;width:96%;margin-top:8px;padding:10px;position:relative;}

-->

</style>

</head>

<body>

<div id="header"><h1>Server Error</h1></div>

<div id="content">

 <div class="content-container"><fieldset>

  <h2>502 - Web server received an invalid response while acting as a gateway or proxy server.</h2>

  <h3>There is a problem with the page you are looking for, and it cannot be displayed. When the Web server (while acting as a gateway or proxy) contacted the upstream content server, it received an invalid response from the content server.</h3>

 </fieldset></div>

</div>

</body>

</html>

At a little over 4 minutes we've also had this

#!/bin/bash -eo pipefail
if [ "${CIRCLE_BRANCH}" == "development" ]; then
  curl -i -X POST -u SNIP --data-binary @"/tmp/drop.zip" https://SNIP.scm.azurewebsites.net/api/zipdeploy
fi
HTTP/1.1 100 Continue



HTTP/1.1 504 Gateway Time-out

Content-Length: 260

Content-Type: text/html

ETag: "5b054e03-104"

Server: nginx

Set-Cookie: ARRAffinity=5d10da0e40ca983507b2d5977c4a4d6507be25813895d6f7c215841f05c3dce8;Path=/;HttpOnly;Domain=SNIP.scm.azurewebsites.net

Date: Thu, 24 May 2018 15:46:33 GMT



<html>

    <head>

        <title> Server Error </title>

    </head>

    <body>

        <font color =\"#aa0000\">

            <h2>Server Error.</h2>

        </font>



        There was an unexpected error in the request processing.

    </body>

</html>

At 3 minutes we've had this.

#!/bin/bash -eo pipefail
if [ "${CIRCLE_BRANCH}" == "development" ]; then
  curl -i -X POST -u SNIP --data-binary @"/tmp/drop.zip" https://SNIP.scm.azurewebsites.net/api/zipdeploy
fi
HTTP/1.1 100 Continue



curl: (56) SSL read: error:00000000:lib(0):func(0):reason(0), errno 104
Exited with code 56

Ultimately deployment ends up taking 10-20 minutes even on the premium tier Kudu hosts.

A few extra notes:

We're going with CircleCI over VSTS because CircleCI is is reporting Lint and Test results in ~40 seconds, VSTS is still checking out code at the point CircleCI finishes. Ultimately it takes nearly 5 minutes.
CircleCI gives us vastly more configuration options. We're using a 4core/8gb host for the test phase, there's no easy way to do this in VSTS
Kudu deployment is extremely slow, 10-20 minutes in most cases.
We're having a nightmare of a time with service principals, hence not using the Azure CLI for deployment

Source

phawxby

All 11 comments

It is by design that any request to App Service will time out after 230 seconds. Please use the async API instead. It lets you poll the API to know when it is done.

Also, please try using the new Run-From-Zip, which should make your deployment much faster. See this announcement for details. Basically, in your Azure App Settings, set WEBSITE_RUN_FROM_ZIP to 1, and then deploy as normal using zipdeploy.

davidebbo on 25 May 2018

Ah, 230 isn't documented anywhere I've seen but I think that's counter to the what is expected. I'll investigate using the async API as an alternative.

Run-From-Zip sounds like exactly what we need, however we're using a Linux host and being experimental I'm nervous about relying on it for a production site. I think we're slowly being pushed to AWS for hosting, Linux and Node feel like second class citizens on the Azure platform.

phawxby on 25 May 2018

I've renamed the issue to better capture what it is about.

davidebbo on 25 May 2018

@davidebbo: I believe I'm running into similar issues when using /api/wardeploy to deploy to Azure using the Linux Tomcat 9.0 (preview) web app. I'm able deploy a .war file from my development box via curl -X POST ... /api/wardeploy (I still get a 502, but the deployment works), but getting a 502 would (probably?) prevent me from deploying reliably in VSTS.

To be fair, the .war file is ~280 MB, so maybe that's a factor?

I noticed that your Run-From-Zip announcement doesn't apply to Linux (yet). Do you have any ideas on how to avoid the 502 in the Linux ecosystem?

weltan on 30 May 2018

@weltan I'm on the WIndows side, so I'll let the Linux experts comment further. @rramachand21, who can look at this?

davidebbo on 30 May 2018

👍1

@rramachand21 should I open an issue on a separate repo, or is here fine? Didn't want to hijack the issue.

weltan on 31 May 2018

hi @rramachand21, just following up here: any word on timeout issues related with the api/wardeploy and api/zipdeploy giving a 502 for timeouts for larger artifacts on Linux?

weltan on 5 Jun 2018

👍1

@rramachand21 @davidebbo
Any update on this topic? I'm struggling for deployment options with my Linux WebApp setup. ZipDeploy seems to do the job but I have similar issues to the above - getting 502's instead of a success message. Would be good to have a solution for this.

BenWalters on 18 Dec 2018

@BenWalters I am no longer working on this project, so I will let @rramachand21 or others comment further.

davidebbo on 18 Dec 2018

👍1

You guys could be suffering from this too.
https://github.com/Azure/azure-cli/issues/8048

phawxby on 18 Dec 2018

@phawxby Looks like it could resolve the issue. We aren't using ZipDeploy directly in the CLI, we've been using a ZipDeploy VSTS marketplace task. So think we'd need to implement the CLI tasks to get this running. Will have a look, thanks for pointing it out!

BenWalters on 18 Dec 2018

Was this page helpful?

0 / 5 - 0 ratings