Azure-rest-api-specs: App Service Environment: unable to provision after 30 hours

[tombuildsstuff]

👋

We've been trying to provision App Service Environments using the API - either for a ASEv1 or an ASEv2; however we've been having issues where either version isn't provisioned after 30 hours (which is the documented maximum scaling time for an App Service Environment).

During this development work on our side I've noticed that the Azure Portal submits an ARM Template to provision an App Service Environment, and that the Azure CLI omits support for App Service Environments altogether - as such I'm wondering if there's any examples of successfully using the API to provision an App Service Environment (either a v1 or v2)? It's worth noting provisioning using an ARM Template provisions successfully, and porting the fields in the ARM Template over to the API call fails.

From our side we're using the Azure SDK for Go and API version 2016-09-01 (here's the code if you're interested) - whilst we could investigate switching to the 2018-02-01 API version, I'm unconvinced that'd help fix this since I can find no other users of this API, as such would it be possible to confirm if this is a known issue in the App Service Environment API?

Thanks!

[tombuildsstuff]

@jhendrixMSFT would it be possible to escalate this to the Service Team? This appears to be a wider issue with App Service Environments, since users report the API is considerably slower than an ARM Template: https://github.com/terraform-providers/terraform-provider-azurerm/issues/2616

[jhendrixMSFT]

@tombuildsstuff I've started an email thread with the service team asking them to comment on this issue, sorry for the delay.

[TraGicCode]

Any updates on this?

[jhendrixMSFT]

I've pinged the service team for an update, sorry this is taking so long.

[TraGicCode]

This is affecting my team as well as we are attempting to automate app service environments via ARM templates 👎

[jonarmstrong]

I'm being blocked by issues with deploying App Service Environments via ARM templates as well. I had a successful deployment of one on 2/6/2019 and I've had 5 deployments timeout in the past 48 hours at the 5 hours mark. The environment is left in a 'Provisioning' state after 5 hours and I'm forced to delete the resource to try the deployment again.

If anyone from Microsoft needs examples templates of what I'm trying to accomplish, please reach out. I desperately need this resolved to help my move my payments application into production.

[snerting]

What is the status on this issue? Very strange Microsoft don't focus on fixing this since this is an Enterprise thing. By not fixing it, Microsoft will force companies looking for something else. Maybe to use ARM instead, and leave Terraform, or keep Terraform, use AWS instead who seams to have more focus in Terraform. It's terrible to use lot of time learning Terraform, and then it's can't be used to create ASE.

[Tridy]

@snerting there is currently a workaround for that to use an ARM template inside Terraform. look at this post and see if this will help you for now:
https://stackoverflow.com/questions/49040510/provision-app-service-environment-with-terraform

[tombuildsstuff]

@jhendrixMSFT 👋 could we re-ping the service team here?

[jhendrixMSFT]

@tombuildsstuff just pinged, sorry it's taking so long.

[tombuildsstuff]

@jhendrixMSFT no worries - thanks for that :)

[Lachlan-White]

@jhendrixMSFT / @tombuildsstuff any update on this one? Would it help to push from a customer side of Microsoft as well to the Product teams?

[jhendrixMSFT]

Sorry for the delay I'm following up with the service team (again).

[jhendrixMSFT]

OK if somebody could provide the name of their failed App Service Environment deployment and subscription ID I've got somebody from the service team that can help investigate. If you prefer you can send me this info directly instead of posting it here, my email address is my github name minus the MSFT at microsoft.com

[Lachlan-White]

@tombuildsstuff Is this something you can do, as i haven't run a build with the test code you are referring to with the 30 hour time out?

[miwebst]

Without this information we can't make any progress to help you identify the deployment issue. Going to resolve this for now as it has been 10 days with no response.

[Lachlan-White]

@naveedaz Considering there has been months without a response from Microsoft i would advise you leave this issue open a bit longer and give @tombuildsstuff a chance to respond!

[tombuildsstuff]

@miwebst @naveedaz sorry, I'd missed the ping - since this API is still broken can we get this bug re-opened?

We last provisioned one in March named acctestase-6712679629713806056, and before that in December (App Service Environment "acctestase-453917183117101087" (Resource Group "acctestrg-453917183117101087"). Whilst I don't have one to hand it should be possible to reproduce that by sending the payload shown here to the API: https://github.com/terraform-providers/terraform-provider-azurerm/blob/app-service-environment/azurerm/resource_arm_app_service_environment.go#L175-L213

Thanks!

[miwebst]

@tombuildsstuff ,

The reason I was asking for your ASE details was because this is most likely not an API issue but an issue with how the virtual network is configured. If the virtual network (vnet) is configured incorrectly, the ASE cannot connect to its dependencies and thus may spin until we delete it. If you have a recent ASE deployment (in the past few weeks), I can verify this.

Please see this page for more information on networking problems: https://docs.microsoft.com/en-us/azure/app-service/environment/network-info

My suggestion to you is to test by deploying into a bare bones virtual network (no Network Security Groups or route tables) and verify that it is successful. I usually use a PUT and api-version = 2019-01-01 and a payload that looks something like:

{
"location": "West US 2",
"kind" : "Asev2",
"properties": {
"virtualNetwork": {
"id": "/subscriptions/{vnet subscription}/resourceGroups/{vnet resource group}/providers/Microsoft.Network/virtualNetworks/{vnet name}/subnets/{vnet subnet}"
}
}
}

After this succeeds, you can continue to add fields at your discretion. (P.S. for ASEv2 you don't need worker pools) Just now, I have verified that this api works as expected.

[tombuildsstuff]

@miwebst

If the virtual network (vnet) is configured incorrectly, the ASE cannot connect to its dependencies and thus may spin until we delete it. If you have a recent ASE deployment (in the past few weeks), I can verify this.

This is a brand new virtual network, with a brand new subnet spun up especially for this purpose - but in either case that's a bug in the API (since the API should return the error during polling, as every other Azure API does, rather than spinning indefinitely).

In addition, given the payloads fails for both an ASEv1 and an ASEv2 (which come from creating these by hand in the portal and using those same payloads via the API) - the API should be updated to return the actual error here, rather than having consumers try and reverse-engineer how the API behaves. If it's helpful we can put that together, but the payloads available here and should be transferable: https://github.com/terraform-providers/terraform-provider-azurerm/blob/app-service-environment/azurerm/resource_arm_app_service_environment.go#L175-L213

As such, given the App Service Environment API is (still) broken in both of these manners, can we get this issue re-opened to track this?

cc @jhendrixMSFT / @salameer

[miwebst]

@tombuildsstuff ,

Sorry if I was not clear before, the configuration that can break the ASE will come from the Network Security Group (NSG) or the Routes applied to the subnet. It doesn't matter if the subnet is new or old if a bad NSG is applied to it. We do as much validation as we can on whether the ASE will be able to stabilize in the subnet, but we are missing some information and thus cannot validate all scenarios. We have a feature coming that should help address this, and we will be able to throw errors during the api call rather than just spinning and timing out.

As for your second comment, I am still under the impression that the issue is not specific to the API. If the call succeeds and the ASE simply never provisions, then whether it is ASEv1 or ASEv2, it is most likely either a networking configuration issue or a regression on our side. If it is the later, it is most certainly fixed as we see successes for other customers.

For your specific scenario, if you are creating an ASE into a subnet with no NSG/routes and it is failing to come up despite a successful response, please redeploy the ASE and provide the subscription and name of the ASE to this thread.

To sum up, we are working on features to help move more of this validation to the API level rather than the runtime level, however the issue present is most likely a configuration issue and thus if we'd like to get to the bottom of this, you should redeploy an ASEv2 and supply this thread with the ASE name and subscription.

[tombuildsstuff]

@miwebst

@logachev has taken a look into this and has added some more comments on how both the API and the Swagger have issues here, which appears to be the root cause of this issue: https://github.com/terraform-providers/terraform-provider-azurerm/issues/438#issuecomment-493725235

Since both the API and the Swagger are broken, can we please re-open this Github issue?

Thanks!

[stefangordon]

I can also confirm this is broken and is easily reproducible (and a serious blocker). If we think @logachev has a more detailed version of the issue then we should clarify that we are closing this one in favor of that one, and update progress there.

[miwebst]

Based on what I saw on the feature request thread. It is a very different issue than initially described in this thread. As such, we should leave this as closed and use the other issue to track the issue.