Aks-engine: Standard LB-enabled cluster fails to scale out

[jackfrancis]

Seeing failures like this testing Standard LB-enabled cluster during scale out:

INFO[0000] validating...                                
INFO[0004] Name suffix: 69830483                        
time="2019-05-01T17:18:18Z" level=info msg="Found no resources with type Microsoft.Network/routeTables in the template." source="scaling command line"
INFO[0004] Starting ARM Deployment (kubernetes-canadaeast-64618-1930930191). This will take some time... 
Error: Code="DeploymentFailed" Message="At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details." Details=[{"code":"Conflict","message":"{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"VMExtensionProvisioningError\",\r\n        \"message\": \"VM has reported a failure when processing extension 'vmssCSE'. Error message: \\\"Enable failed: failed to execute command: command terminated with exit status=50\\n[stdout]\\n\\n[stderr]\\n\\\".\"\r\n      }\r\n    ]\r\n  }\r\n}"}]
INFO[0312] Finished ARM Deployment (kubernetes-canadaeast-64618-1930930191). Error: Code="DeploymentFailed" Message="At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details." Details=[{"code":"Conflict","message":"{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"VMExtensionProvisioningError\",\r\n        \"message\": \"VM has reported a failure when processing extension 'vmssCSE'. Error message: \\\"Enable failed: failed to execute command: command terminated with exit status=50\\n[stdout]\\n\\n[stderr]\\n\\\".\"\r\n      }\r\n    ]\r\n  }\r\n}"}] 

[mo-saeed]

https://github.com/Azure/acs-engine/issues/4328#event-2208016351

[palmerabollo]

I'm also facing this issue with aks-engine 0.32.x and kubernetes 1.11.8. Cluster autoscaler v1.3.9
Also using "loadBalancerSku": "standard" and "singlePlacementGroup": false

I've added a comment to the issue in acs-engine, but I'm posting it here as well since I think the other repo is deprecated. This is the event I see in a kubernetes service that expects a load balancer to be created:

Warning CreatingLoadBalancerFailed 4m41s service-controller Error creating load balancer (will retry): failed to ensure load balancer for service mynamespace/traefik: Long running operation terminated with status 'Failed': Code="VMExtensionProvisioningError" Message="VM has reported a failure when processing extension 'vmssCSE'. Error message: "Enable failed: failed to execute command: command terminated with exit status=50\n[stdout]\n\n[stderr]\n"."

Any known workarounds?

[sylr]

My aks-engine made cluster deployed with version 1.13.5 was working fine:

• Multiple Zones
• Multiple VMSS across zones
• Standard load balancers
• Scale up/down OK

Then I tried the process upgrade to upgrade it in v1.13.6 and since new instances of VMSS do not have internet access and therefore can't execute the CSE.

I've tried deleting all VMSS and recreating them. The instances created along the VMSS are fine but once you try to scale out the new instances do not have internet.

[ritazh]

Based on failures we have observed and per @feiskyer, this seems to be the same behavior as this issue: https://github.com/kubernetes/cloud-provider-azure/issues/88
Next we will create a SLB with outbound rules, then add nodes to SLB backends.

[mo-saeed]

Hi @ritazh, Thanks for the update. As this is very important for us, do you have any estimation when this can be ready and implemented in aks-engine ?

[alexeldeib]

@ritazh I found my way back to this issue from some of the related VMSS/SLB issues. This is a bit of a catch-22, no? AKS engine needs to attach the VMSS to the backend pool, but the LB won't exist until after the cluster is provisioned because the LB is managed by cloud provider? Or can AKS engine provision the LB, then let Kubernetes take it over?

[ritazh]

@mo-saeed Working on this now.

@alexeldeib

can AKS engine provision the LB, then let Kubernetes take it over

that's what im doing now

[mo-saeed]

@ritazh is the fix included in any current aks-engine releases ?

[mboersma]

@mo-saeed not yet, but it will be in v0.39.0, coming soon.

[mo-saeed]

Thanks @mboersma

[pierluigilenoci]

I have the same issue with v0.39.1