Aks-engine: DifferentSkuLoadBalancersAndPublicIPAddressNotAllowed using loadBalancerSku: standard

[palmerabollo]

Is this a request for help?:
Yes

Is this an ISSUE or FEATURE REQUEST? (choose one):
ISSUE

What version of aks-engine?:
0.35.1

Kubernetes version:
At least 1.11.8 and 1.11.9

What happened:

We occasionally get an error with code "DifferentSkuLoadBalancersAndPublicIPAddressNotAllowed" when deploying a Kubernetes pod + service (type: LoadBalancer) in a cluster that uses the cluster-autoscaler and vmss with "loadBalancerSku": "standard".

The error details are "Different basic sku and standard sku load balancer or public Ip resources in availability set is not allowed".

In the Azure portal:

• Both the load balancer and the public IP address are shown as SKU=Standard
• The load balancer has the expected public IP address assigned to it.

This is the service definition:

apiVersion: v1
kind: Service
metadata:
  name: traefik
  labels:
    app: traefik
spec:
  externalTrafficPolicy: Local
  selector:
    app: traefik
  ports:
    - port: 80
      name: http
      targetPort: http
    - port: 443
      name: https
      targetPort: https
  type: LoadBalancer

The cluster-autoscaler (1.3.9) is enabled and the pod doesn't fit in the existing nodes, so the cluster-autoscaler has to add a new node to the vmss. This might be closely related to the issue because we are not able to reproduce it if we disable the autoscaler (e.g. min_size = max_size).

This is the service status (EXTERNAL-IP = pending):

kubectl get service traefik
traefik LoadBalancer 10.0.69.227 80:30559/TCP,443:30049/TCP 40m

And the associated logs:

kubectl describe service traefik
Warning LoadBalancerUpdateFailed 9m52s service-controller Error updating load balancer with new hosts map[xxx]: Long running operation terminated with status 'Failed': Code="DifferentSkuLoadBalancersAndPublicIPAddressNotAllowed" Message="Different basic sku and standard sku load balancer or public Ip resources in availability set is not allowed /subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Network/loadBalancers/xxx and /subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Network/loadBalancers/xxx."
Normal EnsuringLoadBalancer 4m1s (x9 over 31m) service-controller Ensuring load balancer
Warning CreatingLoadBalancerFailed 3m29s (x9 over 26m) service-controller Error creating load balancer (will retry): failed to ensure load balancer for service xxx/traefik: Long running operation terminated with status 'Failed': Code="DifferentSkuLoadBalancersAndPublicIPAddressNotAllowed" Message="Different basic sku and standard sku load balancer or public Ip resources in availability set is not allowed /subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Network/loadBalancers/xxx and /subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Network/loadBalancers/xxx"
Warning LoadBalancerUpdateFailed 33s (x4 over 7m19s) service-controller (combined from similar events): Error updating load balancer with new hosts map[xxx]: Long running operation terminated with status 'Failed': Code="DifferentSkuLoadBalancersAndPublicIPAddressNotAllowed" Message="Different basic sku and standard sku load balancer or public Ip resources in availability set is not allowed /subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Network/loadBalancers/xxx and /subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Network/loadBalancers/xxx"

The situation is permanent once we get the error. Kubernetes/Azure is not able to reconcile the status.

What you expected to happen:

The Kubernetes service should end up working.

How to reproduce it (as minimally and precisely as possible):

It is hard to reproduce. You need to create a Kubernetes cluster with the autoscaler enabled. It is easier to reproduce if you set min_size=0 in the autoscaler settings.

[sylr]

I have the same problem with the current setup:

• 3 Availability Zones in EUW
• 1 or 2 VMSS per Availability Zones
• Accelerated Networking enabled
• Using both Standard Load Balancer & Internal Standard Load Balancer

sylvain@ubuntu-1604-dev:~$ kubectl --context=k8s-cluster -n monitoring describe service prometheus-alertmanager-internal-load-balancer
Name:                     prometheus-alertmanager-internal-load-balancer
Namespace:                monitoring
Labels:                   <none>
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"service.beta.kubernetes.io/azure-load-balancer-internal":"true"},"name":"p...
                          service.beta.kubernetes.io/azure-load-balancer-internal: true
Selector:                 app=prometheus,component=alertmanager
Type:                     LoadBalancer
IP:                       10.200.251.239
Port:                     <unset>  80/TCP
TargetPort:               9093/TCP
NodePort:                 <unset>  32027/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type     Reason                      Age                  From                Message
  ----     ------                      ----                 ----                -------
  Normal   EnsuringLoadBalancer        22s (x3 over 3m54s)  service-controller  Ensuring load balancer
  Warning  CreatingLoadBalancerFailed  11s (x3 over 58s)    service-controller  Error creating load balancer (will retry): failed to ensure load balancer for service monitoring/prometheus-alertmanager-internal-load-balancer: Code="DifferentSkuLoadBalancersAndPublicIPAddressNotAllowed" Message="Different basic sku and standard sku load balancer or public Ip resources in availability set is not allowed /subscriptions/xxxxx/resourceGroups/k8s-cluster-RG/providers/Microsoft.Network/loadBalancers/k8s-cluster and /subscriptions/xxxxx/resourceGroups/k8s-cluster-RG/providers/Microsoft.Network/loadBalancers/k8s-cluster-internal."

[palmerabollo]

Thanks @sylr. If you find a workaround or come across a way to make it work, please share.

[khenidak]

Thank you @palmerabollo @sylr
Can you please confirm that
1- you have no tooling that overrides the sku value in azure.json (controller manager)?
2- the LB sku didn't change after the cluster was created?

[palmerabollo]

@khenidak Thanks for your help. Confirmed.

[sylr]

I confirmed that I'm only using Standard SKU for both Load Balancers and IP Public addresses.

[CecileRobertMichon]

@sylr do you also have cluster-autoscaler enabled?

[sylr]

Yes.

On 14 Jun 2019, at 20:43, Cecile Robert-Michon [e-mail] wrote:

@sylr do you also have cluster-autoscaler enabled?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[alexeldeib]

@sylr @palmerabollo i'm trying to repro this, putting my clusters through the autoscaling paces while creating/recreating services. I'm sharing my configuration below, if you see anything I could align more closely to your set up that might help with repro, let me know. I'll keep poking at it.

api model -- 1 VMSS in each of 3 AZ, SLB, MSI, minimal addons.

{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorRelease": "1.13",
      "orchestratorVersion": "1.13.5",
      "kubernetesConfig": {
        "loadBalancerSku": "Standard",
        "excludeMasterFromStandardLB": true,
        "useManagedIdentity": true,
        "addons": [
          {
            "name": "tiller",
            "enabled": false
          },
          {
            "name": "keyvault-flexvolume",
            "enabled": false
          },
          {
            "name": "blobfuse-flexvolume",
            "enabled": false
          }
        ]
      }
    },
    "masterProfile": {
      "count": 3,
      "dnsPrefix": "",
      "vmSize": "Standard_D4_v3",
      "availabilityZones": [
        "1"
      ]
    },
    "agentPoolProfiles": [
      {
        "name": "agentpool1",
        "count": 1,
        "vmSize": "Standard_D2_v2",
        "availabilityProfile": "VirtualMachineScaleSets",
        "storageProfile": "ManagedDisks",
        "availabilityZones": [
          "1"
        ]
      },
      {
        "name": "agentpool2",
        "count": 1,
        "vmSize": "Standard_D2_v2",
        "availabilityProfile": "VirtualMachineScaleSets",
        "storageProfile": "ManagedDisks",
        "availabilityZones": [
          "2"
        ]
      },
      {
        "name": "agentpool3",
        "count": 1,
        "vmSize": "Standard_D2_v2",
        "availabilityProfile": "VirtualMachineScaleSets",
        "storageProfile": "ManagedDisks",
        "availabilityZones": [
          "3"
        ]
      }
    ],
    "linuxProfile": {
      "adminUsername": "azureuser",
      "ssh": {
        "publicKeys": [
          {
            "keyData": "REMOVED"
          }
        ]
      }
    }
  }
}

autoscaling all 3 vmss 0:10

          command:
            - ./cluster-autoscaler
            - --v=3
            - --logtostderr=true
            - --cloud-provider=azure
            - --skip-nodes-with-local-storage=false
            - --nodes=0:10:k8s-agentpool1-21542944-vmss
            - --nodes=0:10:k8s-agentpool2-21542944-vmss
            - --nodes=0:10:k8s-agentpool3-21542944-vmss

[sylr]

@alexeldeib try re-applying the generated ARM template with VMSS instances count increased on several VMSS. The new nodes should have no internet.

[alexeldeib]

Using the above config and temporarily disabling autoscaler (to avoid fighting downscaling automation), I manually updated each agent pool in api model from 1 to 3 nodes, and applied with az group deployment. The new nodes came up successfully with internet access. I've also been running automation to load/deload pods to trigger cluster autoscale, and then cycle creating/deleting services to see if I can trigger some scaling edge case somewhere.

Feel free to ping me on slack for a quicker back and forth, same handle as Github.

[alexeldeib]

ehh...you may be on to something there. Let me take another look.

[alexeldeib]

That’s in addition to the issue mentioned originally in this thread, not in place of, correct?

[sylr]

(We spoke on slack)

[alexeldeib]

@palmerabollo has this occurred again since the original time window? The VMSS network connectivity issue mentioned above definitely exists but is separate. I'm wondering if the DifferentSkuLoadBalancersAndPublicIPAddressNotAllowed issue was a point in time issue with LB backend or something. As far as I understand from @sylr, he hasn't seen this since the original window?

I'm still looking to repro, let me know if you have additional details.

[palmerabollo]

No, it hasn't occurred in the last month at least. Thanks.

[craiglpeters]

If that is the case, then can we close this issue?

[alexeldeib]

I was following with Cecile offline to determine any known issues but we didn’t link it to anything specific. Without any recent repro I think we can either timebox or close?

[palmerabollo]

I'm closing it. I'll reopen it with more details if we manage to reproduce it. Thank you for your time.

[alexeldeib]

If anyone is interested, I found a scenario where this occurs. If you provision VMSS and add a public IP per instance as part of the network profile, the IP will cause this error when you try to add the VMSS to SLB.

[alexeldeib]

More comments on a long closed issue, but I got to the root of this issue with Standard LB team. There was a bug in their behavior with VMSS. ~It also caused the above^ I believe~ (turns out Standard Public IPs are not supported per instance on VMSS).

AKS-engine worked around this long ago, but if you happen to be coming to this issue from elsewhere, VMSS + SLB teams are rolling out a fix this week.

[brichins]

@alexeldeib - Can you elaborate how aks-engine worked around the problem? I think I may be fighting the same issue you encountered.

[alexeldeib]

I'm glad someone found my comment useful 😄

Workaround is to provision LB before VMSS such that at time of creation VMSS always will have SLB specified.

The bug occurs when VMSS is created without any networking by default (no per-instance IPs, no SLB, no basic LB). In this scenario, it is possible to get the DifferentSku error (typically with the same resource ID, this is the smoking gun).

The most reliable repro I have found is creating VMSS with 10 instances and no networking and adding the VM instances once by one to the backend pool. Usually you can get a few instances in the backend pool successfully, but I've never been able to successfully add >= 10 instances -- by the 5th, I typically am already getting the DifferentSku error.

Also, if you hit weird behavior with outbound connectivity, ensure you define an outbound rule for that backend pool.

[alexeldeib]

whoops, didn't mean to reopen.

[brichins]

Thank you for the extra info. Very useful.