Aks-engine: VMSS last model warning with aks 0.38.0 and 1.14.3

[aasier]

Describe the bug
After the upgrade from aks-e 0.35.4 to 0.38.0 the vmss pool is having all the time the warning in the last model of the image.

Steps To Reproduce
Updating 1.14.1 with aks-e 0.35.4 to 1.14.3 (aks-e 0.38.0)
master distro aks-ubuntu-16.04
agents distro (vmss) ubuntu

Expected behavior
Not having the Latest model in the VMSS Instances.
AKS Engine version
0.38.0
Kubernetes version
1.14.4
Additional context

[welcome[bot]]

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it.

[mo-saeed]

This is due to kubernetes change not aks-engine related, https://github.com/kubernetes/kubernetes/issues/71594#issuecomment-494748086

[aasier]

@mo-saeed Thanks for your reply! ;)
I'm not sure where is the problem, but affter the upgrade from 0.35.4 (1.14.1) to 0.38.0 (1.14.3) We are having strange behaviors into the vmss pools with the LoadBalancers (We are losing VMs in the LoadBalancer backend pool every time, the only way to recover the LoadBalancer is rebooting the instances in the VMSS)
Should I report this into the thread that you mentioned?

Thanks!

[mo-saeed]

Oh that's bad, Are you using Basic or standard LB ?

[aasier]

Basic LB

[mo-saeed]

@feiskyer May be you can help here, that seems to be related to this change https://github.com/kubernetes/kubernetes/pull/76992

[aasier]

[aasier]

@mo-saeed here you have more info. If you need more details please, let me know asap ;)
thanks for your help!

[mo-saeed]

@aasier May be you need to give it sometime ?
Also can you try to describe the k8s load-balancer service to see if any errors ? kubectl describe service .....
Also try to check the kubelet logs in for the not-joining nodes ? journalctl -u kubelet -f

[aasier]

Name: nginx-ingress-internal-svc
Namespace: nginx-ingress
Labels: app=nginx-ingress-controller-internal
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"nginx.ingress.kubernetes.io/vts-filter-key":"$server_name $uri","service.b...
nginx.ingress.kubernetes.io/vts-filter-key: $server_name $uri
service.beta.kubernetes.io/azure-load-balancer-internal: true
Selector: app=nginx-ingress-controller-internal
Type: LoadBalancer
IP: 10.0.199.157
IP: 10.167.68.230
LoadBalancer Ingress: 10.167.68.230
Port: http 80/TCP
TargetPort: 80/TCP
NodePort: http 32399/TCP
Endpoints: 10.244.13.32:80,10.244.13.6:80
Port: https 443/TCP
TargetPort: 443/TCP
NodePort: https 32579/TCP
Endpoints: 10.244.13.32:443,10.244.13.6:443
Session Affinity: None
External Traffic Policy: Local
HealthCheck NodePort: 30233
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning LoadBalancerUpdateFailed 51m (x238 over 7h32m) service-controller Error updating load balancer with new hosts map[k8s-master-20148447-0:{} k8s-pool1-20148447-vmss00002y:{} k8s-pool1-20148447-vmss00003i:{} k8s-pool1-20148447-vmss00003j:{} k8s-pool1-20148447-vmss00003l:{} k8s-pool1-20148447-vmss00003m:{} k8s-pool1-20148447-vmss00003o:{} k8s-pool1-20148447-vmss00003q:{} k8s-pool1-20148447-vmss00004f:{} k8s-pool1-20148447-vmss00004h:{} k8s-pool1-20148447-vmss00004j:{}]: azure - cloud provider rate limited(read) for operation:LBList
Normal EnsuringLoadBalancer 41m (x124 over 10h) service-controller Ensuring load balancer
Warning LoadBalancerUpdateFailed 26m (x4 over 31m) service-controller Error updating load balancer with new hosts map[k8s-master-20148447-0:{} k8s-pool1-20148447-vmss00004m:{} k8s-pool1-20148447-vmss00004n:{} k8s-pool1-20148447-vmss00004o:{} k8s-pool1-20148447-vmss00004p:{} k8s-pool1-20148447-vmss00004q:{} k8s-pool1-20148447-vmss00004r:{} k8s-pool1-20148447-vmss00004s:{} k8s-pool1-20148447-vmss00004t:{} k8s-pool1-20148447-vmss00004u:{} k8s-pool1-20148447-vmss00004v:{}]: azure - cloud provider rate limited(read) for operation:LBList
Warning LoadBalancerUpdateFailed 21m (x10 over 7h54m) service-controller (combined from similar events): Error updating load balancer with new hosts map[k8s-master-20148447-0:{} k8s-pool1-20148447-vmss00004m:{} k8s-pool1-20148447-vmss00004n:{} k8s-pool1-20148447-vmss00004o:{} k8s-pool1-20148447-vmss00004p:{} k8s-pool1-20148447-vmss00004q:{} k8s-pool1-20148447-vmss00004r:{} k8s-pool1-20148447-vmss00004t:{} k8s-pool1-20148447-vmss00004u:{} k8s-pool1-20148447-vmss00004v:{}]: azure - cloud provider rate limited(read) for operation:LBList
Warning ListLoadBalancers 88s (x834 over 10h) azure-cloud-provider azure - cloud provider rate limited(read) for operation:LBList
Warning LoadBalancerUpdateFailed 88s (x11 over 18m) service-controller Error updating load balancer with new hosts map[k8s-master-20148447-0:{} k8s-pool1-20148447-vmss00004m:{} k8s-pool1-20148447-vmss00004n:{} k8s-pool1-20148447-vmss00004o:{} k8s-pool1-20148447-vmss00004q:{} k8s-pool1-20148447-vmss00004r:{} k8s-pool1-20148447-vmss00004t:{} k8s-pool1-20148447-vmss00004u:{} k8s-pool1-20148447-vmss00004v:{}]: azure - cloud provider rate limited(read) for operation:LBList

[aasier]

On Monday I will provide the output of "journalctl -u kubelet -f" ... I'm not connected to my Corp Network :(

[mo-saeed]

@aasier No need i think the issue is known no from the describe command output, have a look here https://github.com/kubernetes/kubernetes/issues/64759#issuecomment-395909117

[feiskyer]

This is expected after kubernetes fixes https://github.com/kubernetes/kubernetes/issues/71594. Because Kubernetes now updates the network configurations per instance, the latest model of each instance is different now from VMSS.

[CecileRobertMichon]

@feiskyer are you saying this is by design and we should close this issue? Are there any future plans to solve this?

[aasier]

@feiskyer @mo-saeed @CecileRobertMichon something strange is happening with this version 1.14.3 with the Azure Load Balancers,
We disabled this morning the in the azure.json file the flag cloudProviderRateLimit to false, and after having some issues with the loadbalancers and the nginx controllers we decided to recreate all the services, and now we are without PublicIps ( failed to ensure host in pool: "not a vmss instance" )

aperez@alinux:~$ kubectl describe service -n nginx-ingress nginx-ingress-external-svc
Name: nginx-ingress-external-svc
Namespace: nginx-ingress
Labels: app=nginx-ingress-controller-external
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"nginx.ingress.kubernetes.io/vts-filter-key":"$server_name $uri","service.b...
nginx.ingress.kubernetes.io/vts-filter-key: $server_name $uri
service.beta.kubernetes.io/azure-dns-label-name: xxxxxxxxxx
service.beta.kubernetes.io/azure-load-balancer-internal: false
Selector: app=nginx-ingress-controller-external
Type: LoadBalancer
IP: 10.0.193.215
Port: http 80/TCP
TargetPort: 80/TCP
NodePort: http 31181/TCP
Endpoints: 10.244.19.14:80,10.244.20.14:80
Port: https 443/TCP
TargetPort: 443/TCP
NodePort: https 31605/TCP
Endpoints: 10.244.19.14:443,10.244.20.14:443
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringLoadBalancer 2m59s service-controller Ensuring load balancer
Warning CreatingLoadBalancerFailed 2m23s service-controller Error creating load balancer (will retry): failed to ensure load balancer for service nginx-ingress/nginx-ingress-external-svc: EnsureHostInPool(nginx-ingress/nginx-ingress-external-svc): backendPoolID(/subscriptions/904ad912-63ba-xxxxx-xxxxxx-xxxxxxxxxx/resourceGroups/x-xx-k8s/providers/Microsoft.Network/loadBalancers/x-xx-k8s/backendAddressPools/x-xx-k8s) - failed to ensure host in pool: "not a vmss instance"

If I go to the LB the PublicIps are present into the FrontEnd IP Configuration:

But they are not accessible from outside.

[feiskyer]

We disabled this morning the in the azure.json file the flag cloudProviderRateLimit to false, and after having some issues with the loadbalancers and the nginx controllers we decided to recreate all the services, and now we are without PublicIps ( failed to ensure host in pool: "not a vmss instance" )

@aasier What changes have you made before seeing this error? And, could you share the logs in kube-controller-manager?

[aasier]

@feiskyer , Sorry, I was blocking all the Development teams and after my research, I did a downgrade with aks-engine 0.38.0 to 1.14.1, and now it works!

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal UpdatedLoadBalancer 11m (x5 over 79m) service-controller Updated load balancer with new hosts

I was digging a bit, I´m not an expert on this area, and I found changes here:
https://github.com/kubernetes/kubernetes/commits/master/staging/src/k8s.io/legacy-cloud-providers/azure/azure_vmss.go

Thanks,
Asier

[aasier]

@CecileRobertMichon why did you close this bug?
IMO there is a problem into aks 0.38.0 and k8s 1.14.3 with the VMSS and the LoadBalancers.
@palma21 could you please help here?

Thanks,
Asier

[CecileRobertMichon]

@aasier sorry, read too fast

[mo-saeed]

@aasier I think could be a good idea to create a test cluster and try to reproduce this issue there then provide here the clear steps you did to reproduce it, like the aks-engine cluster-definition file and the commands you used. I think that will help a lot to investigate.

[feiskyer]

We disabled this morning the in the azure.json file the flag cloudProviderRateLimit to false, and after having some issues with the loadbalancers and the nginx controllers we decided to recreate all the services, and now we are without PublicIps ( failed to ensure host in pool: "not a vmss instance" )

@aasier Are there both VMAS and VMSS nodes within the same cluster?

@aasier I think could be a good idea to create a test cluster and try to reproduce this issue there then provide here the clear steps you did to reproduce it, like the aks-engine cluster-definition file and the commands you used. I think that will help a lot to investigate.

+1 on detailed steps or kube-controller-manager logs.

By the way, after https://github.com/kubernetes/kubernetes/issues/71594, VirtualMachineScaleSetsClient.UpdateInstances() won't be invoked since the LB backend updates per instance. So the error message in https://github.com/Azure/aks-engine/issues/1615#issuecomment-511779528 won't happen on 1.14.3.

[aasier]

@aasier Are there both VMAS and VMSS nodes within the same cluster?

@feiskyer , we are deploying VMAS for Masters and VMSS for agents

[aasier]

@feiskyer @mo-saeed I found the behaviour (IMO wrong behaviour) in 1.14.3 with VMSS and the LB.

Here you have my steps (based on our current configurations / versions that we are using):

Initial installation of the k8s cluster: aks-engine v0.33.0 / k8s v 1.13.4 (deployed using az commands)
Deployment of nginx-controller v0.24.1 + k8s service type LoadBalancer
Deployment of echoserver app using the ingress. (everything works as we expected).

We checked the VMSS, the LB are everything looks good.

Upgrade 1:

• aks-engine 0.35.4
• k8s 1.14.1

We checked the VMSS, the LB are everything looks good. The app is still accessible through the nginx-controller and the LB.

Upgrade 2:

• aks-engine 0.38.0
• k8s 1.14.3

Initially, everything looks ok, but if you go to the instances in the VMSS pool you will see that you have a warning icon in the "last model".

The app is still available, but if you press "update" in the VMSS instances you will lose the connectivity between the LB and the VMSS Instances, and later the "last model" column is with all the instances in "yes".
After this, the App is not accessible anymore.

The only way that I found to recover from this status is:

1. delete the service that nginx-ingress is using k8s with the LB
2. Re-Deploy the nginx-ingress solution to recreate the service to recover the access to the app from outside the cluster.
3. If you go to the VMSS Instances, the Last Model it will appear again the warning icon in the last model column.

Here you have the logs:

`kube-controller-manager-k8s-master-99408755-0
kube-controller-manager-k8s-master-99408755-1
kube-controller-manager-k8s-master-99408755-2

=== CREATION OF THE SERVICE

[kube-controller-manager-k8s-master-99408755-0] I0719 08:08:03.168044 1 service_controller.go:300] Ensuring LB for service nginx-ingress/nginx-ingress-external-svc
[kube-controller-manager-k8s-master-99408755-0] I0719 08:08:03.168085 1 azure_loadbalancer.go:593] reconcileLoadBalancer for service(nginx-ingress/nginx-ingress-external-svc) - wantLb(true): started
[kube-controller-manager-k8s-master-99408755-0] I0719 08:08:03.168358 1 event.go:209] Event(v1.ObjectReference{Kind:"Service", Namespace:"nginx-ingress", Name:"nginx-ingress-external-svc", UID:"54d331d2-a9fc-11e9-8060-000d3a2d8629", APIVersion:"v1", ResourceVersion:"207778", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
[kube-controller-manager-k8s-master-99408755-0] I0719 08:08:03.314170 1 azure_backoff.go:257] LoadBalancerClient.List(xxxxxx) - backoff: success
[kube-controller-manager-k8s-master-99408755-0] I0719 08:08:03.314200 1 azure_loadbalancer.go:302] selectLoadBalancer for service (nginx-ingress/nginx-ingress-external-svc): isInternal(false) - start
[kube-controller-manager-k8s-master-99408755-0] I0719 08:08:03.314209 1 azure_loadbalancer.go:308] selectLoadBalancer: cluster(xxxxxx) service(nginx-ingress/nginx-ingress-external-svc) isInternal(false) - vmSetNames [k8s-standardpool-99408755-vmss]
[kube-controller-manager-k8s-master-99408755-0] I0719 08:08:03.314247 1 azure_loadbalancer.go:600] reconcileLoadBalancer for service(nginx-ingress/nginx-ingress-external-svc): lb(xxxxxx) wantLb(true) resolved load balancer name
[kube-controller-manager-k8s-master-99408755-0] I0719 08:08:03.342751 1 azure_wrap.go:134] Public IP "xxxxxx-a54d331d2a9fc11e98060000d3a2d862" not found with message: "network.PublicIPAddressesClient#Get: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code=\"ResourceNotFound\" Message=\"The Resource 'Microsoft.Network/publicIPAddresses/xxxxxx-a54d331d2a9fc11e98060000d3a2d862' under resource group 'xxxxxx' was not found.\""
[kube-controller-manager-k8s-master-99408755-0] I0719 08:08:03.342787 1 azure_loadbalancer.go:496] ensurePublicIPExists for service(nginx-ingress/nginx-ingress-external-svc): pip(xxxxxx-a54d331d2a9fc11e98060000d3a2d862) - creating
[kube-controller-manager-k8s-master-99408755-0] I0719 08:08:08.225030 1 azure_loadbalancer.go:914] reconcileLoadBalancerRule lb name (xxxxxx) rule name (a54d331d2a9fc11e98060000d3a2d862-TCP-80)
[kube-controller-manager-k8s-master-99408755-0] I0719 08:08:08.225069 1 azure_loadbalancer.go:914] reconcileLoadBalancerRule lb name (xxxxxx) rule name (a54d331d2a9fc11e98060000d3a2d862-TCP-443)
[kube-controller-manager-k8s-master-99408755-0] I0719 08:08:08.225091 1 azure_loadbalancer.go:849] reconcileLoadBalancer: reconcileLoadBalancer for service(nginx-ingress/nginx-ingress-external-svc): lb(xxxxxx) - updating
[kube-controller-manager-k8s-master-99408755-0] I0719 08:08:09.204564 1 azure_vmss.go:716] EnsureHostInPool begins to update vmssVM(k8s-standardpool-99408755-vmss000001) with new backendPoolID /subscriptions/xxxx-xxxxx-xxxxxx/resourceGroups/xxxxxx/providers/Microsoft.Network/loadBalancers/xxxxxx/backendAddressPools/xxxxxx
[kube-controller-manager-k8s-master-99408755-0] I0719 08:08:09.205484 1 azure_vmss.go:716] EnsureHostInPool begins to update vmssVM(k8s-standardpool-99408755-vmss000002) with new backendPoolID /subscriptions/xxxx-xxxxx-xxxxxx/resourceGroups/xxxxxx/providers/Microsoft.Network/loadBalancers/xxxxxx/backendAddressPools/xxxxxx
[kube-controller-manager-k8s-master-99408755-0] I0719 08:08:09.216485 1 azure_vmss.go:716] EnsureHostInPool begins to update vmssVM(k8s-standardpool-99408755-vmss000000) with new backendPoolID /subscriptions/xxxx-xxxxx-xxxxxx/resourceGroups/xxxxxx/providers/Microsoft.Network/loadBalancers/xxxxxx/backendAddressPools/xxxxxx
[kube-controller-manager-k8s-master-99408755-0] I0719 08:09:29.889098 1 azure_loadbalancer.go:880] reconcileLoadBalancer for service(nginx-ingress/nginx-ingress-external-svc): lb(xxxxxx) finished
[kube-controller-manager-k8s-master-99408755-0] I0719 08:09:29.938733 1 azure_loadbalancer.go:394] getServiceLoadBalancerStatus gets ingress IP "52.157.159.189" from frontendIPConfiguration "a54d331d2a9fc11e98060000d3a2d862" for service "nginx-ingress/nginx-ingress-external-svc"
[kube-controller-manager-k8s-master-99408755-0] I0719 08:09:29.938767 1 azure_loadbalancer.go:139] EnsureLoadBalancer: reconciling security group for service "nginx-ingress/nginx-ingress-external-svc" with IP "52.157.159.189", wantLb = true
[kube-controller-manager-k8s-master-99408755-0] I0719 08:09:29.986705 1 azure_loadbalancer.go:1177] reconcileSecurityGroup for service(nginx-ingress/nginx-ingress-external-svc): sg(k8s-master-99408755-nsg) - updating
[kube-controller-manager-k8s-master-99408755-0] I0719 08:09:40.332847 1 azure_loadbalancer.go:593] reconcileLoadBalancer for service(nginx-ingress/nginx-ingress-external-svc) - wantLb(false): started
[kube-controller-manager-k8s-master-99408755-0] I0719 08:09:40.433242 1 azure_backoff.go:257] LoadBalancerClient.List(xxxxxx) - backoff: success
[kube-controller-manager-k8s-master-99408755-0] I0719 08:09:40.433275 1 azure_loadbalancer.go:600] reconcileLoadBalancer for service(nginx-ingress/nginx-ingress-external-svc): lb(xxxxxx-internal) wantLb(false) resolved load balancer name
[kube-controller-manager-k8s-master-99408755-0] I0719 08:09:40.433301 1 azure_loadbalancer.go:880] reconcileLoadBalancer for service(nginx-ingress/nginx-ingress-external-svc): lb(xxxxxx-internal) finished
[kube-controller-manager-k8s-master-99408755-0] I0719 08:09:40.504858 1 azure_backoff.go:303] PublicIPAddressesClient.List(xxxxxx) - backoff: success
[kube-controller-manager-k8s-master-99408755-0] I0719 08:09:40.573897 1 azure_backoff.go:303] PublicIPAddressesClient.List(xxxxxx) - backoff: success
[kube-controller-manager-k8s-master-99408755-0] I0719 08:09:40.618551 1 event.go:209] Event(v1.ObjectReference{Kind:"Service", Namespace:"nginx-ingress", Name:"nginx-ingress-external-svc", UID:"54d331d2-a9fc-11e9-8060-000d3a2d8629", APIVersion:"v1", ResourceVersion:"207778", FieldPath:""}): type: 'Normal' reason: 'EnsuredLoadBalancer' Ensured load balancer
[kube-controller-manager-k8s-master-99408755-0] W0719 08:16:44.155417 1 reflector.go:289] k8s.io/client-go/informers/factory.go:133: watch of *v1beta1.Event ended with: The resourceVersion for the provided watch is too old.
[kube-controller-manager-k8s-master-99408755-0] W0719 08:26:25.176431 1 reflector.go:289] k8s.io/client-go/informers/factory.go:133: watch of *v1beta1.Event ended with: The resourceVersion for the provided watch is too old.

=== DELETE OF THE SERVICE

[kube-controller-manager-k8s-master-99408755-0] W0719 08:50:37.356140 1 reflector.go:289] k8s.io/client-go/informers/factory.go:133: watch of *v1beta1.Event ended with: The resourceVersion for the provided watch is too old.
[kube-controller-manager-k8s-master-99408755-0] I0719 09:00:50.475151 1 service_controller.go:729] Service has been deleted nginx-ingress/nginx-ingress-external-svc. Attempting to cleanup load balancer resources
[kube-controller-manager-k8s-master-99408755-0] I0719 09:00:50.477213 1 event.go:209] Event(v1.ObjectReference{Kind:"Service", Namespace:"nginx-ingress", Name:"nginx-ingress-external-svc", UID:"54d331d2-a9fc-11e9-8060-000d3a2d8629", APIVersion:"v1", ResourceVersion:"207778", FieldPath:""}): type: 'Normal' reason: 'DeletingLoadBalancer' Deleting load balancer
[kube-controller-manager-k8s-master-99408755-0] I0719 09:00:50.664032 1 azure_backoff.go:257] LoadBalancerClient.List(xxxxxx) - backoff: success
[kube-controller-manager-k8s-master-99408755-0] I0719 09:00:50.744134 1 azure_loadbalancer.go:394] getServiceLoadBalancerStatus gets ingress IP "52.157.159.189" from frontendIPConfiguration "a54d331d2a9fc11e98060000d3a2d862" for service "nginx-ingress/nginx-ingress-external-svc"
[kube-controller-manager-k8s-master-99408755-0] I0719 09:00:50.744156 1 azure_loadbalancer.go:193] EnsureLoadBalancerDeleted: reconciling security group for service "nginx-ingress/nginx-ingress-external-svc" with IP "52.157.159.189", wantLb = false
[kube-controller-manager-k8s-master-99408755-0] I0719 09:00:50.865215 1 azure_loadbalancer.go:1177] reconcileSecurityGroup for service(nginx-ingress/nginx-ingress-external-svc): sg(k8s-master-99408755-nsg) - updating
[kube-controller-manager-k8s-master-99408755-0] I0719 09:01:01.173398 1 azure_loadbalancer.go:593] reconcileLoadBalancer for service(nginx-ingress/nginx-ingress-external-svc) - wantLb(false): started
[kube-controller-manager-k8s-master-99408755-0] I0719 09:01:01.261131 1 azure_backoff.go:257] LoadBalancerClient.List(xxxxxx) - backoff: success
[kube-controller-manager-k8s-master-99408755-0] I0719 09:01:01.310237 1 azure_loadbalancer.go:394] getServiceLoadBalancerStatus gets ingress IP "52.157.159.189" from frontendIPConfiguration "a54d331d2a9fc11e98060000d3a2d862" for service "nginx-ingress/nginx-ingress-external-svc"
[kube-controller-manager-k8s-master-99408755-0] I0719 09:01:01.310264 1 azure_loadbalancer.go:600] reconcileLoadBalancer for service(nginx-ingress/nginx-ingress-external-svc): lb(xxxxxx) wantLb(false) resolved load balancer name
[kube-controller-manager-k8s-master-99408755-0] I0719 09:01:01.310402 1 azure_loadbalancer.go:652] reconcileLoadBalancer for service (nginx-ingress/nginx-ingress-external-svc)(false): lb frontendconfig(a54d331d2a9fc11e98060000d3a2d862) - dropping
[kube-controller-manager-k8s-master-99408755-0] I0719 09:01:01.310443 1 azure_loadbalancer.go:797] reconcileLoadBalancer for service (nginx-ingress/nginx-ingress-external-svc)(false): lb rule(a54d331d2a9fc11e98060000d3a2d862-TCP-443) - dropping
[kube-controller-manager-k8s-master-99408755-0] I0719 09:01:01.310461 1 azure_loadbalancer.go:797] reconcileLoadBalancer for service (nginx-ingress/nginx-ingress-external-svc)(false): lb rule(a54d331d2a9fc11e98060000d3a2d862-TCP-80) - dropping
[kube-controller-manager-k8s-master-99408755-0] I0719 09:01:01.310476 1 azure_loadbalancer.go:828] reconcileLoadBalancer for service(nginx-ingress/nginx-ingress-external-svc): lb(xxxxxx) - deleting; no remaining frontendIPConfigurations
[kube-controller-manager-k8s-master-99408755-0] I0719 09:01:01.576293 1 azure_loadbalancer.go:880] reconcileLoadBalancer for service(nginx-ingress/nginx-ingress-external-svc): lb(xxxxxx) finished
[kube-controller-manager-k8s-master-99408755-0] I0719 09:01:01.617113 1 azure_backoff.go:303] PublicIPAddressesClient.List(xxxxxx) - backoff: success
[kube-controller-manager-k8s-master-99408755-0] I0719 09:01:01.617133 1 azure_loadbalancer.go:1366] reconcilePublicIP for service(nginx-ingress/nginx-ingress-external-svc): pip(xxxxxx-a54d331d2a9fc11e98060000d3a2d862) - deleting
[kube-controller-manager-k8s-master-99408755-0] I0719 09:01:11.896103 1 azure_loadbalancer.go:1372] reconcilePublicIP for service(nginx-ingress/nginx-ingress-external-svc): pip(xxxxxx-a54d331d2a9fc11e98060000d3a2d862) - finished
[kube-controller-manager-k8s-master-99408755-0] I0719 09:01:11.896122 1 azure_loadbalancer.go:212] Delete service (nginx-ingress/nginx-ingress-external-svc): FINISH
[kube-controller-manager-k8s-master-99408755-0] I0719 09:01:11.896361 1 event.go:209] Event(v1.ObjectReference{Kind:"Service", Namespace:"nginx-ingress", Name:"nginx-ingress-external-svc", UID:"54d331d2-a9fc-11e9-8060-000d3a2d8629", APIVersion:"v1", ResourceVersion:"207778", FieldPath:""}): type: 'Normal' reason: 'DeletedLoadBalancer' Deleted load balancer

===== RECREATION OF THE SERVICEC

[kube-controller-manager-k8s-master-99408755-0] I0719 09:07:15.258962 1 service_controller.go:300] Ensuring LB for service nginx-ingress/nginx-ingress-external-svc
[kube-controller-manager-k8s-master-99408755-0] I0719 09:07:15.265883 1 azure_loadbalancer.go:593] reconcileLoadBalancer for service(nginx-ingress/nginx-ingress-external-svc) - wantLb(true): started
[kube-controller-manager-k8s-master-99408755-0] I0719 09:07:15.275274 1 event.go:209] Event(v1.ObjectReference{Kind:"Service", Namespace:"nginx-ingress", Name:"nginx-ingress-external-svc", UID:"9a08b932-aa04-11e9-8060-000d3a2d8629", APIVersion:"v1", ResourceVersion:"215806", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
[kube-controller-manager-k8s-master-99408755-0] I0719 09:07:15.334950 1 azure_backoff.go:257] LoadBalancerClient.List(xxxxxx) - backoff: success
[kube-controller-manager-k8s-master-99408755-0] I0719 09:07:15.335003 1 azure_loadbalancer.go:302] selectLoadBalancer for service (nginx-ingress/nginx-ingress-external-svc): isInternal(false) - start
[kube-controller-manager-k8s-master-99408755-0] I0719 09:07:15.335010 1 azure_loadbalancer.go:308] selectLoadBalancer: cluster(xxxxxx) service(nginx-ingress/nginx-ingress-external-svc) isInternal(false) - vmSetNames [k8s-standardpool-99408755-vmss]
[kube-controller-manager-k8s-master-99408755-0] I0719 09:07:15.335029 1 azure_loadbalancer.go:600] reconcileLoadBalancer for service(nginx-ingress/nginx-ingress-external-svc): lb(xxxxxx) wantLb(true) resolved load balancer name
[kube-controller-manager-k8s-master-99408755-0] I0719 09:07:15.377899 1 azure_wrap.go:134] Public IP "xxxxxx-a9a08b932aa0411e98060000d3a2d862" not found with message: "network.PublicIPAddressesClient#Get: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code=\"ResourceNotFound\" Message=\"The Resource 'Microsoft.Network/publicIPAddresses/xxxxxx-a9a08b932aa0411e98060000d3a2d862' under resource group 'xxxxxx' was not found.\""
[kube-controller-manager-k8s-master-99408755-0] I0719 09:07:15.378087 1 azure_loadbalancer.go:496] ensurePublicIPExists for service(nginx-ingress/nginx-ingress-external-svc): pip(xxxxxx-a9a08b932aa0411e98060000d3a2d862) - creating
[kube-controller-manager-k8s-master-99408755-0] I0719 09:07:20.108139 1 azure_loadbalancer.go:914] reconcileLoadBalancerRule lb name (xxxxxx) rule name (a9a08b932aa0411e98060000d3a2d862-TCP-80)
[kube-controller-manager-k8s-master-99408755-0] I0719 09:07:20.108164 1 azure_loadbalancer.go:914] reconcileLoadBalancerRule lb name (xxxxxx) rule name (a9a08b932aa0411e98060000d3a2d862-TCP-443)
[kube-controller-manager-k8s-master-99408755-0] I0719 09:07:20.108183 1 azure_loadbalancer.go:849] reconcileLoadBalancer: reconcileLoadBalancer for service(nginx-ingress/nginx-ingress-external-svc): lb(xxxxxx) - updating
[kube-controller-manager-k8s-master-99408755-0] I0719 09:07:20.663845 1 azure_vmss.go:716] EnsureHostInPool begins to update vmssVM(k8s-standardpool-99408755-vmss000000) with new backendPoolID /subscriptions/xxxx-xxxxx-xxxxxx/resourceGroups/xxxxxx/providers/Microsoft.Network/loadBalancers/xxxxxx/backendAddressPools/xxxxxx
[kube-controller-manager-k8s-master-99408755-0] I0719 09:07:20.673105 1 azure_vmss.go:716] EnsureHostInPool begins to update vmssVM(k8s-standardpool-99408755-vmss000002) with new backendPoolID /subscriptions/xxxx-xxxxx-xxxxxx/resourceGroups/xxxxxx/providers/Microsoft.Network/loadBalancers/xxxxxx/backendAddressPools/xxxxxx
[kube-controller-manager-k8s-master-99408755-0] I0719 09:07:20.673172 1 azure_vmss.go:716] EnsureHostInPool begins to update vmssVM(k8s-standardpool-99408755-vmss000001) with new backendPoolID /subscriptions/xxxx-xxxxx-xxxxxx/resourceGroups/xxxxxx/providers/Microsoft.Network/loadBalancers/xxxxxx/backendAddressPools/xxxxxx
[kube-controller-manager-k8s-master-99408755-0] I0719 09:08:16.167707 1 azure_loadbalancer.go:880] reconcileLoadBalancer for service(nginx-ingress/nginx-ingress-external-svc): lb(xxxxxx) finished
[kube-controller-manager-k8s-master-99408755-0] I0719 09:08:16.206607 1 azure_loadbalancer.go:394] getServiceLoadBalancerStatus gets ingress IP "40.68.39.108" from frontendIPConfiguration "a9a08b932aa0411e98060000d3a2d862" for service "nginx-ingress/nginx-ingress-external-svc"
[kube-controller-manager-k8s-master-99408755-0] I0719 09:08:16.206668 1 azure_loadbalancer.go:139] EnsureLoadBalancer: reconciling security group for service "nginx-ingress/nginx-ingress-external-svc" with IP "xxx.xxx.xx.xx", wantLb = true
[kube-controller-manager-k8s-master-99408755-0] I0719 09:08:16.282032 1 azure_loadbalancer.go:1177] reconcileSecurityGroup for service(nginx-ingress/nginx-ingress-external-svc): sg(k8s-master-99408755-nsg) - updating
[kube-controller-manager-k8s-master-99408755-0] I0719 09:08:26.661348 1 azure_loadbalancer.go:593] reconcileLoadBalancer for service(nginx-ingress/nginx-ingress-external-svc) - wantLb(false): started
[kube-controller-manager-k8s-master-99408755-0] I0719 09:08:26.727859 1 azure_backoff.go:257] LoadBalancerClient.List(xxxxxx) - backoff: success
[kube-controller-manager-k8s-master-99408755-0] I0719 09:08:26.728057 1 azure_loadbalancer.go:600] reconcileLoadBalancer for service(nginx-ingress/nginx-ingress-external-svc): lb(xxxxxx-internal) wantLb(false) resolved load balancer name
[kube-controller-manager-k8s-master-99408755-0] I0719 09:08:26.728186 1 azure_loadbalancer.go:880] reconcileLoadBalancer for service(nginx-ingress/nginx-ingress-external-svc): lb(xxxxxx-internal) finished
[kube-controller-manager-k8s-master-99408755-0] I0719 09:08:26.819558 1 azure_backoff.go:303] PublicIPAddressesClient.List(xxxxxx) - backoff: success
[kube-controller-manager-k8s-master-99408755-0] I0719 09:08:26.871502 1 azure_backoff.go:303] PublicIPAddressesClient.List(xxxxxx) - backoff: success
[kube-controller-manager-k8s-master-99408755-0] I0719 09:08:26.915421 1 event.go:209] Event(v1.ObjectReference{Kind:"Service", Namespace:"nginx-ingress", Name:"nginx-ingress-external-svc", UID:"9a08b932-aa04-11e9-8060-000d3a2d8629", APIVersion:"v1", ResourceVersion:"215806", FieldPath:""}): type: 'Normal' reason: 'EnsuredLoadBalancer' Ensured load balancer `

[aasier]

Today I´m going to test the cluster as new with the aks-e 0.38.0 and aks 1.14.3 in order to check if is something wrong in all of my clusters (5 in PRO, 3 in DEV). I will write back.

[feiskyer]

@aasier Thanks for the information. As said on https://github.com/Azure/aks-engine/issues/1615#issuecomment-511266374, the warning icon in the "last model" is expected because we switched to instance-level APIs. This change is critical to avoid all instances being rebooted. Apply the latest model would roll back these changes, so it would make the LB not working.

I think we may need also add the LB backend updates back to VMSS model, so as to avoid this problem. Opened https://github.com/kubernetes/kubernetes/issues/80365 to address this.

Note that even after we add this back to VMSS model, the latest model issues are still there because Kubernetes won't do the model upgrade for each instance. Customers may decide to do that, but should also aware that all the instances may be rebooted at the same time.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.