Aks-engine: Windows + VMSS external svc routing broken

Created on 20 Mar 2019 · 5Comments · Source: Azure/aks-engine

Repro:

Create a deployment + service (e.g., IIS). Like this:

$ k get pods --all-namespaces
NAMESPACE     NAME                                                  READY   STATUS    RESTARTS   AGE
default       dns-liveness                                          1/1     Running   0          18m
default       iis-kubernetes-westus2-87576-68128-745594868b-l4fn7   1/1     Running   0          12m
default       php-apache-long-running-7549b5558f-qzsfm              1/1     Running   0          17m
kube-system   azure-cni-networkmonitor-jrr4g                        1/1     Running   0          20m
kube-system   azure-ip-masq-agent-5qb4s                             1/1     Running   0          20m
kube-system   blobfuse-flexvol-installer-7tlsc                      1/1     Running   0          20m
kube-system   coredns-6fcd4ffdc-rlghv                               1/1     Running   0          20m
kube-system   keyvault-flexvolume-6fvks                             1/1     Running   0          20m
kube-system   kube-addon-manager-k8s-master-20982975-0              1/1     Running   0          19m
kube-system   kube-apiserver-k8s-master-20982975-0                  1/1     Running   0          20m
kube-system   kube-controller-manager-k8s-master-20982975-0         1/1     Running   0          20m
kube-system   kube-proxy-2mcns                                      1/1     Running   0          20m
kube-system   kube-scheduler-k8s-master-20982975-0                  1/1     Running   0          19m
kube-system   kubernetes-dashboard-54b795b69b-6d4hs                 1/1     Running   0          20m
kube-system   metrics-server-6bf85bb69b-5jqhs                       1/1     Running   0          20m
kube-system   tiller-deploy-5d9ffb885b-2s2b9                        1/1     Running   0          20m
$ k get svc --all-namespaces
NAMESPACE     NAME                                 TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)         AGE
default       iis-kubernetes-westus2-87576-68128   LoadBalancer   10.0.180.62    52.247.222.249   80:30497/TCP    7m24s
default       kubernetes                           ClusterIP      10.0.0.1       <none>           443/TCP         20m
default       php-apache-long-running              ClusterIP      10.0.209.58    <none>           80/TCP          15m
kube-system   kube-dns                             ClusterIP      10.0.0.10      <none>           53/UDP,53/TCP   20m
kube-system   kubernetes-dashboard                 NodePort       10.0.25.247    <none>           443:32752/TCP   20m
kube-system   metrics-server                       ClusterIP      10.0.91.96     <none>           443/TCP         20m
kube-system   tiller-deploy                        ClusterIP      10.0.130.173   <none>           44134/TCP       20m

Verify that you can connect to the service internally:

$ k exec -it php-apache-long-running-7549b5558f-qzsfm -- /bin/bash
root@php-apache-long-running-7549b5558f-qzsfm:/var/www/html# curl 10.0.180.62
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
<title>IIS Windows Server</title>
<style type="text/css">
<!--
body {
    color:#000000;
    background-color:#0072C6;
    margin:0;
}

#container {
    margin-left:auto;
    margin-right:auto;
    text-align:center;
    }

a img {
    border:none;
}

-->
</style>
</head>
<body>
<div id="container">
<a href="http://go.microsoft.com/fwlink/?linkid=66138&amp;clcid=0x409"><img src="iisstart.png" alt="IIS" width="960" height="600" /></a>
</div>
</body>
</html>

Verify that you are unable to connect to the service externally:

$ curl http://52.247.222.249
curl: (7) Failed to connect to 52.247.222.249 port 80: Operation timed out

windows

Source

jackfrancis

👍1

Most helpful comment

This is because primaryScaleSetName is wrong in azure.json. Fix the config and restart kube-controller-manager could bring the LoadBalancer services back.

feiskyer on 11 Apr 2019

👍2

All 5 comments

@wuhanyumsft, who has experienced this as well

jackfrancis on 21 Mar 2019

@jackfrancis Do you know who is working on this? Got no update for almost a week.

wuhanyumsft on 27 Mar 2019

JFYI I have the same issue described by @jackfrancis, with AKS-E v0.33.5, K8s 1.14.0 and VMSS for Windows agent nodes.

$ kubectl run iis --image mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019 --port 80
$ kubectl expose deployment iis --port 80 --type LoadBalancer
$ curl http://$(kubectl get svc iis -o jsonpath="{.status.loadBalancer.ingress[*].ip}")
curl: (7) Failed to connect to x.x.x.x port 80: Connection timed out

I'm confirming also that works as expected with VMAS: "availabilityProfile": "AvailabilitySet" for the agentPoolProfiles section.

mathieu-benoit on 9 Apr 2019

This is because primaryScaleSetName is wrong in azure.json. Fix the config and restart kube-controller-manager could bring the LoadBalancer services back.

feiskyer on 11 Apr 2019

👍2

Late to the party, but yes, it seems that the LB created for the service has a empty backend pool, which is to be expected because as @feiskyer said, the primaryScaleSetName is incorrectly set in azure.json.

Basically what happens is that the value set in azure.json is from the primaryScaleSetName variable in the ARM template, which now is set here [1] and is expected to be of the form: "k8s-.*-vmss". However, the windows and linux VMSSs are named differently. The name for the agent VMSS is derived from the variable [agentPoolName]VMNamePrefix [2] , which is populated differently for Windows vs Linux agentpools [3] , [4].

PR #1028 should fix this.

[1] https://github.com/Azure/aks-engine/blob/56fc4e6df988257e14a5b97de8cd925bfdbc4b53/pkg/engine/armvariables.go#L330
[2] https://github.com/Azure/aks-engine/blob/56fc4e6df988257e14a5b97de8cd925bfdbc4b53/pkg/engine/virtualmachinescalesets.go#L363
[3] https://github.com/Azure/aks-engine/blob/56fc4e6df988257e14a5b97de8cd925bfdbc4b53/pkg/engine/armvariables.go#L499
[4] https://github.com/Azure/aks-engine/blob/35df683312156647309f1ed113fe32cf7e70d715/pkg/api/types.go#L824-L843