Aks-engine: Azure CNI + Ubuntu 18.04 container networking is broken

[CecileRobertMichon]

Azure CNI + 18.04 container networking is broken:

Azure Container Cluster using the Kubernetes Orchestrator
/go/src/github.com/Azure/aks-engine/test/e2e/kubernetes/kubernetes_test.go:87
  regardless of agent pool type
  /go/src/github.com/Azure/aks-engine/test/e2e/kubernetes/kubernetes_test.go:88
    should be able to launch a long-running container networking DNS liveness pod [It]
    /go/src/github.com/Azure/aks-engine/test/e2e/kubernetes/kubernetes_test.go:319

    Expected error:
        <*errors.fundamental | 0xc0001fa180>: {
            msg: "Timeout exceeded (2m0s) while waiting for Pods (dns-liveness) to become ready in namespace (default), got 0 of 6 required successful pods ready results",
            stack: [0x99ca24, 0x45e901],
        }
        Timeout exceeded (2m0s) while waiting for Pods (dns-liveness) to become ready in namespace (default), got 0 of 6 required successful pods ready results
    not to have occurred

apimodel to reproduce:

{   "apiVersion": "vlabs",   "properties": {     "orchestratorProfile": {       "orchestratorType": "Kubernetes",       "kubernetesConfig": {         "networkPlugin": "azure"       }     },     "masterProfile": {       "count": 1,       "dnsPrefix": "",       "vmSize": "Standard_D2_v2",       "distro": "aks-18.04"     },     "agentPoolProfiles": [       {         "name": "agentpool1",         "count": 2,         "vmSize": "Standard_D2_v2",         "distro": "aks-18.04"       }     ],     "linuxProfile": {       "adminUsername": "azureuser",       "ssh": {         "publicKeys": [           {             "keyData": ""           }         ]       }     },     "servicePrincipalProfile": {       "clientId": "",       "secret": ""     }   } }

The same thing with "networkPlugin": "kubenet" passes E2E.

[CecileRobertMichon]

@tamilmani1989 @sharmasushant can someone from your team please help investigate this issue? AKS enabling 18.04 is dependent on resolution of this.

[tamilmani1989]

@CecileRobertMichon I will look into this. BTW what test is failing here..so that I will setup the pod and run it manually to check.

[CecileRobertMichon]

@tamilmani1989 thanks. The container DNS E2E test is failing It should be able to launch a long-running container networking DNS liveness pod. I've also noticed that pods aren't able to pull container images if they're aren't already on the machine (to repro this, change imagePullPolicy of an addon to Always).

In order for you to test, https://github.com/Azure/aks-engine/pull/795 needs to merge first in order to fix OS provisioning timeouts. I'll let you know when that is the case.

[tamilmani1989]

@CecileRobertMichon ok thanks. let me know when #795 is merged

[CecileRobertMichon]

@tamilmani1989 #884 merged and fixed the OSPTOs issue

[tamilmani1989]

@CecileRobertMichon Sorry. I missed this. Can you tell me how to run this dns e2e test once I deployed the cluster?

[CecileRobertMichon]

@tamilmani1989 sorry about that, I missed your answer... You can use

CLUSTER_DEFINITION=examples/kubernetes.json SUBSCRIPTION_ID="<YOUR_SUB_ID>" CLIENT_ID="<YOUR_CLIENT_ID" CLIENT_SECRET="<YOUR_CLIENT_SECRET>" TENANT_ID="<YOUR_TENANT_ID>" LOCATION=<REGION> CLEANUP_ON_EXIT=false make test-kubernetes

To deploy a cluster and run e2e on it locally. The CLUSTER_DEFINITION should be the one I pasted in the issue description. Let me know if you need help getting a repro.

[palma21]

@tamilmani1989 have you had the change to look at this?

we'd love to get 18.04 out rolling

[tamilmani1989]

@palma21 No I didn't start working on this. I will plan this next sprint.

[tamilmani1989]

@palma21 @CecileRobertMichon I did some investigation on this today. I created Kubernetes cluster with ubuntu18 image and tried to create dns-liveness pod manually. Copied dns-liveness yaml from aks-engine repo.
I found that CNI call to create dns-liveness pod is succeeding but agent node not able to pull image. The things I noticed are :

1. dns working fine in master. It has ubuntu16 image
2. agent nodes (ubuntu18) are not able to resolve any dns names. The dns configuration looks different from master
3. In master , the dns server is configured as 168.63.129.16
4. In agent nodes dns server is configured as 127.0.0.53. Even google.com resolution is failing in agent node.

rnmadmin@k8s-agentpool1-26029301-0:~$ wget google.com
--2019-04-20 01:10:02-- http://google.com/
Resolving google.com (google.com)... failed: Temporary failure in name resolution.
wget: unable to resolve host address ‘google.com’

In ubuntu 16, the dns server is configured as 168.63.129.16. Why this was changed to 127.0.0.53 in ubuntu 18?

[CecileRobertMichon]

@tamilmani1989 the default DNS resolver in 18.04 is systemd-resolved. The standard /etc/resolve.conf file is now managed by systemd-resolved.

https://askubuntu.com/questions/1080230/how-to-make-ubuntu-server-18-04-use-dns-nameservers-from-etc-network-interfaces.

For some reason the exact same test works with kubenet as networkPlugin (you can try this by replacing "azure" by "kubenet" in the networkPlugin field of the apimodel.

[tamilmani1989]

@CecileRobertMichon Yes, I investigated from that perspective and found that systemd-resolved configures dns server on interface. By default, in azure ubuntu18 image , eth0 was configured with 168.63.129.16. In case of kubenet, eth0 is main interface and there is no issue with dns. In azure cni case, a linux bridge(azure0) will be created and eth0 is slave of azure0. No dns server was configured on azure0.
Using systemd-resolve command I set dns server on azure0 interface and it started to work. Right now
I'm investigating two things:

1. is there any way to read dns server from eth0 interface.
2. How to set dns server on interface programatically using golang. If its not available, then we have to rely on systemd-resolve command

[CecileRobertMichon]

@tamilmani1989 any updates?

[CecileRobertMichon]

fixed by #1192