Cluster-api-provider-azure: `systemd-timesyncd` is failing to sync with `ntp.ubuntu.com` on Control Plane nodes

[jseely]

/kind bug

What steps did you take and what happened:
When you deploy a CAPZ cluster, the control plane nodes are unable to connect to ntp.ubuntu.com. This can lead to clock drift which can seriously impact cluster functionality (etcd especially).

systemd-timesyncd logs show:

Apr 15 19:51:15 jseely-capz-control-plane-qh9zn systemd-timesyncd[759]: Timed out waiting for reply from 91.189.94.4:123 (ntp.ubuntu.com).
Apr 15 19:51:25 jseely-capz-control-plane-qh9zn systemd-timesyncd[759]: Timed out waiting for reply from 91.189.89.199:123 (ntp.ubuntu.com).
Apr 15 19:51:35 jseely-capz-control-plane-qh9zn systemd-timesyncd[759]: Timed out waiting for reply from 91.189.91.157:123 (ntp.ubuntu.com).
Apr 15 19:51:46 jseely-capz-control-plane-qh9zn systemd-timesyncd[759]: Timed out waiting for reply from 91.189.89.198:123 (ntp.ubuntu.com).
Apr 15 20:09:00 jseely-capz-control-plane-qh9zn systemd-timesyncd[759]: Timed out waiting for reply from 91.189.89.199:123 (ntp.ubuntu.com).
Apr 15 20:09:10 jseely-capz-control-plane-qh9zn systemd-timesyncd[759]: Timed out waiting for reply from 91.189.91.157:123 (ntp.ubuntu.com).
Apr 15 20:09:21 jseely-capz-control-plane-qh9zn systemd-timesyncd[759]: Timed out waiting for reply from 91.189.89.198:123 (ntp.ubuntu.com).
Apr 15 20:09:31 jseely-capz-control-plane-qh9zn systemd-timesyncd[759]: Timed out waiting for reply from 91.189.94.4:123 (ntp.ubuntu.com).

Ping also fails to connect to ntp.ubuntu.com servers:

capi@jseely-capz-control-plane-qh9zn:~$ ping ntp.ubuntu.com
PING ntp.ubuntu.com (91.189.91.157) 56(84) bytes of data.
^C
--- ntp.ubuntu.com ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4078ms

What did you expect to happen:
Control plane VMs should have network access to ntp.ubuntu.com by default and they should therefore be able to sync time against those servers.

Environment:

• cluster-api-provider-azure version: v0.4.1
• cluster-api version: v0.3.3
• Kubernetes version: (use kubectl version): v1.17.3
• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.4 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

[jseely]

^ @CecileRobertMichon @justaugustus @detiber @jayunit100

[CecileRobertMichon]

/assign

[CecileRobertMichon]

The failure to ping is expected @jseely, see https://docs.microsoft.com/en-us/azure/cloud-services/cloud-services-connectivity-and-networking-faq#can-i-ping-a-cloud-service

Tested I can curl www.google.com from a capz VM so not all outbound connectivity is broken. Not sure why timesyncd is timing out reaching NTP though, will look into that. Thanks for filing the issue.

[CecileRobertMichon]

related: https://github.com/Azure/aks-engine/pull/2815

https://docs.microsoft.com/en-us/azure/virtual-machines/linux/time-sync

[jseely]

@CecileRobertMichon Ping failure is not expected.

Ping results from standard Ubuntu 18.04 VM on Azure:

capi@capz-jump:~$ ping ntp.ubuntu.com
PING ntp.ubuntu.com (91.189.89.198) 56(84) bytes of data.
64 bytes from chilipepper.canonical.com (91.189.89.198): icmp_seq=1 ttl=40 time=137 ms
64 bytes from chilipepper.canonical.com (91.189.89.198): icmp_seq=2 ttl=40 time=137 ms
64 bytes from chilipepper.canonical.com (91.189.89.198): icmp_seq=3 ttl=40 time=137 ms
^C
--- ntp.ubuntu.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 137.393/137.461/137.507/0.306 ms

vs Ping results from Control Plane nodes:

root@jseely-capz2-control-plane-24xjw:/etc/kubernetes# ping ntp.ubuntu.com
PING ntp.ubuntu.com (91.189.89.198) 56(84) bytes of data.
^C
--- ntp.ubuntu.com ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3049ms

[jseely]

I was just thinking you may have been right with Basic Sku Load Balancers. I do remember that ICMP didn't work at some point in Azure Networking, which would block ping.

That said, here are the ping results from a VM behind a Standard Sku Load Balancer:

capi@jseely-capz2-jump:~$ ping ntp.ubuntu.com
PING ntp.ubuntu.com (91.189.91.157) 56(84) bytes of data.
64 bytes from alphyn.canonical.com (91.189.91.157): icmp_seq=1 ttl=44 time=80.3 ms
64 bytes from alphyn.canonical.com (91.189.91.157): icmp_seq=2 ttl=44 time=80.5 ms
64 bytes from alphyn.canonical.com (91.189.91.157): icmp_seq=3 ttl=44 time=80.5 ms
64 bytes from alphyn.canonical.com (91.189.91.157): icmp_seq=4 ttl=44 time=80.3 ms
64 bytes from alphyn.canonical.com (91.189.91.157): icmp_seq=5 ttl=44 time=80.4 ms
64 bytes from alphyn.canonical.com (91.189.91.157): icmp_seq=6 ttl=44 time=80.4 ms
64 bytes from alphyn.canonical.com (91.189.91.157): icmp_seq=7 ttl=44 time=80.3 ms
64 bytes from alphyn.canonical.com (91.189.91.157): icmp_seq=8 ttl=44 time=80.5 ms
64 bytes from alphyn.canonical.com (91.189.91.157): icmp_seq=9 ttl=44 time=80.4 ms
64 bytes from alphyn.canonical.com (91.189.91.157): icmp_seq=10 ttl=44 time=80.2 ms
64 bytes from alphyn.canonical.com (91.189.91.157): icmp_seq=11 ttl=44 time=80.4 ms
64 bytes from alphyn.canonical.com (91.189.91.157): icmp_seq=12 ttl=44 time=80.1 ms
64 bytes from alphyn.canonical.com (91.189.91.157): icmp_seq=13 ttl=44 time=80.1 ms
64 bytes from alphyn.canonical.com (91.189.91.157): icmp_seq=14 ttl=44 time=80.4 ms
64 bytes from alphyn.canonical.com (91.189.91.157): icmp_seq=15 ttl=44 time=80.5 ms
^C
--- ntp.ubuntu.com ping statistics ---
15 packets transmitted, 15 received, 0% packet loss, time 14021ms
rtt min/avg/max/mdev = 80.111/80.390/80.553/0.243 ms

[jseely]

So I just did more digging and I owe you an apology @CecileRobertMichon, looks like you're right about ping not working behind the Standard LB.

[CecileRobertMichon]

@jseely @devigned @alexeldeib and I did some poking around and I'm pretty sure we found the root cause / fix (spoiler: it looks like a missing LB rule for UDP traffic), will put out a PR shortly.

[detiber]

@CecileRobertMichon should we also follow up and ensure that VMICTimeSync is enabled on images and standardize on using ntpd over systemd-timesyncd to align with AKS-Engine?

[CecileRobertMichon]

@detiber we should, I'll file a separate issue to track that.

Also want to look into using host based timesync and/or chrony