Aks-engine: flatcar nodes don't work w/ Azure CNI + bridge mode

[jackfrancis]

Repro:

azureuser@k8s-pool1-34506848-vmss000000 ~ $ dig +short mcr.microsoft.com
;; connection timed out; no servers could be reached

azureuser@k8s-pool1-34506848-vmss000000 ~ $ cat /etc/resolv.conf 
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

# No DNS servers known.

[jackfrancis]

@johnsonshi any reasons why flatcar VM would have no resolvers?

[jackfrancis]

cc @invidian @iaguis

[invidian]

I guess by default it's expected to get DNS settings from DHCP server and that failed for some reason? Journal would probably tell more. Does it happen all the time?

[jackfrancis]

After more testing it seems that the Azure CNI configuration, which adds a bunch of secondary IP addresses (30 by default, matching the default --max-pods configuration), is the breaking configuration. Kubenet does not repro.

So we want to triage why the Azure CNI configuration regressed (roughly around mid-September).

[jackfrancis]

@invidian You could potentially repro by building a flatcar VM w/ 30 secondary IP addresses. I suspect that's the breaking vector.

[invidian]

Thanks for reporting @jackfrancis. I'll have a look either this or next week :+1:

[invidian]

@invidian You could potentially repro by building a flatcar VM w/ 30 secondary IP addresses. I suspect that's the breaking vector.

Hmm, I have tested it with many secondary IP addresses configured. I had to configure them by hand using ip addr add, as it seems DHCP does not support leasing multiple IPs. So I suspect that this is a regression in Azure CNI.

[invidian]

Okay, so I found the following logs in /var/log/azure-vnet.log:

2020/10/26 15:40:46 [6511] [cni-net] Plugin started.
2020/10/26 15:40:46 [6511] [cni-net] Processing ADD command with args {ContainerID:1ca2a22ccbffaafd1243d8d368a193300eb708480f911c2275686e740932eced Netns:/proc/6361/ns/net IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=coredns-autoscaler-5c7db64899-m4vkm;K8S_POD_INFRA_CONTAINER_ID=1ca2a22ccbffaafd1243d8d368a193300eb708480f911c2275686e740932eced Path:/opt/cni/bin StdinData:{"bridge":"azure0","cniVersion":"0.3.0","ipam":{"type":"azure-vnet-ipam"},"ipsToRouteViaHost":["169.254.20.10"],"mode":"bridge","name":"azure","type":"azure-vnet"}}.
2020/10/26 15:40:46 [6511] [cni-net] Read network configuration &{CNIVersion:0.3.0 Name:azure Type:azure-vnet Mode:bridge Master: Bridge:azure0 LogLevel: LogTarget: InfraVnetAddressSpace: IPV6Mode: ServiceCidrs: VnetCidrs: PodNamespaceForDualNetwork:[] IPsToRouteViaHost:[169.254.20.10] MultiTenancy:false EnableSnatOnHost:false EnableExactMatchForPodName:false DisableHairpinOnHostInterface:false DisableIPTableLock:false CNSUrl: Ipam:{Type:azure-vnet-ipam Environment: AddrSpace: Subnet: Address: QueryInterval:} DNS:{Nameservers:[] Domain: Search:[] Options:[]} RuntimeConfig:{PortMappings:[] DNS:{Servers:[] Searches:[] Options:[]}} AdditionalArgs:[]}.
2020/10/26 15:40:46 [6511] Get number of endpoints for ifname  network azure
2020/10/26 15:40:46 [6511] Result from multitenancy <nil>
2020/10/26 15:40:46 [6511] [cni-net] Creating network azure.
2020/10/26 15:40:46 [6511] [cni] Calling plugin azure-vnet-ipam ADD nwCfg:&{CNIVersion:0.3.0 Name:azure Type:azure-vnet Mode:bridge Master: Bridge:azure0 LogLevel: LogTarget: InfraVnetAddressSpace: IPV6Mode: ServiceCidrs: VnetCidrs: PodNamespaceForDualNetwork:[] IPsToRouteViaHost:[169.254.20.10] MultiTenancy:false EnableSnatOnHost:false EnableExactMatchForPodName:false DisableHairpinOnHostInterface:false DisableIPTableLock:false CNSUrl: Ipam:{Type:azure-vnet-ipam Environment: AddrSpace: Subnet: Address: QueryInterval:} DNS:{Nameservers:[] Domain: Search:[] Options:[]} RuntimeConfig:{PortMappings:[] DNS:{Servers:[] Searches:[] Options:[]}} AdditionalArgs:[]}.
2020/10/26 15:40:46 [6511] [cni] Plugin azure-vnet-ipam returned result:IP:[{Version:4 Interface:<nil> Address:{IP:10.240.0.12 Mask:fff00000} Gateway:10.240.0.1}], Routes:[{Dst:{IP:0.0.0.0 Mask:00000000} GW:10.240.0.1}], DNS:{Nameservers:[168.63.129.16] Domain: Search:[] Options:[]}, err:<nil>.
2020/10/26 15:40:46 [6511] [cni-net] Found master interface eth0.
2020/10/26 15:40:46 [6511] [net] Added ExternalInterface eth0 for subnet 10.240.0.0/12.
2020/10/26 15:40:46 [6511] [net] Save succeeded.
2020/10/26 15:40:46 [6511] [cni-net] nwDNSInfo: { [168.63.129.16] []}
2020/10/26 15:40:46 [6511] [net] Creating network &{MasterIfName:eth0 Id:azure Mode:bridge Subnets:[{Family:2 Prefix:{IP:10.240.0.0 Mask:fff00000} Gateway:10.240.0.1}] DNS:{Suffix: Servers:[168.63.129.16] Options:[]} Policies:[] BridgeName:azure0 EnableSnatOnHost:false NetNs:/proc/6361/ns/net Options:map[] DisableHairpinOnHostInterface:false IPV6Mode: ServiceCidrs:}.
2020/10/26 15:40:46 [6511] opt map[] options map[]
2020/10/26 15:40:46 [6511] create bridge
2020/10/26 15:40:46 [6511] [net] Connecting interface eth0.
2020/10/26 15:40:46 [6511] [net] Creating bridge azure0.
2020/10/26 15:40:46 [6511] [Azure-Utils] sysctl -w net.ipv6.conf.azure0.accept_ra=0
2020/10/26 15:40:46 [6511] [net] Deleting IP address 10.240.0.4/12 from interface eth0.
2020/10/26 15:40:46 [6511] [net] Saved interface IP configuration &{Name:eth0 Networks:map[] Subnets:[10.240.0.0/12] BridgeName: DNSInfo:{Suffix: Servers:[] Options:[]} MacAddress:00:22:48:6f:f2:c6 IPAddresses:[10.240.0.4/12] Routes:[0xc000131950] IPv4Gateway:10.240.0.1 IPv6Gateway:::}.
2020/10/26 15:40:46 [6511] [net] OSInfo: map[ANSI_COLOR:"38;5;75" BUG_REPORT_URL:"https://issues.flatcar-linux.org" BUILD_ID:2020-09-28-2140 FLATCAR_BOARD:"amd64-usr" HOME_URL:"https://flatcar-linux.org/" ID:flatcar ID_LIKE:coreos NAME:"Flatcar Container Linux by Kinvolk" PRETTY_NAME:"Flatcar Container Linux by Kinvolk 2605.6.0 (Oklo)" VERSION:2605.6.0 VERSION_ID:2605.6.0]
2020/10/26 15:40:46 [6511] [net] Setting link eth0 state down.
2020/10/26 15:40:46 [6511] [net] Setting link eth0 master azure0.
2020/10/26 15:40:46 [6511] [net] Setting link eth0 state up.
2020/10/26 15:40:46 [6511] [net] Setting link azure0 state up.
2020/10/26 15:40:46 [6511] [net] Adding SNAT rule for egress traffic on eth0.
2020/10/26 15:40:46 [6511] [Azure-Utils] ebtables -t nat -A POSTROUTING -s unicast -o eth0 -j snat --to-src 00:22:48:6f:f2:c6 --snat-arp --snat-target ACCEPT
2020/10/26 15:40:46 [6511] [net] Adding ARP reply rule for primary IP address 10.240.0.4.
2020/10/26 15:40:46 [6511] [Azure-Utils] ebtables -t nat -A PREROUTING -p ARP --arp-op Request --arp-ip-dst 10.240.0.4 -j arpreply --arpreply-mac 00:22:48:6f:f2:c6 --arpreply-target DROP
2020/10/26 15:40:46 [6511] [net] Adding DNAT rule for ingress ARP traffic on interface eth0.
2020/10/26 15:40:46 [6511] [Azure-Utils] ebtables -t nat -A PREROUTING -p ARP -i eth0 --arp-op Reply -j dnat --to-dst ff:ff:ff:ff:ff:ff --dnat-target ACCEPT
2020/10/26 15:40:46 [6511] [net] Setting link eth0 hairpin on.
2020/10/26 15:40:46 [6511] [net] Adding IP address 10.240.0.4/12 to interface azure0.
2020/10/26 15:40:46 [6511] [net] Adding IP route &{Family:2 Dst:<nil> Src:10.240.0.4 Gw:10.240.0.1 Tos:0 Table:254 Protocol:16 Scope:0 Type:1 Flags:0 Priority:1024 LinkIndex:4 ILinkIndex:0}.
2020/10/26 15:40:46 [6511] [net] Connected interface eth0 to bridge azure0.
2020/10/26 15:40:46 [6511] [net] Connecting interface eth0 completed with err:<nil>.
2020/10/26 15:40:46 [6511] [net] Created network azure on interface eth0.
2020/10/26 15:40:46 [6511] [net] Save succeeded.
2020/10/26 15:40:46 [6511] [cni-net] Created network azure with subnet 10.240.0.0/12.
2020/10/26 15:40:46 [6511] [cni-net] Creating endpoint 1ca2a22c-eth0.
2020/10/26 15:40:46 [6511] [net] Creating endpoint &{Id:1ca2a22c-eth0 ContainerID:1ca2a22ccbffaafd1243d8d368a193300eb708480f911c2275686e740932eced NetNsPath:/proc/6361/ns/net IfName:eth0 SandboxKey: IfIndex:0 MacAddress: DNS:{Suffix: Servers:[168.63.129.16] Options:[]} IPAddresses:[{IP:10.240.0.12 Mask:fff00000}] IPsToRouteViaHost:[169.254.20.10] InfraVnetIP:{IP:<nil> Mask:<nil>} Routes:[{Dst:{IP:0.0.0.0 Mask:00000000} Src:<nil> Gw:10.240.0.1 Protocol:0 DevName: Scope:0 Priority:0}] Policies:[] Gateways:[] EnableSnatOnHost:false EnableInfraVnet:false EnableMultiTenancy:false EnableSnatForDns:false AllowInboundFromHostToNC:false AllowInboundFromNCToHost:false NetworkContainerID: PODName:coredns-autoscaler-5c7db64899-m4vkm PODNameSpace:kube-system Data:map[vethname:azure1ca2a22ccbffaafd1243d8d368a193300eb708480f911c2275686e740932ecedeth0] InfraVnetAddressSpace: SkipHotAttachEp:false IPV6Mode: VnetCidrs: ServiceCidrs:} in network azure.
2020/10/26 15:40:46 [6511] Generate veth name based on the key provided azure1ca2a22ccbffaafd1243d8d368a193300eb708480f911c2275686e740932ecedeth0
2020/10/26 15:40:46 [6511] Bridge client
2020/10/26 15:40:46 [6511] [net] Creating veth pair azv894be745adc azv894be745adc2.
2020/10/26 15:40:46 [6511] [net] Setting link azv894be745adc state up.
2020/10/26 15:40:46 [6511] [Azure-Utils] sysctl -w net.ipv6.conf.azv894be745adc.accept_ra=0
2020/10/26 15:40:46 [6511] [net] Setting link azv894be745adc master azure0.

I suspect this might be related to recent systemd updates in Flatcar stable (CC @t-lo).

As far as I understand, the following events occur:

1. Machine boots and eth0 interface gets configured via DHCP, including resolver configuration.
2. Machine provisioning runs fine, kubelet starts triggering CNI script
3. CNI script creates azure0 bridge interface
4. eth0 IP address gets removed, assigned to azure0 interface and added to the bridge
5. Resolving breaks at this point.

I suspect when eth0 gets into degraded state, as IP address gets removed manually from it, then systemd-resolved also removed DNS configuration from the node and Azure CNI is not restoring it?

[jackfrancis]

I wonder if using Azure CNI in transparent mode (no bridge interface) will not repro this, in which case we can just ensure folks building clusters w/ flatcar node pools get that config. Will test.

cc @johnsonshi @tamilmani1989

[invidian]

I think we could also workaround it by configuring systemd-resolved to use static DNS entries from Ignition configuration.

However, this does not resolve the issue of azure-vnet tinkering with network configuration while systemd-networkd is running. Perhaps azure-vnet should somehow integrate with systemd, or at least make sure that interfaces azure0 and azv* are not managed by systemd-networkd via drop-in file like:

cat /etc/systemd/network/20-ignore.network
[Match]
Name=azure0 azv*

[Link]
Unmanaged=true

The problem right now is that systemd-networkd is used until node is provisioned and kubelet starts running CNI script, so we cannot disable the management of eth0 interace.

@jackfrancis what are the implications of enabling transparent mode?

[jackfrancis]

PR #3958 is actually aiming to turn on transparent mode by default everywhere. In theory the L3 bridge interface is supposed to optimize packet delivery, but folks aren't 100% sure in practice if this is true, and as you've observed there's more OS overhead involved in "transitioning" to that add'l interface desired state. In addition fairly high % UDP edge cases are observed that produce regular high latency DNS lookups.

@matmerr @chandanAggarwal can maybe add more to my description (or correct it if I've misrepresented anything)

[jackfrancis]

Validated that transparent mode is functional. Will put up a PR that adds that validation, and re-enables flatcar testing.