Aks-engine: Ipv6 is not working when using kubenet + iptables dualstack

[TonyDou]

Describe the bug
Can't connect loadbalancer service with ipv6. Connection will timeout.

Steps To Reproduce

aks-engine version: v0.55.4

{
    "apiVersion": "vlabs",
    "location": "southcentralus",
    "properties": {
        "featureFlags": {
            "enableIPv6DualStack": true
        },
        "orchestratorProfile": {
            "orchestratorType": "Kubernetes",
            "orchestratorRelease": "1.18",
            "kubernetesConfig": {
                "networkPlugin": "kubenet",
                "useManagedIdentity": true,
                "cloudProviderDisableOutboundSNAT": true,
                "apiServerConfig": {
                    "--feature-gates": "IPv6DualStack=true"
                },
                "kubeletConfig": {
                    "--feature-gates": "IPv6DualStack=true"
                },
                "controllerManagerConfig": {
                    "--feature-gates": "IPv6DualStack=true"
                }
            }
        },
        "masterProfile": {
            "count": 1,
            "dnsPrefix": "xxxx",
            "vmSize": "Standard_D2_v3"
        },
        "agentPoolProfiles": [
            {
                "name": "linuxpool1",
                "count": 2,
                "vmSize": "Standard_D2_v3",
                "availabilityProfile": "VirtualMachineScaleSets"
            }
        ],
        "linuxProfile": {
            "adminUsername": "xxxx",
            "ssh": {
                "publicKeys": [
                    {
                        "keyData": ""
                    }
                ]
            }
        }
    }
}

Expected behavior
Both ipv4 and Ipv6 Ip address should be able to connect and call through.

AKS Engine version
v0.55.1

Kubernetes version
1.18

Additional context
If change kubeproxy to ipvs it will work for both ipv4 and ipv6.

[welcome[bot]]

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it.

[aramase]

@TonyDou If this works fine for ipvs proxy mode but not iptables, then this could be an issue in Kubernetes upstream. Can you confirm if the endpoints for the service are populated correctly? Also, could you attach a snapshot of ip6tables-save?

[TonyDou]

Hi @aramase, here is the output for the service. I can call the service with Cluster-IP (fd00::fec8) in one of the pods. Connection can't be created with the ipv6 Public IP.

tony@xxx:~/aks$ kubectl get svc
NAME         TYPE           CLUSTER-IP   EXTERNAL-IP           PORT(S)        AGE
kubernetes   ClusterIP      10.0.0.1     <none>                443/TCP        8d
nginx-ipv4   LoadBalancer   10.0.2.224   40.124.123.103        80:30938/TCP   8d
nginx-ipv6   LoadBalancer   fd00::fec8   2603:1030:805:1::13   80:31276/TCP   8d

Snapshots of ip6tables are attached.

node2.ip6table.txt
node1.ip6table.txt

[TonyDou]

I checked cluster-info dump. There are lots of KUBE-MARK-DROP related errors.

Error occurred at line: 51
Try `ip6tables-restore -h' or 'ip6tables-restore --help' for more information.
)
I0921 07:32:07.901592       1 proxier.go:825] Sync failed; retrying in 30s
E0921 07:32:13.754762       1 proxier.go:1555] Failed to execute iptables-restore: exit status 2 (ip6tables-restore v1.8.3 (legacy): Couldn't load target `KUBE-MARK-DROP':No such file or directory

And there's no rule for the Loadbalancer IP found in the ip6tables-save result. A similar rule for IPv6 should exist in IPv6 table as following.

-A KUBE-SERVICES -d 13.73.xxx.xx/32 -p tcp -m comment --comment "default/nginx-se-ipv4: loadbalancer IP" -m tcp --dport 80 -j KUBE-FW-WQUINNJKOMGKGNH4

@aramase , do you know why KUBE-MARK-DROP can't be loaded? Any known issues? I tested on aks-engine v0.55.4 and saw the same issue.

[aojea]

@aramase , do you know why KUBE-MARK-DROP can't be loaded? Any known issues? I tested on aks-engine v0.55.4 and saw the same issue.

This seems related to this https://github.com/kubernetes/kubernetes/pull/94474, that was merged past week

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.