Installer: kube-proxy not functional on release-4.1

[tmjd]

Version

$ openshift-install version
./openshift-install-4.1.0 v4.1.0-201905212232-dirty
built from commit 71d8978039726046929729ad15302973e3da18ce
release image quay.io/openshift-release-dev/ocp-release@sha256:b8307ac0f3ec4ac86c3f3b52846425205022da52c16f56ec31cbe428501001d6

Platform (aws|libvirt|openstack):

aws

What happened?

Kube-proxy is not able to setup its iptables rules.
The logs from kube-proxy indicate that modules (for iptables rule manipulation) do not exist in the image and kube-proxy is not able to set up kubernetes services, specifically the kubernetes service itself.

W0604 21:21:41.650330       1 proxier.go:498] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0604 21:21:41.651837       1 proxier.go:498] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0604 21:21:41.652773       1 proxier.go:498] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0604 21:21:41.653687       1 proxier.go:498] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0604 21:21:41.696610       1 node.go:103] Failed to retrieve node info: nodes "ip-10-0-148-50" not found
I0604 21:21:41.696635       1 server_others.go:148] Using iptables Proxier.
W0604 21:21:41.696744       1 proxier.go:314] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
I0604 21:21:41.696831       1 server_others.go:178] Tearing down inactive rules.
E0604 21:21:41.699527       1 proxier.go:232] Error removing userspace rule: error checking rule: exit status 3: modprobe: ERROR: could not insert 'ip_tables': Exec format error
iptables v1.6.0: can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
E0604 21:21:41.701417       1 proxier.go:238] Error removing userspace rule: error checking rule: exit status 3: modprobe: ERROR: could not insert 'ip_tables': Exec format error
iptables v1.6.0: can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
E0604 21:21:41.704851       1 proxier.go:246] Error removing userspace rule: error checking rule: exit status 3: modprobe: ERROR: could not insert 'ip_tables': Exec format error
iptables v1.6.0: can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
E0604 21:21:41.708046       1 proxier.go:252] Error removing userspace rule: error checking rule: exit status 3: modprobe: ERROR: could not insert 'ip_tables': Exec format error
iptables v1.6.0: can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
E0604 21:21:41.709778       1 proxier.go:259] Error removing userspace rule: error checking rule: exit status 3: modprobe: ERROR: could not insert 'ip_tables': Exec format error
iptables v1.6.0: can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
E0604 21:21:41.711465       1 proxier.go:274] Error flushing userspace chain: error flushing chain "KUBE-PORTALS-CONTAINER": exit status 3: modprobe: ERROR: could not insert 'ip_tables': Exec format error
iptables v1.6.0: can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
E0604 21:21:41.713073       1 proxier.go:274] Error flushing userspace chain: error flushing chain "KUBE-PORTALS-HOST": exit status 3: modprobe: ERROR: could not insert 'ip_tables': Exec format error
iptables v1.6.0: can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
E0604 21:21:41.714695       1 proxier.go:274] Error flushing userspace chain: error flushing chain "KUBE-NODEPORT-HOST": exit status 3: modprobe: ERROR: could not insert 'ip_tables': Exec format error
iptables v1.6.0: can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
E0604 21:21:41.716310       1 proxier.go:274] Error flushing userspace chain: error flushing chain "KUBE-NODEPORT-CONTAINER": exit status 3: modprobe: ERROR: could not insert 'ip_tables': Exec format error
iptables v1.6.0: can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
E0604 21:21:41.717887       1 proxier.go:274] Error flushing userspace chain: error flushing chain "KUBE-NODEPORT-NON-LOCAL": exit status 3: modprobe: ERROR: could not insert 'ip_tables': Exec format error
iptables v1.6.0: can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
I0604 21:21:41.717983       1 server.go:484] Version: v1.13.6
I0604 21:21:41.725328       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I0604 21:21:41.725387       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0604 21:21:41.725495       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0604 21:21:41.725575       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I0604 21:21:41.725774       1 config.go:202] Starting service config controller
I0604 21:21:41.725796       1 controller_utils.go:1027] Waiting for caches to sync for service config controller
I0604 21:21:41.726415       1 config.go:102] Starting endpoints config controller
I0604 21:21:41.726434       1 controller_utils.go:1027] Waiting for caches to sync for endpoints config controller
I0604 21:21:41.826011       1 controller_utils.go:1034] Caches are synced for service config controller
I0604 21:21:41.826615       1 controller_utils.go:1034] Caches are synced for endpoints config controller
E0604 21:21:41.829212       1 proxier.go:666] Failed to ensure that filter chain KUBE-EXTERNAL-SERVICES exists: error creating chain "KUBE-EXTERNAL-SERVICES": exit status 3: modprobe: ERROR: could not insert 'ip_tables': Exec format error
iptables v1.6.0: can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
E0604 21:21:50.546403       1 proxier.go:666] Failed to ensure that filter chain KUBE-EXTERNAL-SERVICES exists: error creating chain "KUBE-EXTERNAL-SERVICES": exit status 3: modprobe: ERROR: could not insert 'ip_tables': Exec format error
iptables v1.6.0: can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
E0604 21:22:00.806724       1 proxier.go:666] Failed to ensure that filter chain KUBE-EXTERNAL-SERVICES exists: error creating chain "KUBE-EXTERNAL-SERVICES": exit status 3: modprobe: ERROR: could not insert 'ip_tables': Exec format error
iptables v1.6.0: can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
E0604 21:22:30.553263       1 proxier.go:666] Failed to ensure that filter chain KUBE-EXTERNAL-SERVICES exists: error creating chain "KUBE-EXTERNAL-SERVICES": exit status 3: modprobe: ERROR: could not insert 'ip_tables': Exec format error
iptables v1.6.0: can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.

What you expected to happen?

I expect kube-proxy to be deployed like it does with the same configuration when using the 0.16.0 release.

How to reproduce it (as minimally and precisely as possible)?

1. Start openshift-install
2. Once it is at INFO Waiting up to 30m0s for the Kubernetes API run the following to wait for the apiserver: while ! kubectl get nodes; do sleep 1; done
3. Disable the network-operator with (I'm not sure if this is necessary but it is part of my workflow since I'm testing another network operator and this is needed:
   kubectl patch --type=json -p "$(cat hack/overrides-patch.yaml)" clusterversion version kubectl scale deployment -n openshift-network-operator network-operator --replicas 0
4. Install kube-proxy using the following configMap and DaemonSet manifest:

apiVersion: v1
data:
  config.conf: |
    apiVersion: kubeproxy.config.k8s.io/v1alpha1
    bindAddress: 0.0.0.0
    clientConnection:
      acceptContentTypes: ""
      burst: 10
      contentType: application/vnd.kubernetes.protobuf
      kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
      qps: 5
    clusterCIDR: 10.128.0.0/14
    configSyncPeriod: 15m0s
    conntrack:
      max: null
      maxPerCore: 32768
      min: 131072
      tcpCloseWaitTimeout: 1h0m0s
      tcpEstablishedTimeout: 24h0m0s
    enableProfiling: false
    healthzBindAddress: 0.0.0.0:10256
    hostnameOverride: ""
    iptables:
      masqueradeAll: false
      masqueradeBit: 14
      minSyncPeriod: 0s
      syncPeriod: 30s
    ipvs:
      excludeCIDRs: null
      minSyncPeriod: 0s
      scheduler: ""
      syncPeriod: 30s
    kind: KubeProxyConfiguration
    metricsBindAddress: 127.0.0.1:10249
    mode: iptables
    nodePortAddresses: null
    oomScoreAdj: -999
    portRange: ""
    resourceContainer: /kube-proxy
    udpIdleTimeout: 250ms
  kubeconfig.conf: |
    apiVersion: v1
    kind: Config
    clusters:
    - cluster:
        certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        server: https://api.<cluster-name>.<dnsname>:6443
      name: default
    contexts:
    - context:
        cluster: default
        namespace: default
        user: default
      name: default
    current-context: default
    users:
    - name: default
      user:
        tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
kind: ConfigMap
metadata:
  name: kube-proxy
  namespace: kube-system

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-proxy
  namespace: kube-system
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kube-proxy
  template:
    metadata:
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ""
      creationTimestamp: null
      labels:
        k8s-app: kube-proxy
    spec:
      containers:
      - args:
        - --config=/var/lib/kube-proxy/config.conf
        command:
        - /usr/local/bin/kube-proxy
        image: k8s.gcr.io/kube-proxy:v1.13.6
        imagePullPolicy: Always
        name: kube-proxy
        resources: {}
        securityContext:
          privileged: true
          procMount: Default
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/kube-proxy
          name: kube-proxy
        - mountPath: /run/xtables.lock
          name: xtables-lock
        - mountPath: /lib/modules
          name: lib-modules
          readOnly: true
      dnsPolicy: ClusterFirst
      hostNetwork: true
      priorityClassName: system-node-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: kube-proxy
      serviceAccountName: kube-proxy
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoSchedule
        operator: Exists
      - effect: NoExecute
        operator: Exists
      volumes:
      - configMap:
          defaultMode: 420
          name: kube-proxy
        name: kube-proxy
      - hostPath:
          path: /run/xtables.lock
          type: FileOrCreate
        name: xtables-lock
      - hostPath:
          path: /lib/modules
          type: ""
        name: lib-modules
  templateGeneration: 1
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate

I hope the above are right, I dumped them from a cluster where I was observing this issue.

Anything else we need to know?

I have also tried with a binary I built from master and saw the same results.

References

@caseydavenport FYI

[caseydavenport]

I think this is at least part of what's going on here: https://github.com/kubernetes/kubernetes/issues/71305

[DanyC97]

I think this is at least part of what's going on here: kubernetes/kubernetes#71305

it may not be the case since OP's log show iptables v1.6.0 while in the k/k issue folks are talking about iptables >= 1.8.

[caseydavenport]

I think the logs in OP are getting emitted from within the kube-proxy container, which is using v1.6.0.

Running iptables -V on the host shows a version of v1.8.2 when I run it using the v4.1.0 release.

[squeed]

RHCOS uses iptables in nftables mode. That is to say, the iptables binaries exist on disk, but they don't use the iptables subsystem in the kernel. Rather, it translates to nftables internally.

The way to fix this is to bind-mount the host's filesystem and use the host's iptables. This is a hack, but there's no good way around it. See https://github.com/openshift/origin/tree/master/images/sdn/scripts for how we do this.

We're working on getting an openshift-specific kube-proxy image that includes this hack, but it will take some time.

[tmjd]

As a follow up. I've tried out the image quay.io/openshift/origin-kube-proxy:4.1 after seeing https://github.com/openshift/origin/pull/22155 and https://github.com/openshift/release/pull/3711.

The comment https://github.com/openshift/origin/pull/22155#issuecomment-476749320 suggests that it is for Calico but I wanted to report that it does not work for an install with 4.1.2.

The logs from this version of kube-proxy show the following and indicate that iptables 1.4.2 is being used.

E0625 18:23:05.161594       1 proxier.go:1350] Failed to execute iptables-restore: exit status 2 (iptables-restore v1.4.21: Couldn't load target `KUBE-MARK-DROP':No such file or directory

Error occurred at line: 286
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
)
E0625 18:23:35.061223       1 proxier.go:1350] Failed to execute iptables-restore: exit status 2 (iptables-restore v1.4.21: Couldn't load target `KUBE-MARK-DROP':No such file or directory

Error occurred at line: 294
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
)
E0625 18:24:05.081387       1 proxier.go:1350] Failed to execute iptables-restore: exit status 2 (iptables-restore v1.4.21: Couldn't load target `KUBE-MARK-DROP':No such file or directory

Error occurred at line: 160
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
)

[squeed]

Yup, there was a bug in the dockerfile that snuck in. We need https://github.com/openshift/origin/pull/23235 to merge.

[abhinavdahiya]

closing due to inactivity, please reopen if needed.

/close

[openshift-ci-robot]

@abhinavdahiya: Closing this issue.

In response to this:

closing due to inactivity, please reopen if needed.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.