Amazon-vpc-cni-k8s: hostPort Not Working with Deployments

[brockoffdev]

Issue I've run into with EKS...we launch our Vault containers utilizing a hostPort for a number of reasons, the biggest being due to our reliance on Proxy Protocol headers and the ELB's health check to balance the standby Vault containers in an HA setup.

Because of this, we need to bring up the containers as follows:

      - name: vault
        image: vault:0.10.2
        command:
          - vault
          - server
          - -config=/vault/config/config.hcl
        ports:
        - containerPort: 8200
          hostPort: 8200
          name: vaultport
          protocol: TCP
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: VAULT_API_ADDR
          value: https://[HTTPS Address]
        readinessProbe:
          failureThreshold: 3
          tcpSocket:
            port: 8200
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          failureThreshold: 3
          tcpSocket:
            port: 8200
          initialDelaySeconds: 15
          periodSeconds: 20
        volumeMounts:
        - name: config-volume
          mountPath: /vault/config/config.hcl
        - name: vault-audit-logs
          mountPath: /vault/log  
        resources:
          requests:
            cpu: 250m
            memory: 256Mi
          limits:
            cpu: 500m
            memory:  1024Mi
        securityContext:
          readOnlyRootFilesystem: true
          capabilities:
            add:
              - IPC_LOCK

Unfortunately, this does not work, as the hostPort is never created. Additionally, setting the securityContext to "privileged" also does not work (OS: Amazon Linux 2).

Any thoughts on this? I think it may be related to the EKS VPC CNI, but I'm still trying to debug....

[brockoffdev]

Going to close this issue, but want to provide some information for those looking 😄.

This is built on top of the CNI Plugin in Kubernetes on EKS (makes sense), but unfortunately a downside of that is hostPort definitions are ignored.

There is a way to work around this, however. You can specify hostNetwork: true on your deployment to allow for these pods/containers to utilize the host network to direct traffic directly to the container (and open up the proper host port). Obvious problem here is that, essentially, the pod/container is somewhat separated from the rest of the Kubernetes ecosystem as it no longer runs on the overlay network itself. As an example of this, utilizing hostNetwork: true and kube2iam together is not doable.

[igorvpcleao]

Related: #153