Amazon-vpc-cni-k8s: Using public subnet always binds private address

[mathewmoon]

Using version 1.3.3 with `AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=trueand the following ENIConfigs:

Name:         us-east-1a
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  crd.k8s.amazonaws.com/v1alpha1
Kind:         ENIConfig
Metadata:
  Creation Timestamp:  2019-04-03T04:20:00Z
  Generation:          1
  Resource Version:    11851842
  Self Link:           /apis/crd.k8s.amazonaws.com/v1alpha1/eniconfigs/us-east-1a
  UID:                 bf18a586-55c7-11e9-bb0e-0e1271346d4a
Spec:
  Security Groups:
    sg-12345678
  Subnet:  subnet-12345678
Events:    <none>


Name:         us-east-1b
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  crd.k8s.amazonaws.com/v1alpha1
Kind:         ENIConfig
Metadata:
  Creation Timestamp:  2019-04-03T04:20:00Z
  Generation:          1
  Resource Version:    11851845
  Self Link:           /apis/crd.k8s.amazonaws.com/v1alpha1/eniconfigs/us-east-1b
  UID:                 bf1e9173-55c7-11e9-bb0e-0e1271346d4a
Spec:
  Security Groups:
    sg-12345678
  Subnet:  subnet-12345678
Events:    <none>


Name:         us-east-1c
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  crd.k8s.amazonaws.com/v1alpha1
Kind:         ENIConfig
Metadata:
  Creation Timestamp:  2019-04-03T04:20:00Z
  Generation:          1
  Resource Version:    11851822
  Self Link:           /apis/crd.k8s.amazonaws.com/v1alpha1/eniconfigs/us-east-1c
  UID:                 bf1228ab-55c7-11e9-bb0e-0e1271346d4a
Spec:
  Security Groups:
    sg-123456787
  Subnet: subnet-12345678
Events:    <none>

Kubectl describe node shows the proper annotation:

$ kubectl describe node ip-10-0-185-255.ec2.internal|grep Annotations
Annotations:        k8s.amazonaws.com/eniConfig=us-east-1b

Expected Outcome

Pod gets public IP from public subnet
No error in log

What happens instead

I see this error in the aws-node pod, but only on worker nodes that use the custom ENIConfig (Second line):

=====Starting installing AWS-CNI =========
=====Starting amazon-k8s-agent ===========
ERROR: logging before flag.Parse: W0403 13:42:15.659234      13 client_config.go:533] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2019-04-03T13:42:15Z" level=error msg="failed to initialize service object for operator metrics: OPERATOR_NAME must be set"

Pod gets private address instead of public address even though the subnet is public. Is this expected behavior? Is it possible to get a public address assigned to the pod?

Looking at IPamd logs and such mounted inside of the container has not helped. This error doesn't occur on worker nodes that are not annotated to use a custom ENIConfig.

[joseph-wortmann]

The idea here, as I'm understanding it, is to have a private subnet node that has pods that receive ENIs and public addresses from a public subnet.

However, bare ENIs created in a public subnet do not automatically get public addresses assigned to them.

I think for this to work the CNI plugin will have to be updated to have a flag such as ASSIGN_PUBLIC_ADDRESS=true and then do the following:

1. Create an EIP for the ENI
2. Create or obtain ENI from the pool
3. Associate the EIP to the ENI
4. Assign the ENI to the pod

I would like this capability too. Keeping my nodes private and unaddressable publicly while having the pods on that node publicly addressable is desirable from a security perspective.

[mathewmoon]

@mogren
Do you know if this this expected behavior?

[thinalai]

The idea here, as I'm understanding it, is to have a private subnet node that has pods that receive ENIs and public addresses from a public subnet.

However, bare ENIs created in a public subnet do not automatically get public addresses assigned to them.

I think for this to work the CNI plugin will have to be updated to have a flag such as ASSIGN_PUBLIC_ADDRESS=true and then do the following:

1. Create an EIP for the ENI
2. Create or obtain ENI from the pool
3. Associate the EIP to the ENI
4. Assign the ENI to the pod

I would like this capability too. Keeping my nodes private and unaddressable publicly while having the pods on that node publicly addressable is desirable from a security perspective.

The current AWS CNI proposal only use secondary IPs of the ENI, and the primary IP is used for pods accessing the external network by SNAT to it.

To achieve this, there must be:

1. One ENI with One Pod.
   Associate EIP with ENI, then EIP will SNAT/DNAT to the primary IP of the ENI used by the Pod.
2. Associate EIP with ENI with IP specified.
   EIP will configure SNAT/DNAT with the IP specified. SNAT from the IP and DNAT to the IP.

[mathewmoon]

I am not sure if this is even possible. I didn't realize until digging in further that the public IP address of an ENI is actually not assigned directly to the VM. EG: That address is not bound to an interface in /dev. It looks like public addresses are actually just a form of NAT. I may be wrong about this though. We got around this issue by running an application pod that monitors the kube API for pod events then gets their metadata to find the host node and gets its public IP address. We then call the AWS API to register the DNS. The pods we are monitoring have their ports bound with hostPort. The only downside is you cannot bind that same port with multiple pods.

[mogren]

@mathewmoon Is this still an issue?

[mathewmoon]

@mogren As far as I know it is. I wrote this to work around it though:
https://hub.docker.com/r/quinovas/aws_kube_public_dns

It listens for a pod to be created and then creates a route53 record for the host that the pod is running on. The pod has to be using HostPort. Hope this helps.

[mogren]

Thanks @mathewmoon. Since this CNI doesn't support having one ENI per pod, I'm closing this issue.

The Container Roadmap has some details on the next generation CNI https://github.com/aws/containers-roadmap/issues/398

[devkid]

We have written a controller for allocating and assigning EIPs to pods: https://github.com/LogMeIn/k8s-eip-controller