Eksctl: Specify additional security group inbound rules in cluster-config

[aparamon]

Eksctl either creates a dedicated VPC or can reuse existing VPC and attach a populated security group from that VPC to a node group: https://github.com/weaveworks/eksctl/pull/412.

It is wished that additional security group can be created and attached to a node group, with rules populated from config-file. A quick cluster with moderate security level (ingress/load balancer endpoint only accessible from "safe" hosts) can then be created with a single eksctl command.

The proposed syntax is something like
````yaml
nodeGroups:

• name: public-workers
  privateNetworking: true
  securityGroups:
  
  
  
  • public:
    
    inboundRules:
    
    - type: All traffic
    
    protocol: All
    
    portRange: All
    
    source: 12.34.56.0/24
    
    description: Corporate hosts
    
    - type: HTTP
    
    protocol: TCP
    
    portRange: 80
    
    source: 34.56.78.0/24
    
    description: Service users
  
  

[errordeveloper]

Thanks! Something like this was foreseen, but we've not had a specific request yet, so thanks a lot for putting one in.

[errordeveloper]

The umbrella issue is #448.

[aparamon]

Thanks @errordeveloper !
One crazy thing about the config I sketched is the use of privateNetworking: true. It's greatly confusing as in reality public network access is intended. The human-friendly way of config specification is yet to be found.

[errordeveloper]

If you have the time, please consider submitting a design proposal, we have
some examples in 'docs'. Once we have a PR, we can discuss field names etc! :)

[aparamon]

@errordeveloper Thank, will try to! :-)

One more preliminary question: I noticed that new eksctl puts private nodegroup nodes into both group-specific "private" security group, and also into all-nodes "shared" security group. Is there a reason to keep it this way, or modifying "shared" security group to include rule for communication to control plane and removing "private" security group will work equally well?
Maybe keeping "private" security group is a good idea (to enable granular per-group access to some resources like EFS), but what about having its rules empty for new privateNetworking: true nodegroups? This should simplify stuff.

[mcblair]

If this functionality is implemented, am I correct in saying this will allow us to limit access to the public EKS API without making it private?

Related:

• https://github.com/aws/containers-roadmap/issues/108
• https://github.com/aws/containers-roadmap/issues/221

[errordeveloper]

I suppose it could. As one should be able to amend control plane SG ingress rules in the same way they can amend node SG ingress rules. This issue refers to node ingress, but certainly same config fields can be added for control plane too.

[blakestoddard]

Is there any movement in getting custom security group support added?

cc @jicowan

[ImIOImI]

I would REALLY like to attach pre-existing security groups during the creation process

[f-ld]

I think the complete solution would be:

• a separate section (like the VPC one) to create security groups like described by @aparamon (and give them names for later reference), like one called "AdminSSHAccess" giving access to port 22 from some source, one called "PublicHTTPS" configured to give access to port 443 from 0.0.0.0/0, and all the groups one may need
• and in each node group section have the ability to indicate security groups created above from their given name (ID not known) or pre-existing security groups from their ID (as requested by @ImIOImI )

So this could look like this:
```yaml
securityGroups:

• name: AdminSSHAccess
  inboundRules:
  
  
  
  • type: SSH
    
    protocol: TCP
    
    portRange: 22
    
    source: 12.34.56.78/32
    
    description: OPsVPN1
  
  
  • type: SSH
    
    protocol: TCP
    
    portRange: 22
    
    source: 78.56.34.12/32
    
    description: OPsVPN2
  
  
• name: PublicHTTPS
  
  
  
  • type: HTTP
    
    protocol: TCP
    
    portRange: 443
    
    source: 0.0.0.0/0
    
    description: Public access to service
  
  

nodeGroups:

• name: public-workers
  privateNetworking: true
  securityGroups:
  ids:
  - sg-abcdef012345
  refs:
  - AdminSSHAccess
  - PublicHTTPS
  ```

[pjain287074]

nodeGroups:

• name: public-workers
  privateNetworking: true
  securityGroups:
  ids:
  
  
  
  • sg-abcdef012345
    
    ```
    
    above code is not working for me as i am getting error "cannot unmarshal array into Go struct field NodeGroup.securityGroups of type v1alpha5.NodeGroupSGs"
    
    This is something what i have tried
  
  

nodeGroups:

• name: POC-ng
  instanceType: t3.micro
  privateNetworking: true
  securityGroups:
  ids:
  - sg-abcdef012345
  desiredCapacity: 1
  minSize: 1
  maxSize: 1
  availabilityZones: ["eu-central-1a", "eu-central-1b", "eu-central-1c"]
  ami: auto
  volumeSize: 10
  volumeType: gp2
  ssh:
  allow: true
  publicKeyPath: dt-env
  iam: {
  withAddonPolicies: {
  autoScaler: true,
  imageBuilder: true,
  externalDNS: false
  }
  }

[ear-dev]

I would like to add a security group to the control plane during cluster creation. This would allow us to create clusters with private access only, and to have a workstation inside the VPC where we manage our deployments with kubectl, etc.. Is this on the roadmap? Thanks.....

[atheiman]

yes we would also like what @ear-dev is describing. We would like other nodes in our VPC and peered VPC to communicate with the eks api server. It is a security violation in many organizations to have to route all traffic to the eks api server over the internet to the public api endpoint.

[spjmurray]

Likewise, I need to install security group rules so I have L3 connectivity between pods in different peered clusters. It's a bit of a pain to have to retroactively add them manually to all the nodes. Obviously I can just manually spin up the VPC, but it's nice having it all automated.

Obviously not your problem but the end goal -- for completeness -- is to use kubernetai to do DNS replication between the two clusters enabling private (we already support doing it over the public internet with external-dns, but you know customers...) TLS communication between pods in different geographic locations to facilitate application replication/backups; something made particularly challenging by Kubernetes, but doable at least!

[p0bailey]

Just wondering whether would be possible to declaratively attach a pre existing SG to a nodegroup. I found some examples around the web but unsure whether the feature is in place into eksctl. Had a look into the repo but I couldn’t find anything useful.

nodeGroups:
  - name: ng-1
    instanceType: t2.micro
    desiredCapacity: 1
    securityGroups:
      attachIDs: [sg-XXXXXXXXXXXXX] 

[cPu1]

Just wondering whether would be possible to declaratively attach a pre existing SG to a nodegroup. I found some examples around the web but unsure whether the feature is in place into eksctl. Had a look into the repo but I couldn’t find anything useful.

nodeGroups:
  - name: ng-1
    instanceType: t2.micro
    desiredCapacity: 1
    securityGroups:
      attachIDs: [sg-XXXXXXXXXXXXX] 

@p0bailey that's the correct config for passing custom SGs to a nodegroup. If you use eksctl create nodegroup -f config.yaml on that file, it should attach your specified SGs to the nodegroup. I should note that nodegroups are immutable, so if you're trying to attach the SGs to an existing nodegroup, you'll have to drain and delete the nodegroup, and create a new one with the new config.

[p0bailey]

Thanks a lot! @cPu1

This would be quite useful to be added to the current documentation.

[MeloHo]

Just wondering whether would be possible to declaratively attach a pre existing SG to a nodegroup. I found some examples around the web but unsure whether the feature is in place into eksctl. Had a look into the repo but I couldn’t find anything useful.

nodeGroups:
  - name: ng-1
    instanceType: t2.micro
    desiredCapacity: 1
    securityGroups:
      attachIDs: [sg-XXXXXXXXXXXXX] 

@p0bailey that's the correct config for passing custom SGs to a nodegroup. If you use eksctl create nodegroup -f config.yaml on that file, it should attach your specified SGs to the nodegroup. I should note that nodegroups are immutable, so if you're trying to attach the SGs to an existing nodegroup, you'll have to drain and delete the nodegroup, and create a new one with the new config.

@cPu1 If eksctl created a dedicated VPC and this security group is in another VPC, would this attachment be successful? Or to use this SG, the same VPC should be used?

[cPu1]

@cPu1 If eksctl created a dedicated VPC and this security group is in another VPC, would this attachment be successful? Or to use this SG, the same VPC should be used?

@MeloHo It'll not be successful. The SG should exist in the same VPC.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.