Eksctl: Custom subnet topologies

[chrisdrobison]

Why do you want this feature?
We have to use an existing VPC and it was created with two CIDR blocks and all the subnets (9 of them across 3 AZs in our case) created are balanced between the two CIDR blocks. Yes, it was over engineered, but all routing is setup correctly.

What feature/behavior/change do you want?
Right now I get an error saying that "subnet ID x is not the same as y". It would be nice, especially for cases where existing VPCs have to be used to be able to tell eksctl to just assume everything is okay and proceed at my own risk.

Here is the command I use:

eksctl.exe create cluster --name=cluster1 --asg-access --full-ecr-access --alb-ingress-access --vpc-private-subnets=subnet-1,subnet-2,subnet-3,subnet-4,subnet-5,subnet-6,subnet-7,subnet-8,subnet-9 --node-private-networking

[errordeveloper]

Could you please detail your configuration more specifically, we are looking at way to extend this functionality. Do you have a cloudformation template that you can share, for example?

[chrisdrobison]

Keep in mind this was setup for us. We have a VPC with two CIDR blocks a 10.x.x.x/22 and 10.x.x.x/23. In the us-west-2 region, we have created a set of 3 subnets in 3 of the availability zones, for a total of 9 subnets. The original though behind this design was to categorize types of workloads into separate highly available subnets. For example, 3 of subnets (one in each AZ) are labeled as web subnets, another 3 are labeled for database, etc. Also, we have a direct connect setup with AWS so these IP blocks need to mesh with our on-prem network as well. Does this clarify?

[errordeveloper]

Thanks, this is very helpful. Would you be able to share a CloudFormation
template or something else that we could use in attempt to reproduce this
setup? Do you have 3 classes of subnets or just 2?

We will review this and see how we can accommodate this use-case. It's hard
to say when it would be possible to implement it. We are looking into a few
use-cases, and adding dedicated networks and security groups for pods is of
interest, but that is a little different.

For the time being, you should be able to import most of your subnets. You
can try to pass one of the sets under 'public' and the other under
'private' (albeit both are actually private, it should be okay to try it
that way just as an experiment). To use public one you will need to use
'privateNetworking: false' for your nodegroup, of course, which should only
select the set of subnets and won't affect anything significantly (there is
one minor detail - the SSH port on SG would be open for any source address,
but it'd be okay to ignore for the sake of the experiment).

On Mon, 3 Jun 2019, 10:34 pm Chris Robison, [e-mail]
wrote:

Keep in mind this was setup for us. We have a VPC with two CIDR blocks a
10.x.x.x/22 and 10.x.x.x/23. In the us-west-2 region, we have created a set
of 3 subnets in 3 of the availability zones, for a total of 9 subnets. The
original though behind this design was to categorize types of workloads
into separate highly available subnets. For example, 3 of subnets (one in
each AZ) are labeled as web subnets, another 3 are labeled for database,
etc. Also, we have a direct connect setup with AWS so these IP blocks need
to mesh with our on-prem network as well. Does this clarify?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/weaveworks/eksctl/issues/806?email_source=notifications&email_token=AAB5MS2XTU2V6UH73BXDNQDPYWE5HA5CNFSM4HOCXYPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW2YH5Q#issuecomment-498435062,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB5MS773N3JWTQ3Z66JAELPYWE5HANCNFSM4HOCXYPA
.

[chrisdrobison]

Unfortunately I do not have the cloud formation template nor does it look like a CF template was used. We have 3 classes of subnets (app, db, web). I will experiment with those settings.

[errordeveloper]

@chrisdrobison thanks, this is crystal clear now. Not to worry about reproducing this. I assume that you'd be looking to map each class of subnets to a nodegroup and have nodes as well as pods on those subnets, correct? It's possible to isolate pods from nodes via SGs, and even via subnets, but you might have to define more subnets. In any case, our plan is to add custom subnet topologies, right now we have 'public' and 'private', and we are looking to extend that to allow custom classes of subnets.

[chrisdrobison]

Yes, we do want to map those subnets to nodegroups.

[ghost]

We are also looking for this feature although our use case is a little different. We need private subnets for internal worker nodes, "semi" private subnets for load balancers facing different types of VPNs, as well as public subnets for Internet facing load balancers.

[itlinux]

I would like to see how to include an external lb to replace the standard elb from aws.

Thanks

Il giorno 13 giu 2019, alle ore 12:24, jpvz [e-mail] ha scritto:

We are also looking for this feature although our use case is a little different. We need private subnets for internal worker nodes, "semi" private subnets for load balancers facing different types of VPNs, as well as public subnets for Internet facing load balancers.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[errordeveloper]

To be clear, what we are going to do is what @amarrella proposed in https://github.com/weaveworks/eksctl/issues/475#issuecomment-498995479.

[errordeveloper]

I would like to see how to include an external lb to replace the standard elb from aws.

@itlinux this sounds like a completely different use-case, please open another issue with some details of what is it that you'd like to achieve, and why you think it's something that we should cater for.

[lacatus]

hey, came across same issue, is this planned to be covered at any time? thanks

[leepatrick-goop]

+1 for this feature! also if it was possible to specify multiple nodegroups in the yaml to use specific subnets.

[SeanTobin]

So to add a usecase for this - Our cluster is starting to outgrow our existing subnet. We provisioned larger subnets and assumed we could just redeploy our nodegroups to the new subnets. Sadly, for the reasons in this thread, that was not possible.

+1 for this as it looks like it would solve our need

[lwimmer]

+1 for this too. We have multiple private subnets because they have different route tables.
The current eksctl (0.13.0) has no support for multiple private subnets (per AZ) sadly.

[lwimmer]

I have added support for custom subnets here:
https://github.com/lwimmer/eksctl/commit/6e673d049bd48ae2e02cd41b1521af6422dbec2b
This is for sure not PR-ready, but works for us.

You can define custom subnets like this:

vpc:
  subnets:
    custom:
      my-custom-subnets:
        eu-central-1a:
          id: subnet-123
        eu-central-1b:
          id: subnet-234

and use them in a nodegroup like this:

nodeGroups:
  - name: test
    customSubnets: my-custom-subnets

Definitely missing:

• CLI support
• Support for creating the custom subnets(?)
• Documentation
• Tests

Maybe this is helpful for others too.

[dougbyrne]

Hi, we also have a cluster that outgrew our subnets. I think my prefered solution would be to specify a subnet ID in my nodegroup definition, instead of just the availability zone.

As a workaround, I've manually edited vpc zone identifier in the cloudformation template generated for a nodegroup.

[connorearl]

I also need this to use a separate subnet for nodes with just Istio egress gateways on it. We don't want to open egress to our VPN to all nodes on the corporate firewall side, so I need a separate subnet just for them.

[michaelbeaumont]

Support for this is now in master and will be released at the end of the week! Please test it out and verify it works for your use cases!