Installer: RFE: Support for install in existing VPC

[pierreozoux]

During today's call with openshift 4.0 HTB this was mentioned this wasn't currently possible.
This is a blocker for our enterprise env.

[abhinavdahiya]

we are already tracking this internally https://jira.coreos.com/browse/CORS-912

some important things from that discussion:

• the current plan is to just allow a user to specify VPC CIDR. This ensures that a user can launch a cluster that would not conflict with existing infrastructure (and could be peered as well). But the automated installation is assumed to create/own the VPC and related resources.
• also the destroy functionality does not currently allow for sharing the VPC and therefore, we had to drop the capability for now.

[wking]

One possible mitigation for folks who want to tie an OpenShift cluster into an existing VPC would be to create the OpenShift cluster in a separate AWS account and use VPC sharing to connect it to the pre-existing VPC.

[wking]

... use VPC sharing to connect it to the pre-existing VPC.

Hmm, I actually don't see how that would work. Maybe folks were talking about VPC peering.

[pierreozoux]

I am in an Enterprise context.
Already each project gets their own account with root (in the best case).
You get this account with no Internet access. And you have only one special VPC that is connected to Intranet. This comes with an IP range. And even private IP are not infinite. So requesting a bigger range, or another range in an Enterprise context, it is already challenging.
So I advise RedHat to research how their clients are using VPC in AWS, and I think my Enterprise is quiet common (but maybe I'm wrong).

Then in the solution space. I understand that this is challenging to keep track of resources that are owned by OpenShift and resources that are not. That's why people have been using CloudFormation.
In a privious project where I used kops, I actually didn't trust it to do this job well, so I created each cluster in a new VPC (this was in a more startup environment).
But currently, I'm using the Amazon quickstart for OpenShift, and thanks to CloudFormation, I'm confident to deploy all the clusters in the same VPC.

[wking]

Already each project gets their own account with root (in the best case).

You mean "team" or something, right? Not OpenShift project?

You get this account with no Internet access. And you have only one special VPC that is connected to Intranet.

This comes with your AWS account? And despite having account-root, you are not allowed to create new VPCs?

This comes with an IP range.

IP ranges should not be a problem; install-config already supports install-time configuration for those.

So I advise RedHat to research how their clients are using VPC in AWS, and I think my Enterprise is quiet common (but maybe I'm wrong).

Collecting this information is what issues like this are for :).

Then in the solution space. I understand that this is challenging to keep track of resources that are owned by OpenShift and resources that are not. That's why people have been using CloudFormation.

CloudFormation is a tool for user-managed infrastructure. It's great for that, but is unnecessary for cluster-managed infrastructure. And I don't see how it would help synchronize with the boundary between cluster-managed and user-managed resources. Wouldn't that have to be something negotiated out of band which fed into the OpenShift installer and CloudFormation configuration?

But currently, I'm using the Amazon quickstart for OpenShift, and thanks to CloudFormation, I'm confident to deploy all the clusters in the same VPC.

From those docs:

The Quick Start includes AWS CloudFormation templates that build the AWS infrastructure using AWS best practices, and then pass that environment to Ansible playbooks to build out the OpenShift environment. The deployment provisions OpenShift master instances, etcd instances, and node instances in a highly available configuration.

This is the 3.x approach (and also a possible 4.x bring-your-own-infrastructure approach). In a cluster-managed 4.x approach, the installer bootstraps the masters and etcd, and the cluster itself creates and manages the nodes (and many other resources). We have long-term plans to have cluster-managed masters and etcd instances too, but those are currently blocked by the complexity of automatically managing the Kubernetes etcd cluster (e.g. authorizing new etcd peers). Once we have that, there will be very little left for the installer to create during the bootstrapping phase.

If you want cluster-managed infrastructure (with all its benefits, like the ability to configure the cluster to automatically upgrade itself as RHCOS and other OpenShift components are updated), then I don't see much benefit to being able to stuff multiple clusters into the same VPC. From their docs, it looks like multiple clusters share VPCs and subnets, but it's not clear to me what else they share. Probably NAT gateways and internet gateways. Probably not load balancers ;). Maybe security groups depending on how tightly you wanted to sandbox your clusters vs. lumping them into the same trust bucket. It feels like there are many possible choices for how you'd want to set that up. And it feels simpler and equally powerful to create clusters in isolated VPCs and then to use VPC peering to tie together clusters that you do want to put into the same trust bucket (or, with some custom security groups, managed trust between the peered VPCs). Do you have use cases where that would not work?

[aaronlevy]

Hi @pierreozoux

I wanted to dig a bit more into the constraints you mentioned.

In one case, it sounds like a disconnected/offline/private mode is a feature you'd like to see supported (e.g. the new cluster has no external internet access and can only retrieve assets from an intranet that already exists in a VPC). Supporting these types of "disconnected" use-cases are on the backlog - but unfortunately not something that is currently available.

That being said, assuming that feature did exist, I'm still curious about the VPC constraints. You mention that you're using the quickstart for OpenShift, and allowing CloudFormation templates to create clusters within your existing VPC. It sounds like making modifications to AWS assets is something under your control - so I'm assuming that the fact that the installer is "creating" VPCs isn't the problem unto itself (compared to maybe, "I am not allowed to create VPCs - I have to request them from another team").

With that assumption:

• Would VPC peering be a viable option in your use-case? For example, when you run the openshift-installer, it would generate a new VPC dedicated to the OpenShift cluster, and then peer that VPC with your "intranet" VPC?

  I ask about this, as peering would allow you to still address "intranet" assets as if they were on the same network, while keeping OpenShift cluster assets all under their own dedicated VPC (and managed by the cluster, and not externally/manually).

  This is fairly speculative, but I'm curious about the viability of an option like peer-to-vpc as an install-time option, vs use-existing-vpc. As the latter has a number of downsides in automated-management cases. Namely that it is difficult / impossible to determine safety of operations as ownership is shared - and there is more than just VPC assets involved: subnets, security groups, gateways, etc.

• Alternatively, if that option is not viable - would you be able to expand more on some of the reasoning / constraints (e.g. how peering fails for this use case).

At this stage I can't say the specific direction we would go - but want to gather more specific use-cases / constraints so we can better inform our decisions.

[pierreozoux]

Hi,

I had a lengthy conversation with the network team.
We are a company with 300k employees with 100s of AWS accounts and datacenters.

Basically, for security and configuration point of view it is better to have a 1 to 1 relation between accounts and VPCs:

• packets have to go to a central point for security reasons
• they need to manage IP ranges from a central point to avoid overlaps

That's why we will always have a 1to1 relation between AWS accounts and VPC. And I think every Enterprise customers will.

[wking]

Basically, for security and configuration point of view it is better to have a 1 to 1 relation between accounts and VPCs...

That doesn't rule out "create a new AWS account for OpenShift and use VPC peering to connect it to other internal VPCs".

But if, for some reason, there's a constraint on account creation as well, which other resource types would you provide alongside? Subnets? Security groups? NAT gateways? Internet gateways? DHCP options? Route 53 zones? Route 53 records? Load balancers? Some of those (Route 53 records and load balancers) have some instances created by the installer for the Kubernetes API, some created by the ingress operator for the web console, and some created for Kubernetes LoadBalancer objects.

[aaronlevy]

Thanks for following up on this @pierreozoux.

To expand a bit on what @wking mentioned:

If we work from the constraint that an existing VPC will be used, we still must solve for all of the other resource types that are involved. @wking mentioned some of these, for example multiple subnets will need to be created (or already exist) within the VPC for exclusive use of the OpenShift cluster. Security groups that conform to the OpenShift cluster expectations will need to be created (or already exist) & attached to those subnets. Etc.

A consideration that we've been exploring on our side is where does the "use existing resources" constraint begin/end for various customers. I suspect that this will vary between use-cases. For some, we can create all of those mentioned resources, for others, we can manage none of them.

We will end up needing to target various use-cases on the spectrum of "fully-managed infrastructure --> customer controlled infrastructure". The former allows OpenShift the ability to provide out-of-box best-practice clusters, and on going automated management of that infrastructure. The latter allows users flexibility to control all aspects of their infrastructure - but the user is responsible for ensuring ongoing conformance to OpenShift cluster infrastructure expectations.

Neither of these ends of the spectrum is right/wrong in enterprise (or other) contexts, just targeting different needs. Personally, I would lean toward dedicated resources for the OpenShift cluster as a means of increasing safety / security (isolated, easier to audit, no concerns of shared-owner messing up required resources). I would also greatly value the automated management of those components. But I also fully understand that there are going to be highly custom or constrained use-cases that this path might not address.

@pierreozoux for a couple follow up questions to hopefully get a better idea of what might work for your use-cases:

they need to manage IP ranges from a central point to avoid overlaps

We allow the selection of the VPC CIDR that will be used, so this should be supported today. When you peer VPCs there can't be overlap between the ranges being used, so selection of this network range is important even in the case that we're creating a dedicated VPC. This may alleviate this as a blocker if the organization is already used to creating 100s of separate accounts/VPCs?

packets have to go to a central point for security reasons

Does this mean that the cluster can have no outbound internet traffic (e.g. from within the cluster curl google.com won't work)? Or just that all packets are routed through a particular gateway before going to public internet?

In the former, this is somewhat of a separate discussion as I mentioned (we don't yet support offline clusters). In the latter, peering VPCs might still solve for this, or possibly even a separate discussion about cluster proxies (would need more detail). But assuming re-using VPC did solve for this - couple follow-up questions below:

• In your re-use existing VPC use-case, would you prefer (or is it required):

  • All sub-resources are controlled/created by you as well? For a (small) example, we would need there to be a subnet created that matches the expected Pod CIDR for exclusive use within the cluster (for all pod-pod network traffic).

  Or

  • The installer would still create all sub-resources, but you would just provide a VPC ResourceID for those assets to target?

[DanyC97]

@aaronlevy any timeline ( i know i might asking too much at this stage) when the _we don't yet support offline clusters_ use-case might get a bump in priority ? maybe 4.1 or 4.2 cycle?

[aaronlevy]

@DanyC97 Unfortunately, I don't have a timeline to share. At this stage I would suggest reaching out to your account manager so that they are aware that you have this feature request. It also would be helpful to include as many specifics of your needs/use-case as you can. For example, "all traffic must go through corporate proxy first" VS. "there is no internet connectivity" - generally are going to end up with different needs/solutions/potentially timelines.

[simongun]

If I may add additional reasons that my company has to install OCP 4.x in existing VPCs

• Pre-existing clusters & corresponding firewall rules
  In additional to application layer security we have coarse grained firewall (FW) rules in place that limit access to and from an OCP cluster. Re-configuring all these FWs for new clusters would be a huge overhead and complicate our migration from 3.x to 4.x massively.
• Need to have control over account's networking & security
  It is not acceptable for us to have the accounts networking & network security configuration generated by OCP. For one, we need to control this configuration. Secondly, there is a very high likelyhood that our own configuration will deviate from OCPs one.

[wking]

Pre-existing clusters & corresponding firewall rules...

So do you also expect to control master security groups? Load-balancer security groups? The control-plane load-balancers themselves? The ingress/service load balancer? And so on. It's currently not clear to me how much overlap there is for these associated resources between the various users who want existing-VPC integration, and it would help a lot to have explicit, exhaustive lists of the resources you expect to provide to the installer.

[wking]

In case it helps ground resource-owner lists, you can get a list of all non-bootstrap resources removed after a recent CI run with:

$ curl -s https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_installer/1192/pull-ci-openshift-installer-master-e2e-aws/3434/artifacts/e2e-aws/installer/.openshift_install.log | grep 'level=info.*\(Deleted\|Disassociated\|Released\)'
time="2019-02-05T06:46:26Z" level=info msg=Deleted arn="arn:aws:ec2:us-east-1:460538899914:natgateway/nat-04d5d8506ac5118b7" id=nat-04d5d8506ac5118b7
time="2019-02-05T06:46:26Z" level=info msg=Deleted arn="arn:aws:ec2:us-east-1:460538899914:natgateway/nat-06092d67f87a7c815" id=nat-06092d67f87a7c815
time="2019-02-05T06:46:26Z" level=info msg=Deleted arn="arn:aws:ec2:us-east-1:460538899914:natgateway/nat-061a451e0be3ef30a" id=nat-061a451e0be3ef30a
...

Then decide whether you expect to create and manage the NAT gateways (and so on, for the elided results of that grep as well), or whether you expect the installer and OpenShift cluster to create and manage them for you.

[simongun]

Reg the various security groups, we would need to be able to control the ones where users directly interact with it.

• Assume there is an LB in front of Master --> we don't need to control it
• LBs : definitely need to control this
• control plane LBs: dont' yet fully understand their archtecture, probably not
• ingress/service load balancer: definitely need to control this

exhaustive lists of the resources you expect to provide to the installer.
Will get back to you on that.

[wking]

Reg the various security groups, we would need to be able to control the ones where users directly interact with it.

• Assume there is an LB in front of Master --> we don't need to control it

This sort of thing depends on who you're classifying as "users". Folks who run oc (e.g. devs creating new projects, launching new pods within projects, etc.) will need access to this load balancer for the Kubernetes/OpenShift API.

There's a separate load balancer managed by the ingress operator fronting the Kubernetes router, and that one covers console-openshift-console.apps.${CLUSTER_NAME}.${BASE_DOMAIN} and other non API routes, which can also be more user-y than the OpenShift web console.

Folks can also use the Kubernetes API to create additional load balancers (which is the same way the ingress operator creates its load balancer). If you wanted to limit that, you'd do so via RBAC or similar if you don't trust your users to create load balancers.

• control plane LBs: dont' yet fully understand their archtecture, probably not

This is probably the same thing as the master load balancer. The installer currently sets up this topology, although the location of etcd is not important as far as control-plane load balancing goes. There's a similar control-plane load balancer facing outwards to decouple internal communication from external requests. And the ingress/router load balancer points at workers (at least until kubernetes/kubernetes#65618 is addressed).

[maci0]

This would actually be a very common requirement with our clients.
Right now IHAC who want's to try 4.x as soon as possible. But they want to manage their VPCs and accounts through aws landing zone.

[patlachance]

In addition to @simongun

If I may add additional reasons that my company has to install OCP 4.x in existing VPCs

VPC Peering connections in the same AWS Region is charged at $0.01/GB in each direction.

[wking]

There's now documentation for a user-provided infrastructure approach on AWS (see here and here) which you can use to install a cluster on some existing AWS resources.

[ckyriakidou]

I'm in a similar situation where IHAC where IPI would absolutely be suitable apart from the fact that my customer wants to define the networking and permissions around the VPC. Actually it would be even more useful to have something like "IPI light" - pre-existing VPC where defined restrictions apply on what it can contact within an organisation (doesn't have to be peering - it could be transit gateways or something else) but everything else including security group creation, instance creation etc could be something that OCP installer still does.

[slmingol]

We manage our AWS accounts such that we have a VPC with CIDR blocks in 3 AZs. I'm pretty sure this is a standard type of setup if you look at the AWS best practices around this.

I would expect the installer to either be able to discover the CIDRs and then procure IPs from them or be told which CIDR blocks to procure IPs from and then provision using these.

Any solution has to work within our existing VPCs IMO.

[wking]

2438 is in flight with most of what you'd need to combine user-provided VPC/subnets/networking/... with installer-provided instances/DNS/security-groups/... It still needs an install-config interface to feed in subnet IDs, validation on the fed IDs, etc., etc., but folks on this issue might want to take a look, kick the tires, and provide feedback.

[jeffspahr]

Thanks for working on this feature @wking. We're happy to test it out once it lands in one of the nightly builds:
https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/latest/?extIdCarryOver=true&sc_cid=7013a000002CvFWAA0

Your list in https://github.com/openshift/installer/pull/2438 of what will and will not be included in this flow looks good to me.

Are you targeting 4.2 or 4.3 for this feature?

[wking]

Are you targeting 4.2 or 4.3 for this feature?

4.2 already getting release candidates like this, so it would be 4.3 at the earliest. I'll comment here when there's a nightly to play with.

[wking]

I'll comment here when there's a nightly to play with.

Here's a nightly. No validation yet, but it should work with a procedure like this. I'll close this issue, but feel free to comment further or open new issues with anything you turn up testing 4.3 nightlies :).

/close

[openshift-ci-robot]

@wking: Closing this issue.

In response to this:

I'll comment here when there's a nightly to play with.

[Here's a nightly][1]. No validation yet, but it should work with a procedure like [this][2]. I'll close this issue, but feel free to comment further or open new issues with anything you turn up testing 4.3 nightlies :).

/close

[1]: https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/4.3.0-0.nightly-2019-10-18-120524/
[2]: https://github.com/openshift/installer/pull/2477#issuecomment-542960555

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.