Installer: [RFE] Install on OpenStack without Floating IP

[rvanbutselaar]

Version

$ openshift-install version
./openshift-install v4.2.0
built from commit 90ccb37ac1f85ae811c50a29f9bb7e779c5045fb
release image quay.io/openshift-release-dev/ocp-release@sha256:c5337afd85b94c93ec513f21c8545e3f9e36a227f55d41bc1dfb8fcc3f2be129

Platform:

OpenStack

What happened?

We are not able to install OpenShift on OpenStack without providing a Floating IP.

What you expected to happen?

I would like to be able to install OpenShift on OpenStack without providing a Floating IP. Our OpenShift 3.11 installation currently runs inside a "private" subnet which is only accessible using a VPN connection. We expose applications to the internet using a F5 load-balancer.

# How to reproduce it (as minimally and precisely as possible)?
-
# Anything else we need to know?
-
# References
-

[pierreprinetti]

/label platform/openstack

[MartijnStraatman]

This would be a nice feature.

[dlbewley]

We use provider networks, and would also like to have the option of avoiding floating IPs.

[jichenjc]

+1 to the feature as we don't have floating ip as well

[pierreprinetti]

The automated installer requires a floating IP in order to expose the API endpoint.

You may want to refer to the UPI document in order to adapt it to your needs.

[tomassedovic]

If I understand it correctly, there are two use cases here:

1. Provider networks
2. Private subnet with no default external accessibility (and bring your own load balancer)

Provider networks are something that we should add support for I think. In that case (as far as I'm aware), every Nova server's static IP address is in fact accessible (so it's as if every server got its own floating IP).

I.e. the installer should just not create the floating IPs and everything should keep working.

I don't have an access to a provider network-enabled env right now, so I can't test this. So here's a question @dlbewley: do the VMs need to be placed in a special network/subnet for this to work?

Currently, the installer creates a network and a subnet as well (and places the servers inside). Will that still work, or would we have to change that as well (by e.g. placing the servers in a network/subnet that the user would specify)?

The second use case (no external access) is trickier. As @pierreprinetti mentioned, the installer binary itself talks to the deployed Kubernetes/OpenShift API and reports the progress. It also uses this information to e.g. recognise that the control plane is fully deployed and tear down the bootstrap server.

If there is no floating IP (or other means configured out of band such as a load balancer), the installer will not be able to connect to the Kubernetes API and it will error out. The good news is, that the deployment should still succeed (if it doesn't, I'd consider that a bug). Nothing in the OpenShift internal pods or processes requires that the cluster is publicly accessible.

But there's no way for the installer to know -- so you would have to monitor the logs of the OpenShift nodes, query the OpenShift events and tear down the bootstrap node yourself.

At any rate, all of these cases should be currently handled by the User Provisioned Infrastructure (UPI). The process is documented here:

https://github.com/openshift/installer/blob/master/docs/user/openstack/install_upi.md

It is not fully supported yet, but we have seen it work in our testing. Following that process, you create the networking, servers, load balancers etc. yourself and therefore can do whatever you want (so long as the basic connectivity requirements are satisfied).

That said, I think there is a value in allowing at least provider-networks-enabled deployments in IPI too. I can't promise when we'll get to it, however.

[dlbewley]

@tomassedovic Provider network implies the gateway and possibly IPAM is beyond the sole purview of OpenStack Neutron as in the baremetal case. Yes the IPs are directly reachable assuming provider infrastructure permits it.

Creation of a provider network requires admin role, not to mention coordination on the physical (provider) resources eg. VLAN trunking and SVI creation, so I expect that openshift-installer should accept the network from the user rather than create it, much like it accepts the identity of the floating IP network now.

My goal being to obviate the Neutron router and integrate with my physical network. The router and floating IPs may be obviated by using an external network as the node network.

If this external network is a provider network it can be placed behind a F5, or made VPN restricted, as @rvanbutselaar described, by the provider infrastructure.

I think the hard coding of the ingress and api 4th octets will possibly become cumbersome, but it seems reasonable to expect a segment to be dedicated to one cluster.

Another probably much heavier lift but perhaps more flexible option may be to add a second NIC to the control plane nodes on the provider network for the keepalived ports. This adds implications for the worker nodes interaction.

Unfortunately, having never successfully deployed 4 I can't do much testing to help you. I'm fully blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1769879

[tomassedovic]

Thanks so much for your detailed explanation @dlbewley!

The cert issue should be now fixed for 4.4 and I believe there are backports for 4.3 as well:

https://bugzilla.redhat.com/show_bug.cgi?id=1769879

Providing an option to specify the node network and optionally not create the router and floating IPs makes complete sense to me. I believe the AWS provider is working on a similar thing under the bring-your-own-networking monicker: you set up the networking infra and have the installer bring up the nodes etc. It's kind of a hybrid between a full UPI and IPI.

I assume in the provider network case you don't create subnets, right? To boot up a Nova server you would just pass in the network name.

[SeanHQF]

+1 to the feature 👍

[dlbewley]

@tomassedovic Thanks, I was successful with the 4.4 nightly build a little while ago.

I'm actually not entirely sure if a subnet is required or not. I think so? I know a subnet can optionally provide DHCP. In our case we actually _are_ using a subnet and Neutron DHCP on the provider network.

In our environment the goal is to obviate passing external traffic through a Neutron gateway (particularly in the bare-metal case), and to leverage the datacenter equipment directly instead.

I believe that multiple subnets per network are quite unusual, and I expect the installer to ask for the network name. I know the --network argument to Nova is sufficient.

[tomassedovic]

@dlbewley thanks!

Yeah I think asking for the network is going to be the common scenario, but we will probably have to add an option to specify a subnet too. They're not common, but they do happen.

[gyohuangxin]

Is there so far any plan to support provider networks without Floating Ips?
We don't have Floating IPs in our environment either, so we're really looking forward to it! 👍

[iamemilio]

In recent patches, we have been laying the groundwork for this feature, and definitely plan to support it in an upcoming release. Would you mind adding a brief user story for your use case? It helps us make sure we have all our bases covered. :)

[MartijnStraatman]

We have clients who don't want their cluster to be reachable over the internet. For 3.11 we created loadbalancers in front of the cluster. Internet traffic will go through these loadbalancers first. On request we can give these loadbalancers a floating-ip. Sometimes our clusters sit behind a vpn.

This is also the case if we have a customer which brings a provider-network. All traffic from this provider-network will go through a loadbalancer to the cluster.

The fact IPI creates a FIP was one of the main reasons for us to use UPI first. We really want to use IPI....

[gyohuangxin]

In recent patches, we have been laying the groundwork for this feature, and definitely plan to support it in an upcoming release. Would you mind adding a brief user story for your use case? It helps us make sure we have all our bases covered. :)

Thanks!👍 Our user story is similar, we haven't floating IP in our env, so we also consider a loadbalancer. But I think it's inflexible.
Could you please briefly describe the plan at this stage?

[jichenjc]

we have use case which is some enterprise customer they don't have VxLAN and GRE tunnel in their environment and they are not planning to do this short term (also concern on this kind of setting as well), for example, they want to manage all IPs centrally doing audit and management.

so we have to provide VLAN for them as primary network solution, in other words ,they are using provider network in openstack with VLAN and currently openshift installer only use Floating IP as only supported method.

# ./openshift-install version
./openshift-install 4.4.3
built from commit 78b817ceb7657f81176bbe182cc6efc73004c841

]# ./openshift-install create install-config
? SSH Public Key /root/.ssh/id_rsa.pub
? Platform openstack
INFO Credentials loaded from file "/root/ttt/djjj/clouds.yaml"
? Cloud openstack
? ExternalNetwork HDC_floating_ips
FATAL failed to fetch Install Config: failed to fetch dependency of "Install Config": failed to fetch dependency of "Base Domain": failed to generate asset "Platform": there are no unassigned floating IP addresses available

so we propose to add a non-FIP method to do the install to help customers without FIP or not in favor of FIP into consideration

[mandre]

Thanks all for the feedback.

It sounds like what we want is a combination of BYO Networks (to deploy on networks that weren't created by the installer) and a way to opt-out for the installer to create or assign FIPs.

You would need to create the subnet for the MachineNetwork beforehand and pass it to the installer via the platform.openstack.machinesSubnet property.

We're also considering allowing a None value for the externalNetwork property of install-config.yaml in order to disable the management of FIPs and effectively remove the requirement for FIPs. In that case, we can assume the above machinesSubnet is reachable from the node running the installer. Would it cover your use cases?

[MartijnStraatman]

I think it does!

[jichenjc]

yes, I think that helps a lot

just for implementation detail
https://github.com/openshift/installer/blob/master/pkg/types/openstack/defaults/platform.go#L30
https://github.com/openshift/installer/blob/master/pkg/types/openstack/defaults/platform.go#L51

has some hard code there, this might be considered
if we want to create several cluster from one given network (of course we can consider it later on )

[mandre]

just for implementation detail
https://github.com/openshift/installer/blob/master/pkg/types/openstack/defaults/platform.go#L30
https://github.com/openshift/installer/blob/master/pkg/types/openstack/defaults/platform.go#L51

has some hard code there, this might be considered
if we want to create several cluster from one given network (of course we can consider it later on )

You can already set apiVIP and ingressVIP via the install-config.yaml file, see https://github.com/openshift/installer/blob/master/docs/user/openstack/customization.md#cluster-scoped-properties

The DNS VIP isn't used anymore since 4.4 and will be removed with https://github.com/openshift/installer/pull/3492.

[jichenjc]

just for implementation detail
https://github.com/openshift/installer/blob/master/pkg/types/openstack/defaults/platform.go#L30
https://github.com/openshift/installer/blob/master/pkg/types/openstack/defaults/platform.go#L51
has some hard code there, this might be considered
if we want to create several cluster from one given network (of course we can consider it later on )

You can already set apiVIP and ingressVIP via the install-config.yaml file, see https://github.com/openshift/installer/blob/master/docs/user/openstack/customization.md#cluster-scoped-properties

The DNS VIP isn't used anymore since 4.4 and will be removed with #3492.

ok, this helps a lot , then we will perform our non-FIP PoC based on the new changes , thanks

[mandre]

I forgot to mention that setting the apiVIP and ingressVIP will be available from 4.5, as part of BYO Network feature. It's already in master if you want to give it a try.

[jichenjc]

@mandre can you help to provide any doc related to BYO? I assume it's bring your own network .. any reference or doc will be helpful here

[iamemilio]

Take a look at the master customization doc

The main features that were added were the following for BYO network:

1. Custom machinesSubnet: Pass a pre-configured subnet to the installer to install the nodes onto. This subnet will also be where the VIP Ports will be created.
2. Custom VIP Port IPs: enter a custom IP addresses for the VIP ports within bound of the CIDR range of the machines subnet
3. Additional Networks and Security Groups for MachinePools: For each machine pool, specify any additional networks and security groups that you want to attach its nodes.

[bjhuangr]

Anyone can help to answer, in case that we can NOT assume machines in provider network have network connection to openstack management network:

• How does bootstrap server get ignition configs from glance service
• How does machine API send request to openstack API to create new ocp nodes

[mandre]

Anyone can help to answer, in case that we can NOT assume machines in provider network have network connection to openstack management network:

* How does bootstrap server get ignition configs from glance service

* How does machine API send request to openstack API to create new ocp nodes

This is not a supported scenario for the reasons you've listed. The machine provisioned in the user-provided subnet must have IP connectivity to the OpenStack endpoints.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[mandre]

/close

Floating IPs are now optional starting with release 4.6, and so it the external network. This effectively allows to deploy OpenShift on OpenStack without using Floating IPs. This feature is the most useful when used alongside the Bring Your Own Network (setting machinesSubnet to an existing subnet in the install-config.yaml) since the installer needs to have connectivity to the machine subnet for the installation to finish successfully.

[openshift-ci-robot]

@mandre: Closing this issue.

In response to this:

/close

Floating IPs are now optional starting with release 4.6, and so it the external network. This effectively allows to deploy OpenShift on OpenStack without using Floating IPs. This feature is the most useful when used alongside the Bring Your Own Network (setting machinesSubnet to an existing subnet in the install-config.yaml) since the installer needs to have connectivity to the machine subnet for the installation to finish successfully.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.