Installer: RFE: support remotely bootstrapping a libvirt install

[dustymabe]

Right now the OPENSHIFT_INSTALL_LIBVIRT_URI is the URI the cluster will use to talk to itself to bring up/down machines if requested. However, it is also the URI the installer will use to start the bootsrap node on the libvirt host, which means that you need to run the installer on the libvirt node itself if you're going to be able to talk to the unathenticated TCP connection on the libvirt network that is recommended in the documentation.

It would be nice to be able to kick off the install from a remote node (via qemu+ssh) so that one could use there dev environment to spin up a libvirt cluster OR an AWS cluster. IOW your installer development container is your control center and you can do everything from there. You want to run in AWS? Source in some env variables and GO. Want to run on a remote libvirt machine somehwere? Source in some different env variables and GO!

The proposal here is to add something like OPENSHIFT_INSTALL_LIBVIRT_INSTALLER_URI, which will only be used by the installer to connect to a remote machine and kick off the bootstrap. If a user doesn't provide OPENSHIFT_INSTALL_LIBVIRT_INSTALLER_URI then fallback to the current behavior (OPENSHIFT_INSTALL_LIBVIRT_URI).

So today for me I'd use something like

OPENSHIFT_INSTALL_LIBVIRT_URI=qemu+tcp://192.168.122.1/system # local libvirt network
OPENSHIFT_INSTALL_LIBVIRT_INSTALLER_URI=qemu+ssh://10.XX.XX.XX/system #lan interface of libvirt host

[steveeJ]

Here's a possible, untested workaround:

Assuming you have SSH access to the remote host you can use SSH port forwarding to make the remote daemon available locally. You should be able to add the address 192.168.122.1 to your local lo interface and then instruct SSH to only bind the forwarded port to that address.

Here's an example:

1. sudo ip addr add dev lo 192.168.122.1/32
2. ssh 10.XX.XX.XX -L 192.168.122.1:16509:127.0.0.1:16509

Now you can instruct the installer with OPENSHIFT_INSTALL_LIBVIRT_URI=qemu+tcp://192.168.122.1/system

[dustymabe]

interesting.. i might try that the next time I run through the installer.. only problem I foresee is if you are actually running libvirt locally then 192.168.122.1 would already be taken.. :man_shrugging:

[wking]

... only problem I foresee is if you are actually running libvirt locally then 192.168.122.1 would already be taken.

Run the installer in a network namespace? ;)

And see also the DNS suggestions in #774; this issue seems like a dup of that one.

[dustymabe]

And see also the DNS suggestions in #774;

yeah the DNS suggestions could work but might require some hacking around (i.e. making sure the cluster resolvs the name right).. I still think separating this out and having two different environment variables for the people who want/need/care for it. For the people who don't want/need/care then they just keep using OPENSHIFT_INSTALL_LIBVIRT_URI and don't know the difference.

this issue seems like a dup of that one.

Yeah.
I'm not asking for this to be worked on now, just that I think there is a legit need for it and it seems like I'm not the only one so people are probably going to keep asking for it.

[zeenix]

/priority awaiting-more-evidence

[zeenix]

@dustymabe Do you intend to work/experiment on this? I think this will need contribution(s) from folks who need this use case.

[dustymabe]

I'm out for some time. If this ticket needs to be closed that's OK

[zeenix]

@dustymabe that's OK. I assigned the lower-priority label so no need to close it since it's possible we want to support this at some point.

[jichenjc]

I tried on x86 env for remote host and I think it works fineI have jj-installer and jj-temp1 2 hosts, I run installer at jj-installer with the
Libvirt Connection URI qemu+tcp://10.43.0.25/system while the ip is the jj-temp1

[root@jj-installer installer]# bin/openshift-install create cluster --log-level debug
DEBUG OpenShift Installer unreleased-master-1784-g05b2a2bb6428dc1b71af08942c8529bd605aaf22
.....
? Platform libvirt
? Libvirt Connection URI qemu+tcp://10.43.0.25/system
.....
DEBUG Apply complete! Resources: 9 added, 0 changed, 0 destroyed.
DEBUG OpenShift Installer unreleased-master-1784-g05b2a2bb6428dc1b71af08942c8529bd605aaf22
DEBUG Built from commit 05b2a2bb6428dc1b71af08942c8529bd605aaf22
INFO Waiting up to 30m0s for the Kubernetes API at https://api.test1.tt.testing:6443...

[root@jj-temp1 ~]# virsh list
Id    Name                           State
----------------------------------------------------
3     test1-6jdrb-master-0           running
4     test1-6jdrb-bootstrap          running

[jichenjc]

@zeenix maybe you can check this.. I didn't wait for the openshfit cluster to startup but I did see the coreos startup

[jichenjc]

my test is based on qemu+tcp and the original request is on qemu+ssh
don't know that has some difference.. anyway, above test is FYI

[jaypoulz]

If direct ssh support is requested, we should branch it out into a separate development task, since bootstrapping a cluster is possible without it. I think encouraging this as the default is a valuable step forward, but there are other ways to secure traffic.

[cfergeau]

I think encouraging this as the default is a valuable step forward, but there are other ways to secure traffic.

I'll chime in to point out that dropping qemu+tcp is a mandatory step forward, at the moment the instructions we are giving regarding qemu+tcp are more or less equivalent to "setup a remote passwordless root shell on the machine". So there is more to it than just "securing traffic". I agree that qemu+ssh is not the only way of solving that though, and that it's probably a separate development task.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[akostadinov]

@cfergeau , if qemu+tcp is dropped then local install will require SSH, which is not very nice IMO. Or are you only talking about a separate OPENSHIFT_INSTALL_LIBVIRT_INSTALLER_URI variable?

Having a feature to support separate bootstrap libvirt URI and installer vibvirt URI will help at least with testing where we can separate test executor machine/container from the place we install cluster. Similar to how presently we separate that process from target public cloud.

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci-robot]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.