Che: Create a job to download images into cluster nodes for offline mode

[tolusha]

Is your task related to a problem? Please describe.

To have Eclipse Che working in offline mode we have to prepull all images into cluster nodes.

Additional context

https://github.com/eclipse/che/issues/14699

[mmorhun]

The goal of this issue is to pull all the required images into cluster on which Che should be run without external internet access.

As of now, I see a few ways of achieving that:

• Try to reuse our Kubernetes image puller. However we cannot just use it, because of installation method (at least). Kubernetes image puller could be installed via Helm, which is not the case for us, because Che doesn't have Helm requirement to be installed, or via oc cli, which is not going to work on Kubernetes cluster.
  Also we have operator for Kubernetes image puller, which might be improved to be able to use for Che offline mode needs.
• To pull all required for offline mode images, we may deploy a set of containers (one per node) which have docker socket mounted and just performs docker pull commands. Downside of such approach is that we need cluster admin rights (and probably set allowHostDirVolumePlugin to true).
• We may host separate docker registry for offline Che needs. It would be easy to Che admins to maintain it (just push new image when needed).
• Write the functionality from scratch in chectl using Kubernetes API

@l0rd @benoitf @davidfestal @amisevsk @tomgeorge @tolusha it would be nice to hear some opinion from you.

[tolusha]

I would like to use #1 . If we miss something let's improve image puller

[l0rd]

I am for #1 as well

[davidfestal]

Not sure I completely understand the purpose of this issue and how it relates to the the work we're currently doing in the context of the support of OpenShift 4.3 restricted environments.

Is the goal here to provide an alternate way to load docker images into the disconnected cluster, that would not be OpenShift-specific ?

How would we manage the automatic mapping of docker image references ?

To me making the docker images available to the disconnected cluster is mainly a question of pushing those images into a docker registry that is visible to the disconnected cluster, as recommended in the OS 4.3 procedure. I don't really see how the KubernetesImagePuller would help here.

But maybe that's not really the main issue. The main issue might be more about mapping the image references. Do we finally plan to keep the legacy air-gap mechanism we had setup to override all docker image references at runtime ?

[mmorhun]

Generally, image puller wasn't designed for offline mode, so maybe we should go another way (if it is suitable for us in terms of quality and time). So, to me the registry way sounds good, but we have to support it in case of Kubernetes as well (which I think shouldn't be a problem, but I haven't done any investigations yet).

[l0rd]

@davidfestal the goal is offline rather then air gapped (think about using Che on your laptop when you are on an airplane). And we want to support it on generic Kubernetes cluster (including minikube, kind, crc etc...).

@mmorhun image puller allows to download the images locally and that has 2 benefits: faster workspace bootstrapping and resiliency in case of bad/absent network connectivity. In this case we are using the puller for the latter.

[tomgeorge]

If you could, say, run the kubernetes image puller in a non-restricted environment, and then lock down your network, this would be super easy to do. So for an offline cluster:

1. cluster is provisioned (it has internet connectivity at this point)
2. run the puller/puller operator, setting the images to be pulled
3. disconnect from the internet

If it is as @davidfestal was suggesting, and this was my initial assumption as well, that we are in a restricted environment, I don't think the puller would work out of the box.

Also, helm 3 is tillerless, so that installation method is super easy.

The operator would also work for this use case.

[l0rd]

@tomgeorge: no that's not about restricted, that's about offline (no container registry available). Beyond image pre-pulling, we need to configure other things as the ImagePullPolicy that defaults to Always for most (all?) of the Che images (server side and workspace side).

[amisevsk]

I wouldn't necessarily trust the cluster image cache for reliability -- unless we know very well what eviction policies are in use, etc., there's a significant chance you'll still run into issues. There's a reason the k8s-image-puller continually refreshes the images.

I think we need to refine the scope of this issue:

• Are we supporting arbitrary clusters here? E.g. most local deployments are single node. A multi-node environment has different considerations than minikube/kind/etc.
• There's a pretty wide range of complexity between "I want to run Che on my laptop, on a plane", and "I want to set up Che on my cluster without registry access". I.e. is this task something I set up for myself or something I setup for my organization?

Personally, if I wanted to set this up for myself, I would docker run a local registry and push images to it, and then use the airgap options to point at that registry instead of the default.

[davidfestal]

I somehow agree with @amisevsk analysis.

[l0rd]

I would docker run a local registry and push images to it, and then use the airgap options to point at that registry instead of the default.

If that works for upstream Che on vanilla Kubernetes it looks like the perfect solution.

[amisevsk]

Another thought I had while reading some other documentation: @tomgeorge did some work on running Che in kind. This might be a silver bullet solution for many offline use cases, as I believe with kind your docker daemon is reused, so all you would need to do is to pull images onto your machine; the only concern would be the image pull policy issue brought up by Mario.

[tolusha]

@amisevsk
Could you me point to the doc? I am interested in reusing prepulled images from local machine.

[amisevsk]

@tolusha The docs contributed by @tomgeorge are here. I don't know if they're linked anywhere in the navigation for che docs -- I had a bit of trouble finding them via the search bar.

However, I did just briefly test it locally, and it's not as simple as I thought -- the kind cluster doesn't naturally see the images in my local daemon. I wish I could remember what got that idea in my head.

[tomgeorge]

Those docs don't take into account the TLS changes, so you will need to add your TLS certs before running chectl.

For prepulling images you are probably looking for kind load docker-image

[tomgeorge]

Sorry if the kind docs were not easy to find. They are in the quick start section, among a bunch of other methods, so it is easy to miss.

[tolusha]

Commit to work private docker registry (ab/offline branch):
https://github.com/che-incubator/chectl/commit/979759796ae0759de4edd311f0fb146a1c263c4a