Che: Add offline flag to chectl server:start

[l0rd]

Is your enhancement related to a problem? Please describe.

We are making some changes to make it possible to run Che offline. But there is not chectl "offline" flag to automatically set this up.

Describe the solution you'd like

Add flag --offline to the chectl server:start that should:

• pull all images locally
• set imagePullPolicy to Never

The deployment of Che should not be blocked while downloading the images: deployment and image pulling should happen in parallel.

If Che is already deployed the command should pull all the images and change the image pull policies (if needed).

Additional context

The images that need to be pulled are the default images for the given version of chectl (e.g. nightly/latest/7.x.y):

• che server
• keycloak, postgres and jwt-proxy (if multi-user mode)
• plugin registry (offline version)
• devfile registry
• plugins sidecars
• plugin broker
• devfile registry stacks images

Subtasks

• [x] ImagePullPolicy should be configurable for every container https://github.com/eclipse/che/issues/14698
• [x] Create a job to download images into cluster nodes for offline mode https://github.com/eclipse/che/issues/16221
• [x] Deploying offline devfile and plugin registries
• [x] Deploy Eclipse Che and set imagePullPolicy
• [ ] Test on minikube/minishift
• [ ] Write documentation

Minor tasks

• [x] Add github action to push image-copier to quay.io automatically
• [x] Sometimes skopeo failed to copy images with timeout i/o error

[nickboldt]

Moving to 7.3, hopefully we can get this done in time. WDYT @benoitf is this something you can do?

[benoitf]

I'm not sure to fully understand

Describe the solution you'd like
Add flag --offline to the chectl server:start that should:

- pull all images locally
- set imagePullPolicy to Never

• pull all images locally

where are the images ? it's a chectl command that is pulling them from tgz archives ?
we give a folder where images are stored in compressed files ?

also as a side note, for now chectl has no visibility to all the images, it's inside helm and che-operator that the list is defined. Now the list will be stored as well inside chectl ? or it's dumb and take .tgz image with a given pattern name and just sent them so the list is not known.

does that mean that we'll use docker push command ?

[l0rd]

@nickboldt @benoitf we can defer this to 7.4.0 because it's not needed for airgap.

@benoitf images are pulled from the online registries (dockerhub and quay.io). The goal is to prepare Che to run in offline mode. We can maintain the list of images in chectl (the list should include plugin sidecars and stacks as well), I don't have a better idea.

[tolusha]

Syntax:
chectl server:start --offline <stacks-list> --extra-images <images-list>

Briefly, if offline flag is set:

1. figure out the current Che version (read from package.json ?)
2. go to devfile registry [1], get given stacks (if <stacks-list> exists otherwise all stacks)
3. get corresponding list of stack images.
4. get list of plugins and get corresponding images in plugin registry.
5. create a job to download images into cluster nodes
6. deploy che and set ImagePullPolicy to Never

[1] https://github.com/eclipse/che-devfile-registry/tree/master/devfiles

@mmorhun @AndrienkoAleksandr

[tolusha]

@l0rd @benoitf
While implementing I bumped into the question.
How was it supposed to work in minikube if nip.io was not accessible due to offline.

[sleshchenko]

@tolusha we could consider to implement that same as crc use - use local dnsmasq server and configure minishift with our routing suffix and host.
The proposed solution may have pitfalls but it worth to be investigated

[nickboldt]

Do we have an ETA for this to be solved? It would be cool if it could be moved out of backlog and into a specific milestone, such as 7.12 or 7.13.

[tolusha]

7.13 is highly possible.

[l0rd]

How was it supposed to work in minikube if nip.io was not accessible due to offline.

Adding a line in /etc/hosts for the nip hostname? That is how I used to deal with it.

[tolusha]

@amisevsk
I need offline versions of devfile and plugin registries.
I am wondering If it make sense to build and push to quay.io them along with theirs online version?

[amisevsk]

@tolusha that makes a lot of sense -- it was something on my todo list at one point but I never got around to it. The reason we weren't/aren't pushing the images currently is due to licensing (can't redistribute vsixes downloaded from the VS Code marketplace, license issues in projects), but I think those are addressed now with the work to build problem vsixes in CI.

My only concern with pushing offline registries is that (IIRC) some of the sample projects in the devfile registry still do not have licenses attached. Technically that means we have no rights to distribute those projects.

[tolusha]

@amisevsk
Do we have issue to track the progress?

[amisevsk]

@tolusha: For project sample licenses there's https://github.com/eclipse/che/issues/14790; for the overall task of enabling offline registries, there's no current issue (but it would be a dozen line change in the CI scripts).

Additionally, the offline image for the plugin reg can get very large due to old versions -- e.g. 1GB compressed for 7.8.0. There are a few options there (only cache latest plugins; remove old plugins from the registry altogether), but I don't think the approach is settled.

[nickboldt]

get very large due to old versions

Well, you could delete the old stuff from time to time. Why are we hoarding old versions we don't support?