Che: Can't use custom registries

[anthonma]

Describe the bug

I have a problem trying to use custom devfile and custom plugin registry.
I can build and deploy them without any errors or problems, but the changes I made don't show up. Neither the custom Stack from the devfile registry nor the added plugin of the plugin registry.
To build the registries I use the build.sh, which is contained in the registries repos.
To deploy I use:
NAMESPACE="kube-che" DOMAIN="$(minikube ip).nip.io" helm upgrade --install che-plugin-registry \ --debug \ --namespace ${NAMESPACE} \ --set global.ingressDomain=${DOMAIN} \ ./kubernetes/che-plugin-registry/
for the plugin registry and:
NAMESPACE="kube-che" DOMAIN="$(minikube ip).nip.io" helm upgrade --install che-devfile-registry \ --debug \ --namespace ${NAMESPACE} \ --set global.ingressDomain=${DOMAIN} \ ./deploy/kubernetes/che-devfile-registry/
for the devfile registry.

Che version

• [ ] latest
• [ ] nightly
• [x] other: 7.4.0

Steps to reproduce

Expected behavior

Runtime

• [x] kubernetes (include output of kubectl version)
  Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.4", GitCommit:"67d2fcf276fcd9cf743ad4be9a9ef5828adc082f", GitTreeState:"clean", BuildDate:"2019-09-18T14:51:13Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"windows/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.4", GitCommit:"67d2fcf276fcd9cf743ad4be9a9ef5828adc082f", GitTreeState:"clean", BuildDate:"2019-09-18T14:41:55Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
• [ ] Openshift (include output of oc version)
• [ ] minikube (include output of minikube version and kubectl version)
• [ ] minishift (include output of minishift version and oc version)
• [ ] docker-desktop + K8S (include output of docker version and kubectl version)
• [ ] other: (please specify)

Screenshots

Installation method

• [x] chectl command: chectl server:start -p minikube
• [ ] che-operator
• [ ] minishift-addon
• [ ] I don't know

Environment

• [x] my computer
  
  
  
  • [x] Windows
  
  
  • [ ] Linux
  
  
  • [ ] macOS
  
  
• [ ] Cloud
  
  
  
  • [ ] Amazon
  
  
  • [ ] Azure
  
  
  • [ ] GCE
  
  
  • [ ] other (please specify)
  
  
• [ ] other: please specify

Additional context

[sparkoo]

@anthonma you have to set urls to your registries in Che. You can do it with chectl. The params are --plugin-registry-url and --devfile-registry-url. See chectl server:start --help

[anthonma]

@sparkoo I have started che with the url params, but it didn't work. I still get only the default contents.

[sparkoo]

@anthonma are you sure you're deploying your built images of registries? If you go to url of the registry, can you see your changes? If not, you're probably deploying public image from quay.io. You will need to set cheDevfileRegistryImage to your built image and probably cheDevfileRegistryImagePullPolicy to IfNotPresent. (Deduce parameters for plugin registry)

[anthonma]

@sparkoo I'm trying to set cheDevfileRegistryImage and cheDevfileRegistryImagePullPolicy within my helm command for deploying, but I don't know what the name of the image is. I'm using the devfile registrys build.sh to build it. I tried to use the -t param, but when I do so the devfile-registry pod gets a ImagePullBackOff status

[sparkoo]

@anthonma you need to get the image to your minikube VM. you can run eval $(minikube docker-env) then build.sh, then the image with custom tag should be on your minikube VM. Or you can tag the built image and push it to your quay.io (or any public hub) and pull it from there.

[amisevsk]

I'm using the devfile registrys build.sh to build it. I tried to use the -t param, but when I do so the devfile-registry pod gets a ImagePullBackOff status

The build.sh scripts also support args to tag the image however you like, so you could do something like

./build.sh --organization <your-org> --registry docker.io --tag <your-tag>

and then push that image to your own docker repo and update the registries deployment to use docker.io/<your-org>/che-devfile-registry:<your-tag>

[anthonma]

Is it the only way to push it to docker.io or quay.io or so to use a custom registry with the curent versions of the registries? Before they have changed some days ago it was possible to do build and deploy a custom registry without pushing it to docker.io or so.

[sparkoo]

You need to get the image where you deploy it, in your case it's minikube VM. You can either get it there through public image hub (docker.io, quay.io, ...), build the image on that machine (with e.g. eval $(minikube docker-env)) or you can even copy image as a file and load it to docker (docker save & docker load)

[anthonma]

@sparkoo I want to do it via the second way you have described. I executed eval $(minikube docker-env), build.sh and the helm comand as shown in the readme of the registry repo. This leads to the "error" I have described in the first comment in this issue. Please tell me, what I'm doing wrong and what I have to do.

[benoitf]

I'm doing triage: is this issue more a question instead of a bug ?

[anthonma]

When I opened this issue it was, in my oppinion, more a bug than a question, but at the moment i think it is indeed more a question than a bug.

[sparkoo]

@anthonma I think eval $(minikube docker-env) does not work on your Windows setup. Can you see your kubernetes images when you run docker images ? Then I guess you'll have to go with docker save ... on local, scp the file to minikube VM and docker load ....

Another approach might be to run docker registry localy and push images there https://hub.docker.com/_/registry. I've never tried that though.

[anthonma]

As far as I know you're wright that eval doesn't work on windows. Because of that I'm using Docker Quickstart Terminal, which provides a little linux environment. Could this be the reason why it doesn't work? I will try to get it work via docker save/load and/or via local docker registry.

[sparkoo]

maybe, I don't know Docker Quickstart Terminal. Anyway, I'm solving similar issue as you with not working docker-env. Here's the script I've came up to upload image to minikube https://github.com/sparkoo/che-scripts/blob/master/che-kuploadimage#L13. Feel free to use. I'm just using image tag I'm usualy using for development as a default, but you can pass any parameter to that little script. Or just take the save&load line.

[anthonma]

I have solved it. I had to specify a custom tag while executing build.sh and specify that custom tag while deploying. Unfortunately I have another problem now, when trying to start a workspace, but since it is not a problem about building/deploying I'll open a new isuue for that.