Jx: Docker registry not configured in GKE when installing with `jx install..`

[dcanadillas]

Summary

I am installing Jenkins X on a previously created cluster in GKE with terraform (generated with jx create terraform, and docker-registry is not configured, so a quickstart execution gets stuck without running the pipeline.

If you create the same cluster from scratch using jx create cluster gke ... it works.

Steps to reproduce the behavior

• Create terraform scripts and plan with jx create terraform
• Install Jenkins X (Tekton) with jx install...:

$ jx install --provider gke \
> --default-admin-password $MYPWD \
> --default-environment-prefix $JX_CLUSTER \
> --environment-git-owner $GH_ORG \
> --tekton \
> --no-tiller

• Create a Go quickstart with jx create quickstart -l go

Expected behavior

The expected behaviour should be the same like installing with creating the cluster command jx create cluster gke. If the cluster was created with the terraform plan generated by Jenkins X, an installation should behave the same as installing with the one lined command.

Actual behavior

• Cluster is created
• Jenkins X is installed
• But any quickstart is not executing the pipeline because it cannot find the docker registry:

$ jx logs pipelinerunner
WARNING: Waiting for a running pod in namespace jx with labels app=pipelinerunner
WARNING: Found newest pod: pipelinerunner-f98fbd455-vwvk8
{"level":"info","msg":"Waiting for dynamic Tekton Pipelines at http://:8080/","time":"2019-05-03T16:24:24Z"}
{"level":"info","msg":"Generated Git credentials file /home/jenkins/git/credentials\n","time":"2019-05-03T16:35:52Z"}
{"level":"info","msg":"triggering pipeline for repo https://github.com/dcanadillas-kube/demo-tf-1.git branch master revision master context \n","time":"2019-05-03T16:35:52Z"}
{"level":"info","msg":"Deleting and cloning the Jenkins X versions repo","time":"2019-05-03T16:35:52Z"}
{"level":"info","msg":"Cloning the Jenkins X versions repo https://github.com/jenkins-x/jenkins-x-versions.git with ref refs/heads/master to /root/.jx/jenkins-x-versions\n","time":"2019-05-03T16:35:52Z"}
Enumerating objects: 1260, done.
Total 1260 (delta 0), reused 0 (delta 0), pack-reused 1260
{"level":"info","msg":"shallow cloning repository https://github.com/dcanadillas-kube/demo-tf-1.git to temp dir /tmp/git290189597\n","time":"2019-05-03T16:35:53Z"}
{"level":"warning","msg":"no SHAs to merge, falling back to initial cloned commit","time":"2019-05-03T16:35:56Z"}
{"level":"info","msg":"setting up docker registry for https://github.com/dcanadillas-kube/demo-tf-1.git\n","time":"2019-05-03T16:35:56Z"}
{"level":"info","msg":"removing the temp directory /tmp/git290189597\n","time":"2019-05-03T16:35:56Z"}
{"level":"error","msg":"Missing option: --docker-registry Missing option: --docker-registry","time":"2019-05-03T16:35:56Z"}
{"level":"error","msg":"Missing option: --docker-registry","time":"2019-05-03T16:35:56Z"}

Just realized that the docker registry Config Map is not well configured:

$ kubectl get cm jenkins-x-docker-registry -o yaml
apiVersion: v1
data:
  docker.registry: "" #THIS SHOULDN'T BE EMPTY
kind: ConfigMap
metadata:
  annotations:
...

So the pipeline doesn't even start

Jx version

The output of jx version is:

NAME               VERSION
jx                 2.0.75
jenkins x platform 2.0.191
Kubernetes cluster v1.11.8-gke.6
kubectl            v1.13.3
helm client        Client: v2.11.0+g2e55dbe
git                git version 2.11.0
Operating System   Debian GNU/Linux 9.8 (stretch)

Jenkins type

• [x] Next Generation (Tekton + Prow)
• [ ] Classic Jenkins
• [ ] Serverless Jenkins (JenkinsFileRunner + Prow)

Kubernetes cluster

Created in GKE with Terraform jx plan:

jx create terraform -c $JX_CLUSTER=gke \
--gke-machine-type n1-standard-2 \
--gke-max-num-nodes=5 --gke-min-num-nodes=3 \
--gke-project-id=emea-sa-demo \
--gke-zone=europe-west1-c \
--gke-use-enhanced-apis --gke-use-enhanced-scopes \
--organisation-name=terraform-gke \
--skip-login

Then:

cd ~/.jx/organisations/organisation-terraform-gke/clusters/dc-cluster/terraform/
terraform init
terraform plan
terraform apply

Operating system / Environment

Google Cloud Shell (Debian based):

$ uname -ar && cat /etc/debian_version
Linux cs-6000-devshell-vm-706ffcc3-3ad6-43a5-b6c3-8be9a6dbfcba 4.14.104+ #1 SMP Tue Apr 23 18:16:25 PDT 2019 x86_64 GNU/Linux
9.8

[dcanadillas]

I can make it work by specifying the docker registry, docker org and kaniko by:

david@penguin:~$ jx install --provider gke \
--default-admin-password $MYPWD \
--default-environment-prefix $JX_CLUSTER \
--environment-git-owner $GH_ORG \
--docker-registry gcr.io --docker-registry-org $GCP_PROJECT \
--tekton \
--kaniko \
--no-tiller 

If I don't specify the docker registry and enabling kaniko in the cli, the ConfigMap for the Docker Registry has an empty value and the kaniko secrets are not created, which seems to be required to build the images for the quickstart.

That is not needed when using the create cluster command because it creates the service account for kaniko, the kaniko secrets and the docker registry config map is configured to GCR by default. So it is not consistent with the jx install, like the create command used:

david@penguin:~$ jx create cluster gke \
--cluster-name $JX_CLUSTER \
--default-admin-password $MYPWD \
--default-environment-prefix $JX_CLUSTER \
--environment-git-owner $GH_ORG \
--tekton \
--no-tiller

[wbrefvem]

So even if jx create cluster is used, it's possible for the docker registry org to be incorrect. I tend to favor explicitly passing the registry & registry org during installation, so maybe what's needed here is some prompting if the user doesn't specify it.

[wbrefvem]

A doc change advising users to pass docker registry & org explicitly is also not a bad way to go.

[jenkins-x-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://jenkins-x.io/community.
/lifecycle stale

[jenkins-x-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Provide feedback via https://jenkins-x.io/community.
/lifecycle rotten

[jenkins-x-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://jenkins-x.io/community.
/close

[jenkins-x-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://jenkins-x.io/community.
/close

[jenkins-x-bot]

@jenkins-x-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://jenkins-x.io/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository.