Jx: Platform update issues

[tdcox]

Summary

I updated client and platform last night and this morning I am seeing some issues:

Firstly, I now have a jenkins container that is attempting to start up but which is failing with a CreateContainerConfigError. This is a Next Generation install and previously did not have a jenkins pod running.

Secondly, I now have multiple instances of jenkins-x-gcactivities, jenkins-x-gcpods, jenkins-x-gcpreviews, jx-gcpreviews which are all errored out.

Jx version

The output of jx version is:

jx                 2.0.319
Kubernetes cluster v1.12.8-gke.10
kubectl            v1.14.2
helm client        Client: v2.14.0+g05811b8
git                git version 2.21.0
Operating System   Mac OS X 10.13.6 build 17G6030

Jenkins type

• [x] Next Generation (Tekton + Prow)
• [ ] Classic Jenkins
• [ ] Serverless Jenkins (JenkinsFileRunner + Prow)

[tdcox]

The jenkins pod is failing due to Error: Couldn't find key docker.registry in ConfigMap jx/jenkins-x-docker-registry

[tdcox]

env/jenkins/values.yaml in the dev environment repo is still set to enabled: false

[tdcox]

@jstrachan Just upgraded a fresh cluster to 2.0.718 and the GitOps promotion build has failed and I'm back with another unwanted Jenkins pod. The build error is:

unable to decode "/tmp/helm-template-workdir-922360716/jx/output/env/charts/jenkins-x-platform/charts/jenkins/templates/pod-template-configs.yaml": Object 'Kind' is missing in '{"metadata":{"annotations":{"jenkins.io/chart":"env"},"labels":{"jenkins.io/chart-release":"jx","jenkins.io/version":"15"}}}'
unable to decode "/tmp/helm-template-workdir-922360716/jx/output/env/charts/jenkins-x-platform/templates/jenkins-x-pod-templates.yaml": Object 'Kind' is missing in '{"metadata":{"annotations":{"jenkins.io/chart":"env"},"labels":{"jenkins.io/chart-release":"jx","jenkins.io/version":"15"}}}'

[tdcox]

This issue seems to be getting worse. If I run jx upgrade platform on an instance that is less than a week old, I get an unwanted and broken Jenkins pod, plus the above error AND the build pipeline stops responding to events, rendering the cluster completely unusable.

[warrenbailey]

I'm struggling to replicate this on my cluster. I did an install with the platform version set to 2.0.178 (https://github.com/warrenbailey/jenkins-x-versions/blob/517ac280448c9900e0511fd3c9873f32ce71e981/charts/jenkins-x/jenkins-x-platform.yml#L2) and then ran a jx upgrade platform to bring it up to version 2.0.822. This seemed to work for me and I didn't get the error in the jenkins pods

[warrenbailey]

I'll try it again later with a downgraded jx client as well

[tdcox]

I had a successful upgrade of a cluster created this morning to 2.0.822 this afternoon and that is the first that has worked for a while. I still have an older cluster that hasn’t been able to update so I will try that and see if it fails with 822.

[warrenbailey]

Thanks @tdcox, i'll try again now with an older client and older upgrade. Hopefully it's something that is now fixed.

[warrenbailey]

I've tried again this morning with two cluster and upgraded from 2.0.822 to the latest version 2.0.861. Both times it was successful so I'm assuming this has been fixed by another change. I'm going to close this issue but @tdcox if you still have a problem please reopen and I'll continue to investigate.

[warrenbailey]

/close

[jenkins-x-bot]

@warrenbailey: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tdcox]

I have also been able to perform multiple successful upgrades today.