Jx: Jx Upgrade platform resets admin password

[jonasliekens]

Summary

When upgrade the platform the Jenkins password is reset to admin and the nexus pod refuses to start due to invalid credentials.

Steps to reproduce the behavior

1. execute jx upgrade platform
2. Enter Y when presented with the question A local Jenkins X versions repository already exists, recreate with latest?

At first the command failed due to a missing secrets.yaml and myvalues.yaml file, but when I created an empty secrets.yaml file with touch secrets.yaml and copied over myvalues.yaml from ~/.jx/cloud-environments/env-gke it executed sucessfully.

Expected behavior

The platform is upgraded to the latest version of Jenkins X

Actual behavior

The platform upgrades, but it seems the admin password is reset to 'admin' and populated through the namespace. Nexus fails to deploy as the secret now contains the value 'admin' as the password instead of the one provided when installing the cluster

Jx version

The output of jx version is:

jx                 1.3.837
jenkins x platform 0.0.3354
Kubernetes cluster v1.10.11-gke.1
kubectl            v1.11.3
helm client        v2.12.1+g02a47c7
helm server        v2.12.1+g02a47c7
git                git version 2.16.3
Operating System   Mac OS X 10.14.2 build 18C54

Jenkins type

• [X] Classic Jenkins
• [ ] Serverless Jenkins

Kubernetes cluster

GKE

jx create cluster gke --skip-login --default-admin-password=myPwD123 -n smb-dev

Operating system / Environment

macOS Mojave (10.14.2)

[polothy]

Same exact thing happened to me, but the command did not fail, it appeared to work fine. The upgrade reset the Jenkins admin password to admin and the nexus secret thinks the password is admin123. It did generate the adminSecrets.yaml file and dumped the contents to the screen and it has the correct credentials there.

Still digging through the Helm release to see if I can figure out where the values went wrong.

[polothy]

Here is my jx version before:

NAME               VERSION
jx                 1.3.839
jenkins x platform 0.0.3313
Kubernetes cluster v1.10.11
kubectl            v1.13.3
helm client        v2.12.1+g02a47c7
helm server        v2.12.1+g02a47c7
git                git version 2.20.1
Operating System   Mac OS X 10.13.6 build 17G4015

And after:

NAME               VERSION
jx                 1.3.839
jenkins x platform 0.0.3354
Kubernetes cluster v1.10.11
kubectl            v1.13.3
helm client        v2.12.1+g02a47c7
helm server        v2.12.1+g02a47c7
git                git version 2.20.1
Operating System   Mac OS X 10.13.6 build 17G4015

[polothy]

Did some Helm digging and I just focused on the Jenkins password. I'm not sure where the Jenkins Helm chart is, but I was able to get Helm to download it. If you look at its templates/secret.yaml it has this:

{{- if .Values.Master.UseSecurity }}
apiVersion: v1
kind: Secret
metadata:
  name: {{ template "jenkins.fullname" . }}
  labels:
    app: {{ template "jenkins.fullname" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
type: Opaque
data:
  {{ if .Values.Master.AdminPassword }}
  jenkins-admin-password: {{ .Values.Master.AdminPassword | b64enc | quote }}
  {{ else }}
  jenkins-admin-password: {{ randAlphaNum 10 | b64enc | quote }}
  {{ end }}
  jenkins-admin-user: {{ .Values.Master.AdminUser | b64enc | quote }}
{{- end }}

If I look at my values for my release via helm get values -a jenkins-x I can lookup jenkins.Master.AdminPassword and it is set to admin.

If I look at what was dumped into adminSecrets.yaml, it has this for Jenkins:

jenkins:
  jenkinssecret:
    password: MY_ACTUAL_PASSWORD

So, looks like something went wrong with the merging of all the Helm values.

tl;dr: Looks like the Jenkins Helm chart is looking for jenkins.Master.AdminPassword but jenkins.jenkinssecret.password is being set in the adminSecrets.yaml

[polothy]

I did find these:

type Jenkins struct {
    JenkinsSecret JenkinsAdminSecret `json:"Master"`
}

type JenkinsAdminSecret struct {
    Password string `json:"AdminPassword"`
}

Maybe we need to update these to include yaml?

[garethjevans]

There is a PR going through that changes the yaml library to one that supports the json annotations

[garethjevans]

This is the PR #3004

[polothy]

Thanks for the feedback. That got me thinking to check what's in the secret on the cluster!

So, if I look at kubectl get secrets jx-install-config -oyaml and decode adminSecrets.yaml it has the jenkins.jenkinssecret.password and that just gets dumped as is to adminSecrets.yaml file:

https://github.com/jenkins-x/jx/blob/311e3c593914b8263f28bff0567599a94cfcd279/pkg/jx/cmd/upgrade_platform.go#L259

Any suggestions on how to fix this? Regenerating a new password is an OK answer too.

[garethjevans]

I'm not 100% at the moment, am wondering if we can read in the file and translate/fix it. I'm going to take a look at this tomorrow

[polothy]

That would be great if you can, much appreciated! I have left the cluster untouched after the jx upgrade platform so let me know if you want me to try anything out or need any extra details!

[polothy]

Looks like all the struct tags were ignored entirely. Here is our cluster's adminSecrets.yaml with all the values REMOVED:

admin:{SHA}REMOVED
chartmuseum:
  chartmuseumenv:
    chartmuseumsecret:
      user: admin
      password: REMOVED
grafana:
  grafanasecret:
    user: admin
    password: REMOVED
jenkins:
  jenkinssecret:
    password: REMOVED
nexus:
  defaultadminpassword: REMOVED
pipelinesecrets:
  mavensettingsxml: |
    REMOVED

I see now that after looking at https://github.com/jenkins-x/jx/pull/2557 and #3004, the config was written correctly to disk but incorrectly to the Kubernetes secret. This is why the install worked and why the upgrade failed.

Well, that was an interesting journey! Looking forward to what you might find/suggest tomorrow @garethjevans. My co-worker might have a correct copy of our adminSecrets.yaml, so I might be able to repair our two clusters with it. Cheers!

[garethjevans]

3013 should fix the secret in the cluster

[garethjevans]

/priority critical-urgent

[polothy]

Took a look at the code, really good idea on how to recover! I'll test it once the PR is merged and released. Thank you very much for making this a priority!

[polothy]

Think this worked! After upgrading jx I ran:

jx upgrade platform --verbose --log-level=debug --always-upgrade

Then I had to restart the Jenkins pod so it would auth the admin user against the updated password (think this was necessary because I had already upgraded the pod previously):

kubectl scale deployment jenkins --replicas 0
kubectl scale deployment jenkins --replicas 1

Not sure if anything else needs a restart. But Nexus pod is now running. I'll report back if anything comes up after some usage.

Thanks all for your help on this!

[garethjevans]

Great, will close the ticket for now.