jx upgrade ingress lost all ingress rules

[zhaochunqi]

Summary

jx upgrade ingress lost all ingress rules

Steps to reproduce the behavior

jx upgrade ingress and set custom template to "{{.Service}}.{{.Domain}}"

Expected behavior

update success

Actual behavior

the console shows ingress rules created. but that's not true, I cannot find any ingress rules after that.

Jx version

The output of jx version is:

NAME               VERSION
jx                 2.0.780
Kubernetes cluster v1.13.10-eks-5ac0f1
kubectl            v1.16.0
helm client        Client: v2.14.3+g0e7f3b6
git                2.23.0
Operating System   Mac OS X 10.14.6 build 18G103

Jenkins type

• [x] Serverless Jenkins X Pipelines (Tekton + Prow)
• [x] Classic Jenkins

I'm using multi-cluster, this happens on the production cluster.

Kubernetes cluster

Operating system / Environment

[zhaochunqi]

here's the record, start at 20s.

https://asciinema.org/a/Wt2rpTGg1dVK2FPBzg0uKBpio

[matiashaller]

Hello @zhaochunqi ,

Having same issue using "jx upgrade ingress" deletes standard ingress rules and don't recreates the new one. No error shown in console just as your record posted above.

When running "jx get applications". shows the version and pods but no URL.
When running "kubectl get ingress -n " it says "no resources found"

This was working fine last week. We created 5 new ingress rules with "jx upgrade ingress". But today is not working.

We are using AWS with Classic Jenkins.

Can any one help?

Thanks
Matias

[matiashaller]

I found a work around to expose ingress rules by hand,

From the logs of jx upgrade ingress --verbose I was able to identify the generated configurations files,
In MAC
/var/folders/fr/j47zgnsn7655b0cwr4vs2h7c0000gp/T/helm-template-workdir-096380705/expose-shiftabalone/output/namespaces/<NAMESPACE>/exposecontroller/templates/

In LINUX
/tmp/helm-template-workdir-996004538/expose-shiftabalone/output/namespaces/<NAMESPACE>/exposecontroller/templates/

After forcing the following command, the ingress configuration start working:

kubectl apply -f part0-job.yaml -f part0-configmap.yaml -f part0-role.yaml -f part0-rolebinding.yaml -f part0-serviceaccount.yaml -n <NAMESPACE>

Could be that jx upgrade ingress command is not applying the new generated charts?

I see that
upgrade_ingress.go was changed 3 days ago. May be this is has something to do with this particular issue.

Jenkins x rocks, thank you for all your hard work!

[zhaochunqi]

@matiashaller I followed your workaround, seems not working. (edited, should change urlTemplate)

This was working fine weeks ago for me too.

[lhr0909]

Hey @matiashaller - so @zhaochunqi and I have found that the issue for us lies in this update https://github.com/jenkins-x/exposecontroller/pull/191 where the configmap will add an extra layer of quotes around the URL template config, and the template generation command that is being run in jx upgrade template cannot take values without quotes. So There is the conflict that is causing the ingress not started.

We basically had to remove the quotes in the generated template and manually run them to make it to work. So it seems like we will have to fix it in the jx command level. I am going to try to file a PR to resolve this.

[Markus-ipse]

Same problem here. It worked fine a few weeks ago to upgrade our ingress to use TLS, now when we try it, it completely removes the ingress instead of just enabling TLS :'(

Can we manually recreate the ingress or do we have to delete our entire environment and start over to at least get a non-TLS ingress?

[hferentschik]

OOI, in which context are you running jx upgrade ingress

[zhaochunqi]

I'm on production context, we've figured it out.

@lhr0909 has submitted a PR for this problem https://github.com/jenkins-x/jx/pull/5816

[hferentschik]

I'm on production context, we've figured it out.

Cool. Looking at the PR. Unfortunately, it seems though that the change breaks the ingress upgrade BDD test - https://github.com/jenkins-x/bdd-jx/blob/master/test/suite/ingress/jx_upgrade_ingress.go

[lhr0909]

@hferentschik we can look into the test and fix the test, as long as it is the right change. Since we have found it is currently inconsistent with the exposecontroller change - https://github.com/jenkins-x/exposecontroller/pull/191

[lhr0909]

@hferentschik Is there a way I can see the error that is in the BDD test? This way I might be able to help identify the issue.

[lhr0909]

@Markus-ipse Hey we only had to trigger the deployment one more time (by making an empty/meaningless commit on the env-staging or env-production repo), the ingress should show back up without tearing down the entire cluster and doing it again. @zhaochunqi and I have learned it the hard way.

Once the ingress is back up, try to run jx upgrade ingress again, but try to find the helm template that is being generated, and in the step0-configmap.yaml, try to remove the double quotes in the urltemplate field, and then run kubectl apply -f part0-job.yaml -f part0-configmap.yaml -f part0-role.yaml -f part0-rolebinding.yaml -f part0-serviceaccount.yaml -n <NAMESPACE> should get the TLS-enabled ingress. (https://github.com/jenkins-x/jx/issues/5800#issuecomment-541952577 tells us how to find the raw template)

[Markus-ipse]

@lhr0909 I'm unable to find the raw template unfortunately. Using verbose didn't output anything indicating where the template files are stored, and I've tried searching /var/folders/ recursivly for any files named part0* and it returned nothing :'(

[matiashaller]

Try to run jx upgrade ingress with --verbose option to see were this files are generated. The full command is

jx upgrade ingress --namespaces=<NAMESPACE> --verbose

[Markus-ipse]

I did run the command with the verbose flag, but I also included domain, services and force:

$ jx upgrade ingress --domain k8s.my-domain.se --services my-client --namespaces jx-production --force --verbose

I've gone over the output line-by-line multiple times and the only lines from the entire output that contain any sort of path (excluding urls) where the following two lines

DEBUG: Current configuration dir: /Users/markus/.jx

DEBUG: using stable version 2.3.112 from charts of jenkins-x/exposecontroller from /Users/markus/.jx/jenkins-x-versions

This is quite the showstopper for us so I really hope the PR gets merged :)

[daveconde]

It is possible that the BDD tests for this are failing due to the expose controller version and needs further investigation