Jx: integration and compatibility with istio

[bygui86]

Summary

When Istio sidecar auto-injection is turned on, "Expose" pod gives ERROR at the end of a SpringBoot application deployment to "jx-staging" environment. This is also preventing all deployments of the application next versions, blocking the environment pipeline.

Steps to reproduce the behavior

• method 1

  • Install JX
    jx create cluster gke \ --zone europe-west4-c \ --min-num-nodes 4 \ --max-num-nodes 6 \ --machine-type n1-standard-2 \ --kubernetes-version 1.11.4-gke.8 \ --cluster-name playground \ --default-environment-prefix playground

  • Install Istio using JX addon
    jx create addon istio --install-dependencies

  • Enable Istio sidecar auto-injection on jx-staging namespace
    kubectl label namespace jx-staging istio-injection=enabled

  • Import SpringBoot application
    jx import

  • Staging pipeline gives error (see same point screenshot in next method)

  • Expose Pod gives error (see same point screenshot in next method)

• method 2

  • Create a new GKE cluster using gcloud SDK
    gcloud container clusters create playground \ --zone europe-west4-c \ --cluster-version 1.11.4-gke.8 \ --num-nodes 4 --enable-autoscaling --max-nodes 6 --min-nodes 4 \ --machine-type n1-standard-2 \ --scopes cloud-platform

  • Install Istio using yaml manifests without mTLS (see Reference 1.)

  • Install JX
    jx install --default-environment-prefix playground

  • Enable Istio sidecar auto-injection on jx-staging namespace
    kubectl label namespace jx-staging istio-injection=enabled

  • Import SpringBoot application
    jx import

  • Expose Pod gives error (see attached expose logs "jx-install-expose-error-logs.txt" and screenshot below)
    

  • Staging pipeline gives error (see screenshot below)
    

Jx version

1.3.642 and 1.3.684

Kubernetes cluster

GKE version 1.11.4-gke.8 and 1.11.5-gke.5

Operating system / Environment

MacOsX 10.14.2

Expected behavior

Creation and import of a new application should not cause Expose pod to fail, giving problems on all next deployments of application new versions.

Actual behavior

At the end of deployment of a SpringBoot application deployment to "jx-staging" environment, Expose pod gives error and the deploy is not completed. The application is deployed and the ingress created, but the environment pipeline gets stuck preventing all next deployments of the application new versions.

Suggestions

For the first version of istio integration, would be great to avoid creation of the ingress for every application, letting the user configure everything manually using istio-ingressgateway. Then in next versions, everything could be automated by jx.
I tried also to exclude the ingress creation using the flag "--skip-ingress" but it seems to have also problems. See other issue I created "skip-ingress flag not working".

References

1. https://istio.io/docs/setup/kubernetes/quick-start/#option-1-install-istio-without-mutual-tls-authentication-between-sidecars
2. https://istio.io/docs/setup/kubernetes/sidecar-injection/#policy

jx-install-expose-error-logs.txt

[ccojocar]

/kind bug
/area istio
/priority important-soon

[polothy]

What we did to get around this was to fork https://github.com/jenkins-x/default-environment-charts and then remove everything from the values and requirements so that the expose controller is no longer installed. Then, when you create the new env, use the --fork-git-repo to use your new, basically empty, env repo. I think it does fail to deploy until you actually add an app to the env because at the start, it's just an empty helm chart.

This also works well for defining Istio Virtual Services, Gateways and friends in the environment repository. That way, apps have no idea they are using Istio vs expose controller. In addition, we made another environment with promotion set to None to provision cluster wide Istio routing rules, like Egress (Service Entry) definitions.

[jstrachan]

BTW to disable exposecontroller Ingress creation we just need to disable/remove this annotation: https://github.com/jenkins-x-buildpacks/jenkins-x-kubernetes/blob/master/packs/maven/charts/values.yaml#L14-L16

then no Ingress resources will be created by exposecontroller. You can also disable exposecontroller fully from an environment - e.g. on a preview by removing this: https://github.com/jenkins-x-buildpacks/jenkins-x-kubernetes/blob/master/packs/maven/preview/requirements.yaml#L3-L10 or on a permanent environment by removing this: https://github.com/jenkins-x/default-environment-charts/blob/master/env/requirements.yaml#L2-L9

Another approach could be to add to exposecontroller the ability to create istio-ingressgateway resources instead of Ingress for applications. We already have a pluggable strategy in exposecontroller so that we can create Ingress on Kubernetes or if you wish use Route resources on OpenShift so we could just add an istio strategy and do whatever Istio needs to take care of the istio ingress gateway stuff?

[dongdan39]

I'm interested in this jenkins-x integration with istio as istio is the de facto standard for service mesh already. So better forget about other ingress controllers. How about the current status of it? With an istio environment already deployed on a on-premise kubernetes cluster, how to install jenkins-x on it? Thanks!

[carlossg]

the original issue was fixed already in https://github.com/jenkins-x/exposecontroller/pull/165
for better istio integration there should be some follow up issues