Jx: x509: certificate signed by unknown authority' error on controllerteam pod

Created on 23 Sep 2019 · 15Comments · Source: jenkins-x/jx

Summary

The controllerteam pod cannot successful start due to x509: certificate signed by unknown authority' error during installation.

Steps to reproduce the behavior

Perform a jx installation on a network that employs self signed root ca.

Expected behavior

Installations succeeds without error

Actual behavior

The controllerteam pod contains the follow error in the log

error: failed to initialise helm: failed to run 'helm init --client-only' command in directory '', output: 'Creating /root/.helm
Creating /root/.helm/repository
Creating /root/.helm/repository/cache
Creating /root/.helm/repository/local
Creating /root/.helm/plugins
Creating /root/.helm/starters
Creating /root/.helm/cache/archive
Creating /root/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Error: error initializing: Looks like "https://kubernetes-charts.storage.googleapis.com" is not a valid chart repository or cannot be reached: Get https://kubernetes-charts.storage.googleapis.com/index.yaml: x509: certificate signed by unknown authority'

Jx version

The output of jx version is:

jx                 2.0.775
jenkins x platform 2.0.1287
Kubernetes cluster v1.14.1
kubectl            v1.14.1
helm client        Client: v2.14.3+g0e7f3b6
git                2.18.0
Operating System   CentOS Linux release 7.6.1810 (Core)


verifying packages

Jenkins type

[ X] Serverless Jenkins X Pipelines (Tekton + Prow)
[ ] Classic Jenkins

Kubernetes cluster

v1.14.1 K8S Cluster on VMWare VMs created by kubespray

Operating system / Environment

CentOS 7.6

areingress aresecurity kinenhancement lifecyclstale prioritimportant-longterm

Source

ajureeruja

Most helpful comment

Would also be interested in the outcome of this. Blocking self-signed certificates outright on the assumiption that everyone is using github.com for helm repos and webhooks seems somewhat short-sighted. Particularly in my case, trying to set up a proof-of-concept install on a development kubernetes cluster and using a development gitlab install (all with company-signed certs) and no external access allowed - it makes running jx boot needlessly painful.

I have no problem with requring properly-signed certs being the default, but it should be configurable or this product becomes effectively useless for a significant number of potential enterprise customers.

TonyNoble on 4 Dec 2019

👍4

All 15 comments

Hi @ajureeruja Self signed certs are not officially supported. Refer to https://github.com/jenkins-x/jx/issues/5187 for the latest discussion regarding this matter.

deanesmith on 23 Sep 2019

@deanesmith I don't think this is about using self signed certs. It looks like from the log output that its actually https://kubernetes-charts.storage.googleapis.com (we don't control this endpoint) that's returning the invalid cert.

I'm not sure if this was an intermittent issue? but it seems to be returning a valid certificate now. Is it possible to try again @ajureeruja?

If it's still an issue, is there something "in the way" between the cluster and the google storage endpoint? a network proxy? or WAF?

garethjevans on 24 Sep 2019

@garethjevans I am currently trying to evaluate Jenkins X on-prem behind my company's firewall which directs all traffic through their proxy which is terminated with a self-signed cert. We have to configure most software to trust the company's custom root ca for them to work properly. What I was hoping for in Jenkins X was the ability for me to configure it to use the OS's cert bundle (which has the company's root ca already trusted) or a root ca cert that I provide for the Jenkins X images to trust.

ajureeruja on 24 Sep 2019

👍2

Any insight as to how Jenkins X can be configured with a trusted root ca? If that is not possible at this time, would I be able to configure Jenkins X to point to a local helm repo instead?

ajureeruja on 22 Oct 2019

@ajureeruja I believe it is possible but would require a good deal of documentation how to properly configure each endpoint involved. At this time we don't have this on our immediate radar but I can foresee it becoming more important as we are taking on more security-related tasks in the coming months. Are you using jx boot?

deanesmith on 22 Oct 2019

Yes, I am using jx boot.

ajureeruja on 22 Oct 2019

@deanesmith To which are you referring to that would need a good deal of documentation? Configuring a root ca or configuring a local helm repo?

ajureeruja on 23 Oct 2019

In this case - a local helm repo with a custom root ca. It would be good to know what (if any) other services you encountered difficulties with once past this.

deanesmith on 23 Oct 2019

Configuring a local helm repo would be the workaround to not being able to add a custom root ca to the Jenkins X install in general for all the services that need to communicate outside the network. We wouldn't need to add a custom root ca to the local helm repo since it will be behind our firewall.

Once I get past this first blocker I will post any other problems with services relating to this topic on this thread.

ajureeruja on 23 Oct 2019

TonyNoble on 4 Dec 2019

👍4

I can't agree more.Even with the convenience of letsencrypt the user should be able to use self signed certs. I trust my own CA more than anybody else's ;). Can anyone at least document or point to directions? I'm using on prem Kubernetes and Gitlab, both are not public and have self signed certs.

VGerris on 17 Feb 2020

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://jenkins-x.io/community.
/lifecycle stale