Kubectl: Unable to connect to the server: x509: certificate signed by unknown authority

[mshahat]

Hi,
trying to find an explanation to this.
i am running kubectl on macOS 10.15.3 (19D76)

~ ❯❯❯ kubectl version --certificate-authority haproxy-cert/config-dir/live/pks.uat.lnd.xxx.com/cert.pem
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-13T18:08:14Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.5", GitCommit:"20c265fef0741dd71a66480e35bd69f18351daea", GitTreeState:"clean", BuildDate:"2019-10-15T19:07:57Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

i have an haproxy setup fronting my kuberentes master api-server, configured to offload TLS them establish another https connection to the backend

 frontend fe-https-all-pks-uat-lnd-xxx-com
        bind 172.16.5.100:8443 ssl crt /etc/ssl/certs/pks.uat.lnd.xxx.com.pem
        bind 172.16.5.100:9021 ssl crt /etc/ssl/certs/pks.uat.lnd.xxx.com.pem

        # Cluster 1 Kubernetes Master node
        acl is-cluster-1-master hdr_dom(host) -i cluster-1.pks.uat.lnd.xxx.com
        use_backend be-cluster-1-pks-uat-lnd-xxx-com if is-cluster-1-master

backend be-cluster-1-pks-uat-lnd-xxx-com
        mode http
        http-request set-header X-Forwarded-Port %[dst_port]
        http-request add-header X-Forwarded-Proto https if { ssl_fc }
        server server2 x.x.x.x:8443 ssl verify none

I'm using a wildcard *.pks.uat.lnd.xxx.com certificate generated from let's encrypt, the root CA for that is Digital Signature Trust Co. - DST Root CA X3 which i can see found in Keychain Access on my macOS machine, i also marked it as "Always Trust"

if i use curl or chrome, the handshake seems to go ok with any issues. if I use kubectl !!!

kubectl was not the only client with this problem, i have another cli which has a similar problem. again curl and chrome had no problem with the communication.

as part of my investigation, i ran "$(go env GOPATH)/bin/macos-roots-test" from https://github.com/golang/go/issues/24652#issuecomment-411069915 , as I had a doubt that it could be a golang issue, at this moment I'm not certain whether it is or not.
my observation was the following, even though I have it trusted under Keychain Access

.
.
crypto/x509: DST Root CA X3 returned 1
crypto/x509: Let's Encrypt Authority X3 returned 2
.
.
crypto/x509: verify-cert rejected CN=DST Root CA X3,O=Digital Signature Trust Co.: "Cert Verify Result: CSSMERR_TP_CERT_SUSPENDED"
crypto/x509: verify-cert approved CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
crypto/x509: ran security verify-cert 26 times
    cgo sys roots: 177.446013ms
non-cgo sys roots: 394.296229ms
The test passed, no need to report the output. Thank you.

I also had this thread on letsencrypt with no conclusion!
https://community.letsencrypt.org/t/macos-catalina-10-15-cn-dst-root-ca-x3/113308

i can have kubectl working if kubectl cluster-info --insecure-skip-tls-verify
also if any of the following !!!

~ ❯❯❯ kubectl cluster-info --certificate-authority haproxy-cert/config-dir/live/pks.uat.lnd.xxx.com/cert.pem                                                                                                                                          

~ ❯❯❯ kubectl cluster-info --certificate-authority haproxy-cert/config-dir/live/pks.uat.lnd.xxx.com/chain.pem

~ ❯❯❯ kubectl cluster-info --certificate-authority haproxy-cert/config-dir/live/pks.uat.lnd.xxx.com/fullchain.pem

also if kubectl cluster-info --insecure-skip-tls-verif
Thanks to any points 🙏

[onesolpark]

you have to change the certificate-authority-data in kube config with your proxy cert
(typically located in ~/.kube/config)

Example)

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ** changethispart with base64 decoded cert
    server: https://172.28.128.11:6443
  name: kubernetes

[mshahat]

thanks a million, @onesolpark ... your suggestion is correct, I figured this out a couple of weeks ago and it helped me solve my problem. in my case, i was using a let's encrypt cert, as I commented away the certificate-authority-data in ~/.kube/config , all worked as desired.

thanks,

[chiducaf]

@mshahat how did you change authority-data in ~/.kube/config