K3s: Invalid certificate for https://127.0.0.1:6443

[splattael]

Hi,

first of all, thanks for k3s :+1:
Currently, I am testing it as an alternative to minikube :)

I see OpenSSL complaining about invalid certificate when accessing https://127.0.0.1:6443 with ca.crt extracted from kubectl get secrets ....

Reproduction:

Setup

$ docker-compose up -d --scale node=3
$ cp kubeconfig.yaml ~/.kube/config
$ kubectl get nodes # works great :+1: 

Extract ca.crt

$ kubectl get secrets default-token-XXXXX -o go-template='{{index .data "ca.crt" | base64decode}}'  | tee ca.crt

Show issuer

$ openssl x509 -in ca.crt -noout -subject -issuer
subject=CN = k3s-token-ca@1549801496
issuer=CN = k3s-token-ca@1549801496

Show server certs

$ openssl s_client -showcerts -connect 127.0.0.1:6443 < /dev/null &> apiserver.crt

depth=0 O = k3s-org, CN = cattle
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 O = k3s-org, CN = cattle
verify error:num=21:unable to verify the first certificate
verify return:1
CONNECTED(00000003)
---
Certificate chain
 0 s:/O=k3s-org/CN=cattle
   i:/O=k3s-org/CN=k3s-ca
-----BEGIN CERTIFICATE-----
MIIDCDCCAfCgAwIBAgIIR4ZQtTBbrXAwDQYJKoZIhvcNAQELBQAwIzEQMA4GA1UE
....
sy5YAOPhEVA6WESx1xWmdDpsAmvBFsdEPdP88pg8jHSB0tkZ9L7BIc/4X6+q0QPZ
1Ls9bSy5vEJUWoL/XY3UNDmP5ki9VkfWOBVHlWz1RIIeugZzFrD9fDiR11AxhtP3
l179G9cbE4GJ3nT7
-----END CERTIFICATE-----
---
Server certificate
subject=/O=k3s-org/CN=cattle
issuer=/O=k3s-org/CN=k3s-ca
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1353 bytes and written 269 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: EF5219395CABD1D9BDE462EAEFF21A7A6C350EF955978231E7A5CC4F504E6D3B
    Session-ID-ctx:
    Master-Key: KEY
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket:
    ........

    Start Time: 1549802771
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: no
---
DONE

Verifying cert fails

$ openssl verify -verbose -CAfile ca.crt apiserver.crt
O = k3s-org, CN = cattle
error 20 at 0 depth lookup: unable to get local issuer certificate
error apiserver.crt: verification failed

curl'ing fails with provided cert

$ curl -vv --cacert ca.crt  https://localhost:6443/
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: ca.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: unable to get local issuer certificate
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

curl'ing works with -k

$ curl -vv -k  https://localhost:6443/
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: O=k3s-org; CN=localhost
*  start date: Feb 10 12:24:58 2019 GMT
*  expire date: Feb 10 12:24:58 2020 GMT
*  issuer: O=k3s-org; CN=k3s-ca
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET / HTTP/1.1
> Host: localhost:6443
> User-Agent: curl/7.52.1
> Accept: */*
> 
< HTTP/1.1 401 Unauthorized
< Content-Type: application/json
< Www-Authenticate: Basic realm="kubernetes-master"
< Date: Sun, 10 Feb 2019 12:50:13 GMT
< Content-Length: 165
< 
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
* Curl_http_done: called premature == 0
* Connection #0  to host localhost left intact
}

Is anyone seeing similar behaviour?

It is very likely I am missing something. Hints/feedback is much appreciated :purple_heart:

Kind regards,
Peter

[splattael]

Also, when installing dashboard I also see errors when trying to access it via kubectl proxy.

Error: 'x509: certificate has expired or is not yet valid'
Trying to reach: 'https://10.42.0.6:8443/'

[ibuildthecloud]

@splattael There's a bit of voodoo with the TLS that I have to explain. :6443 is the public API and is signed by a different CA then the internal https://kubernetes.default. So the ca.crt given to service accounts will not work against the public :6443 address, that ca.crt only works against the internal kubernetes service. So if you want the ca.crt to talk to :6443 then get it from the k3s.yaml that is generated or it's at /var/lib/rancher/k3s/server/tls/ca.crt. If you want the ca.crt for the internal API you can get it from the secret as you've done or from /var/lib/rancher/k3s/server/token-ca.crt.

[spitfire88]

@ibuildthecloud Could you please provide steps on the host and agent sides (different nodes) to fix this issue?

[schubter]

@splattael There's a bit of voodoo with the TLS that I have to explain. :6443 is the public API and is signed by a different CA then the internal https://kubernetes.default. So the ca.crt given to service accounts will not work against the public :6443 address, that ca.crt only works against the internal kubernetes service. So if you want the ca.crt to talk to :6443 then get it from the k3s.yaml that is generated or it's at /var/lib/rancher/k3s/server/tls/ca.crt. If you want the ca.crt for the internal API you can get it from the secret as you've done or from /var/lib/rancher/k3s/server/token-ca.crt.

i could not connect succesfully with the certificate from /var/lib/rancher/k3s/server/tls/ca.crt.
fwiw, there is not a single certificate in any of the directories, server or agent that allows connection to 6443 except the one inside the k3s.yaml

[dbkegley]

I'm having a similar issue. Running k3s on EC2, the apiserver certificate's SAN doesn't contain the public IP of the instance.

2019-03-04 08:44:05,839 ERROR Certificate did not match expected hostname: 3.94.187.228. Certificate: {'subjectAltName': (('DNS', ''), ('IP Address', '127.0.0.1'), ('IP Address', '172.31.89.126')), 'notBefore': u'Mar  4 13:43:51 2019 GMT', 'serialNumber': u'573B3FF63D27F2AF', 'notAfter': 'Mar  3 13:44:05 2020 GMT', 'version': 3L, 'subject': ((('organizationName', u'k3s-org'),), (('commonName', u'cattle'),)), 'issuer': ((('organizationName', u'k3s-org'),), (('commonName', u'k3s-ca'),))}

It only contains an empty dns alt name, localhost, and the private ip

[tsuna]

Running on macOS with Docker for Mac:

~/go/src/github.com/rancher/k3s $ kubectl --kubeconfig kubeconfig.yaml get node
Unable to connect to the server: x509: certificate is valid for , kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc., host.docker.internal, not localhost

[Zikoel]

Same problem here! Any news? I want to try k3s on my vps but is impossible without remote access!

[erikwilson]

Please let me know if the new --tls-san flag helps with these issues.

[GeorgeHahn]

I hit this issue and was able to solve it by using the --tls-san flag: server ... --tls-san <external-ip>. Thanks @erikwilson!

[Zikoel]

@erikwilson many thanks! With version 0.3.0 and your option to the server all work fine!

[dbkegley]

The new cli flag is working for me as well, thanks for resolving this!

[AndreKR]

I'm also seeing the issue that the certificate from /etc/rancher/k3s/k3s.yaml (after base64 decoding) is different from /var/lib/rancher/k3s/server/tls/ca.crt and any other certificate in that directory.

[huapox]

I'm also seeing the issue that the certificate from /etc/rancher/k3s/k3s.yaml (after base64 decoding) is different from /var/lib/rancher/k3s/server/tls/ca.crt and any other certificate in that directory.

The same.

[jadsonlourenco]

Running on macOS with Docker for Mac:

~/go/src/github.com/rancher/k3s $ kubectl --kubeconfig kubeconfig.yaml get node
Unable to connect to the server: x509: certificate is valid for , kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc., host.docker.internal, not localhost

I've got this issue after reboot the node few times... I reinstalled the K3S to get it working again... I'm using the v0.5.0 release.

[huapox]

the new version v0.7.0:

[root@(⎈ |default:default) sec-rbac]$ openssl verify -verbose -CAfile ca.crt apiserver.crt
apiserver.crt: OK

[root@(⎈ |default:default) sec-rbac]$ curl -vv --cacert ca.crt  https://server:6443/
*   Trying 2.3.0.2...
* TCP_NODELAY set
* Connected to server (2.3.0.2) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: ca.crt
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=cattle
*  start date: Jul 28 09:37:03 2019 GMT
*  expire date: Jul 27 09:37:31 2020 GMT
*  subjectAltName: host "server" matched cert's "server"
*  issuer: CN=k3s-server-ca@1564306623
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: server:6443
> User-Agent: curl/7.61.1
> Accept: */*
> 
< HTTP/1.1 401 Unauthorized
< Content-Type: application/json
< Www-Authenticate: Basic realm="kubernetes-master"
< Date: Sun, 28 Jul 2019 11:43:47 GMT
< Content-Length: 165
< 
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
* Connection #0 to host server left intact

@splattael

[huapox]

I generated a userKey by ca, with cluster-admin rbac permission.

but got this:

kc --kubeconfig=koper.kubeconfig get pod
error: You must be logged in to the server (Unauthorized)

k3s server's log:

server_1    | E0728 19:15:27.324625       8 authentication.go:65] Unable to authenticate the request due to an error: x509: certificate signed by unknown authority

[erikwilson]

This should be fixed with v0.7.0 where we refactored the certs and moved from token-ca to server-ca:

[ 2019-08-24 00:29:35 ]
root🐮k3s-1:~$ kubectl get secrets default-token-srd6g -o go-template='{{index .data "ca.crt" | base64decode}}' > ca.crt
[ 2019-08-24 00:29:41 ]
root🐮k3s-1:~$ openssl x509 -in ca.crt -noout -subject -issuer
subject=CN = k3s-server-ca@1566604349
issuer=CN = k3s-server-ca@1566604349
[ 2019-08-24 00:29:52 ]
root🐮k3s-1:~$ openssl s_client -showcerts -connect 127.0.0.1:6443 < /dev/null &> apiserver.crt
[ 2019-08-24 00:30:17 ]
root🐮k3s-1:~$ openssl verify -verbose -CAfile ca.crt apiserver.crt
apiserver.crt: OK
[ 2019-08-24 00:30:28 ]
root🐮k3s-1:~$ curl -vv --cacert ca.crt  https://localhost:6443/
*   Trying ::1:6443...
* TCP_NODELAY set
* Connected to localhost (::1) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: ca.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=cattle
*  start date: Aug 23 23:52:29 2019 GMT
*  expire date: Aug 22 23:52:31 2020 GMT
*  subjectAltName: host "localhost" matched cert's "localhost"
*  issuer: CN=k3s-server-ca@1566604349
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: localhost:6443
> User-Agent: curl/7.65.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< Content-Type: application/json
< Www-Authenticate: Basic realm="kubernetes-master"
< Date: Sat, 24 Aug 2019 07:30:51 GMT
< Content-Length: 165
<
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
* Connection #0 to host localhost left intact
}

Please re-open or create a new issue if there is still a problem @splattael

[dhrp]

FYI: this worked for me on K3OS as well.

I added the following to cloud-init.

k3os:
  k3s_args:
    - server
    - "--tls-san=hostn.name.com"