kops create secret keypair ca seems to not use supplied ca.cert

[ag237]

1. What kops version are you running? The command kops version, will display
   this information.

Version 1.9.0

1. What Kubernetes version are you running? kubectl version will print the
   version if a cluster is running or provide the Kubernetes version specified as
   a kops flag.

Version 1.9.6

1. What cloud provider are you using?

AWS

1. What commands did you run? What is the simplest way to reproduce this issue?

kops create secret keypair ca --cert ~/Desktop/ca.pem --key ~/Desktop/ca-key.pem --name cluster.cluster.com

1. What happened after the commands executed?

I0412 14:12:12.757943    8672 create_secret_keypair_ca.go:138] using user provided cert: /Users/agardner/Desktop/ca.pem
I0412 14:12:12.757989    8672 create_secret_keypair_ca.go:139] using user provided private key: /Users/agardner/Desktop/ca-key.pem

I can see the correct MD5 for the key and the cert in the S3 store.

kops get secrets

shows the correct key as well.

TYPE        NAME    ID
Keypair     ca  12526119128799968316
Keypair     ca  keyset.yaml

However, once I run kops update cluster --yes, a second CA cert appears in the S3 store under pki/issued/ca and kops get secrets shows two CA lines

Keypair     ca              12526119128799968316
Keypair     ca              6543628798532000489776353800
Keypair     ca              keyset.yaml

The cluster does not come up at this point and I am seeing

x509: certificate signed by unknown authority

in the logs.

[pseudometric]

I am seeing the same problem. Using kops create secret keypair ca works partially: the API server presents a certificate signed by the CA I provide. However, kops apparently creates a self-signed CA anyway (noted above). It inserts that as the certificate-authority-data in the kubctl config file, so it doesn't match and kubectl fails. Also, the cluster doesn't come up: the nodes can't join the cluster, I assume because the same CA mismatch has been deployed in some fashion on the cluster (I haven't checked the details yet).

It'd be really nice to get this feature working. It's crucial. It's OK to have random self-signed CAs for testing, but not for real deployment; I can't ask users to constantly add certificate exceptions to their browsers and tell command-line tools to ignore certificate validation.

[pseudometric]

I'm also not sure I'm even using the feature correctly. It was added in https://github.com/kubernetes/kops/pull/4526, but I haven't found any documentation on how it's supposed to actually be used (though there is a plea for documentation at the end of that pull request).

[pseudometric]

Also, the cluster doesn't come up: the nodes can't join the cluster, I assume because the same CA mismatch has been deployed in some fashion on the cluster (I haven't checked the details yet).

Indeed; this is happening on the master:

May 11 05:44:37 ip-10-12-139-55 kubelet[2940]: E0511 05:44:37.258148 2940 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://127.0.0.1/api/v1/services?limit=500&resourceVersion=0: x509: certificate signed by unknown authority

The API server is presenting a certificate signed by the user CA, which suggests that the client side here mistakenly has only the bogus self-signed CA (just as with ~/.kube/config).

[pseudometric]

suggests that the client side here mistakenly has only the bogus self-signed CA (just as with ~/.kube/config).

Yes. /var/lib/kubelet/kubeconfig has certificate-authority-data with the wrong (= self-signed) CA, although the client certificate (client-certificate-data) is issued by the right CA. Fixing certificate-authority-data in that file and restarting the kubelet allows it to get further, but now:

May 11 05:54:25 ip-10-12-139-55 kubelet[9169]: E0511 05:54:25.719969 9169 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Unauthorized May 11 05:54:25 ip-10-12-139-55 kubelet[9169]: E0511 05:54:25.817305 9169 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Unauthorized

I assume some other apiserver component vetting the client certificate now doesn't like the issuer.

[pseudometric]

The data in the kops S3 state directory looks almost right: all the leaf certificates are issued by the user CA. However, the ca entry (pki/issued/ca) contains a spurious self-signed CA that isn't needed. That is installed as a file that I assume TLS participants use to validate peer certificates, which explains the observed errors. Separately, the apiserver-aggregator has its own self-signed CA and client/server certificates. That may work just fine on its own, though I don't see why it should not just have certificates issued by the cluster CA.

When I get time, I'll try to come up with a patch that fixes this.

[derjohn]

Hi,
the same happens when you run a CA that does not have the (hardcoded) cetificate subject "cn=kubernetes" (See: https://github.com/kubernetes/kops/blob/8fad9da430e831cd0a92a61830c8b4834551f592/pkg/model/pki.go )

Could anyone make the CA's subject a config option in kops? We get our root CA from $corporate and don't have full influence on the subject.

[nhoughto]

Also seeing this problem, makes it difficult for us to use the same credentials across multiple clusters, we are currently trying to implement a blue/green cluster setup so would like auth to be the same..

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[ag237]

/remove-lifecycle stale

[derjohn]

Dear Bot, this issue is still there and can kick you in the nuts when upgrading. Why is no one taking care?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[ag237]

/remove-lifecycle stale

[derjohn]

Thanks for removing stale ....

[osela]

@ag237 @pseudometric make sure the certificate you use with kops create secret keypair ca includes the following:

X509v3 extensions:
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE

Otherwise kops doesn't recognise the certificate as a valid CA certificate and creates a new one.

[ag237]

I just tried again with kops version 1.11.1 and a custom cert, and still see the problem.

Here is the cert I used.

```Certificate:
Data:
Version: 3 (0x2)
Serial Number: 13980598271189401243 (0xc205100e25af3e9b)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=kubernetes
Validity
Not Before: Apr 11 17:55:25 2019 GMT
Not After : Apr 9 17:55:25 2024 GMT
Subject: CN=kubernetes
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:af:27:08:cd:34:21:15:9f:68:6d:b9:5b:07:31:
12:55:c7:ca:78:54:c6:a1:e5:df:76:f5:e2:45:38:
83:e4:00:11:68:ed:f4:ad:57:77:70:3d:65:a9:57:
f7:45:e8:e9:53:f2:f1:8f:84:0d:af:2e:6f:98:3e:
11:ae:20:97:28:ab:8e:9a:60:55:da:d7:8e:ad:ad:
08:8b:ae:0b:90:16:34:16:0c:a5:fb:5c:af:e6:3d:
39:a7:53:1f:5c:82:4b:06:e5:2f:4f:00:fd:79:1b:
85:bf:79:0f:ce:9a:79:3d:32:54:2f:86:66:82:5b:
ef:09:a8:c7:2c:b9:5c:85:e7:eb:45:8a:fc:a6:71:
b7:75:62:8b:35:c5:60:5e:7d:fd:2a:3a:b5:cc:76:
d2:19:a0:69:b9:fd:19:9f:88:af:83:93:a2:55:de:
06:12:6e:74:dc:70:29:13:a7:fd:80:d2:00:7f:a2:
f5:3e:f5:30:4a:f2:c5:f7:b3:e5:b1:b5:00:49:e4:
c1:17:5c:b3:03:be:77:e0:19:90:c7:36:94:f4:9b:
58:90:31:f7:31:e0:49:0c:f1:75:00:51:48:b7:86:
12:71:00:09:84:31:89:61:59:59:21:b7:45:6f:43:
e7:62:45:85:46:dc:fc:77:fe:17:43:5a:9f:4d:d2:
e5:6b
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Key Identifier:
68:9E:13:FC:AB:30:2A:03:72:86:70:84:D3:10:0E:B3:51:BD:7B:77
X509v3 Authority Key Identifier:
keyid:68:9E:13:FC:AB:30:2A:03:72:86:70:84:D3:10:0E:B3:51:BD:7B:77

        X509v3 Basic Constraints: critical
            CA:TRUE
        X509v3 Key Usage: critical
            Digital Signature, Certificate Sign, CRL Sign
Signature Algorithm: sha256WithRSAEncryption
     7d:40:ed:17:6f:11:cf:bf:ad:a6:95:61:96:11:d1:11:1b:78:
     00:79:3c:22:a2:d5:8e:d6:d9:f3:24:c3:1a:3b:21:28:ce:81:
     63:71:b0:e2:d4:1b:4b:2f:d3:0a:63:78:d4:2a:07:1e:ee:40:
     d6:76:fe:76:26:98:40:20:99:c4:b1:9b:2c:6c:09:f1:e0:f4:
     92:a6:41:ba:3c:d2:3b:0a:ce:3b:ed:52:77:d9:e3:7d:b2:e2:
     c8:38:8e:00:a8:b8:4c:8b:83:1d:d3:ba:a4:79:f8:d4:3d:7e:
     6e:32:1f:a0:cf:53:4e:18:83:05:b3:64:5c:8a:9f:7c:66:ff:
     7e:6c:c4:39:7b:85:49:60:24:b3:e3:49:5c:9c:d3:72:e1:85:
     cd:73:25:a0:f4:8f:f9:1d:36:28:a8:3c:c7:de:d9:dd:e7:51:
     2a:a6:b8:f9:c3:34:2a:16:92:57:9a:35:de:47:0b:74:0c:fb:
     83:a8:ce:a1:39:d6:31:ec:2f:89:7f:1c:ce:95:8c:b2:63:d8:
     1c:0a:91:31:14:a1:0e:12:c6:31:91:1b:9d:0e:60:45:00:cf:
     1d:8d:10:9b:27:0d:55:3b:c3:22:5c:64:a0:b9:c6:d5:94:b0:
     64:3f:3a:56:1d:05:cb:e0:fa:70:c7:8b:ae:2e:93:04:c6:2c:
     ed:3b:4a:aa```

[osela]

try using a cert without the Digital Signature key usage. The code that checks the certificate is very strict.

[ag237]

That was it! Cluster came up and only one CA secret listed, and md5 matches the one I created.

THANK YOU!!!