Kubeadm: External CA: support for CA bundles

[boluisa]

EDIT [lubomir]

original ticket involved a CA bundle where one of the certs is not a CA, this causes kubeadm to use it if it's the first in the bundle:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/util/pkiutil/pki_helpers.go#L246

and fail as it's not a CA.

later we discovered that the controller-manager does not support bundles at all, see:
https://github.com/kubernetes/kubeadm/issues/1723#issuecomment-521780910

this ticket is tracking future support for CA bundles in kubeadm.

---

the admin.conf file does not exists or it is not valid: a kubeconfig file "/etc/kubernetes/admin.conf" exists already but has got the wrong CA cert```

What you expected to happen:
Expected the Kubeconfig validation to pass. see here for what might be causing issue.
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/util/pkiutil/pki_helpers.go#L244

How to reproduce it (as minimally and precisely as possible):
generate a bundled cert, with ca.crt and ca.key files left in /etc/kubernetes/pki directory.
next, run kubeadm init

Environment:

• Kubernetes version (use kubectl version): v1.14.3

[neolit123]

thanks for logging the ticket @boluisa

from the source code:

// We are only putting one certificate in the certificate pem file, so it's safe to just pick the first one
// TODO: Support multiple certs here in order to be able to rotate certs

we have a lot of users that are doing the custom ca.crt and ca.key files already, but i don't believe they use bundles.

the problem with bundles is that the primary certificate in the bundle might not be the root CA for singing the admin.conf.

generate a bundled cert, with ca.crt and ca.key files left in /etc/kubernetes/pki directory.
next, run kubeadm init

i'm assuming this validation passes if you use non-bundled CA?
why does your setup require a bundle?

[boluisa]

@neolit123 yet, it works fine with non-bundle CA's. I think with documentation we can ensure that users arrange the certificates correctly in the ca.crt file. it was a company security procedure to use bundle certs. it's also worth nothing that with just bundles CA.crt and ca.key that this will fail before it even gets to validation phase. here is one of the culprits https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/util/pkiutil/pki_helpers.go#L246

[neolit123]

I think with documentation we can ensure that users arrange the certificates correctly in the ca.crt file

i guess we can document this. @fabriziopandini WDYT?

it's also worth nothing that with just bundles CA.crt and ca.key that this will fail before it even gets to validation phase. here is one of the culprits

but if the root CA is the first in the bundle does the validation pass?
assuming that the admin.conf gets signed with the root CA?

[neolit123]

ok, so i just confirmed that we should be always treating the root CA as the last in the chain.
https://cleantalk.org/help/ssl-ca-bundle
https://knowledge.digicert.com/solution/SO19523.html

@boluisa i'm assuming that the root CA was not the first in your case too?

[boluisa]

@neolit123 that's right

[neolit123]

thanks for confirming, my only worry at this point is breaking existing users if we change the order, so we have to be careful.

[boluisa]

@neolit123 what do you think about introduction of a new flag for bundle certs so it treats it differently.

[certs] Using existing ca certificate authority error execution phase certs/apiserver-kubelet-client: couldn't load CA certificate ca: ca certificate is not a certificate authority
above is the result after running kubeadm init with bundled ca.crt

[neolit123]

ideally we should not be introducing more flags, but we might have to consider this because hardcoding a new order might break users as mentioned already.

[neolit123]

error execution phase certs/apiserver-kubelet-client: couldn't load CA certificate ca: ca certificate is not a certificate authority above is the result after runningkubeadm init``` with bundled ca.crt

@boluisa
i did some tests yesterday and i couldn't reproduce any problems with ca.crt file that is a bundle of CA certificates.

[certs] Using existing ca certificate authority
[certs] Generating "apiserver" certificate and key
...
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
...

my ca.crt file looks like this:

cert1 <-- this one is used by kubeadm
cert2

could you please provide more details and also outline the exact contents of the external ca.crt and ca.key.

the admin.conf file does not exists or it is not valid: a kubeconfig file "/etc/kubernetes/admin.conf" exists already but has got the wrong CA cert```

if you are only placing the ca.* files in the /etc/kubernetes/pki folder on a cliean node then this message should not happen. perhaps it's an indication that an admin.conf exists in /etc/kubernetes/ and that kubeadm reset was not called.

the process should be:

• call sudo kubeadm reset -f
• copy external ca.key and ca.crt to /etc/kubernetes/pki
• start kubeadm init ...

[fabriziopandini]

might be a silly question, but what is the expected behavior here?
kubeadm should validate the cert in admin.conf against all the ca in the bundle?

[neolit123]

@fabriziopandini

i spoke with @boluisa and it seems that the use case was that a ca.crt bundle contained:

server certificate
CA certificate

then kubeadm tries to use the server certificate as a CA and it fails.

right now what we can do is apply the following patch:

diff --git a/cmd/kubeadm/app/util/pkiutil/pki_helpers.go b/cmd/kubeadm/app/util/pkiutil/pki_helpers.go
index 6e49a924eb..578fb1cfa9 100644
--- a/cmd/kubeadm/app/util/pkiutil/pki_helpers.go
+++ b/cmd/kubeadm/app/util/pkiutil/pki_helpers.go
@@ -241,9 +241,17 @@ func TryLoadCertFromDisk(pkiPath, name string) (*x509.Certificate, error) {
                return nil, errors.Wrapf(err, "couldn't load the certificate file %s", certificatePath)
        }

-       // We are only putting one certificate in the certificate pem file, so it's safe to just pick the first one
-       // TODO: Support multiple certs here in order to be able to rotate certs
-       cert := certs[0]
+       // Iterate over all certificates in the bundle and use the first CA
+       var cert *x509.Certificate
+       for i := range certs {
+               if certs[i].IsCA {
+                       cert = certs[i]
+                       break
+               }
+       }
+       if cert == nil {
+               return nil, errors.Errorf("the certificate file %s does not contain a CA", certificatePath)
+       }

        // Check so that the certificate is valid now
        now := time.Now()

what this does is it tries to find a CA in the bundle and not use a regular cert.
but even if we apply it, k8s as a whole is still not CA bundle ready...

see:
https://github.com/kubernetes/kubernetes/issues/63726

if the ca.crt contains more than one CA or e.g. a server cert and a CA cert the controller manager just bails out (from my tests):

F0815 19:54:43.225772       1 controllermanager.go:238] error starting controllers: failed to start certificate controller: error parsing CA cert file "/etc/kubernetes/pki/ca.crt": {"code":1003,"message":"the PEM file should contain only one object"}

[fabriziopandini]

it seems that the use case was that a ca.crt bundle contained:
server certificate
CA certificate

🤔 I was assuming that in a ca-bundle you should have the chain containing all the intermediate CA up to to the RootCA, but not a client/server certificates...

right now what we can do is apply the following patch...

I'm not sure, because the patch picks up the first CA in the chain; instead, AFAIK, we should try to validate against all the CA in the chain (no matter if intermediate or root).

[neolit123]

thinking I was assuming that in a ca-bundle you should have the chain containing all the intermediate CA up to to the RootCA, but not a client/server certificates...

i guess this is a question for @boluisa as this is the case he was facing for users/customers.
agreed that a CA bundle should contain only CA certs!

I'm not sure, because the patch picks up the first CA in the chain; instead, AFAIK, we should try to validate against all the CA in the chain (no matter if intermediate or root).

actually, i think that we should take no action until the controller-manager supports bundles.
until then such a cluster will not work regardless.

i will mark this ticket as backlog and rename it.

[ereslibre]

/cc

[neolit123]

moving this for 1.17 but there will not be a resolution for this anytime soon.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[neolit123]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[neolit123]

/remove-lifecycle stale

[neolit123]

keeping this issue open to be able to track when the KCM enables bundles support for --cluster-signing-cert-file.
xref https://github.com/kubernetes/kubernetes/pull/88741

[neolit123]

xref kubernetes/kubernetes#88741

that PR actually got closed for some reason.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[neolit123]

/remove-lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.