Amazon-eks-ami: Too many certificate files being generated for EKS Optimized AMI v.12

Created on 10 Apr 2019  路  13Comments  路  Source: awslabs/amazon-eks-ami

What happened:

Server Certificates being generated every second in the /var/lib/kubelet/pki directory

Example Output:

-rw------- 1 root root 1232 Apr 10 08:23 kubelet-server-2019-04-10-08-23-38.pem
-rw------- 1 root root 1232 Apr 10 08:23 kubelet-server-2019-04-10-08-23-39.pem
-rw------- 1 root root 1232 Apr 10 08:23 kubelet-server-2019-04-10-08-23-40.pem
-rw------- 1 root root 1232 Apr 10 08:23 kubelet-server-2019-04-10-08-23-41.pem
-rw------- 1 root root 1232 Apr 10 08:23 kubelet-server-2019-04-10-08-23-42.pem
-rw------- 1 root root 1232 Apr 10 08:23 kubelet-server-2019-04-10-08-23-43.pem
-rw------- 1 root root 1232 Apr 10 08:23 kubelet-server-2019-04-10-08-23-44.pem

What you expected to happen: Limited number of PEM files.

How to reproduce it (as minimally and precisely as possible):

Launch Worker nodes from EKS Optimized AMI 1.12
Associate them with EKS Cluster

Environment:

  • AWS Region: us-east-1
  • Instance Type(s): t2.medium
  • AMI Version: EKS Optimized AMI v1.12
picture zealvora

Most helpful comment

Hi, thanks for reporting this. This issue affects 1.12 and is a result of the EKS certificate signer not adding public DNS names and public IPs to kubelet certificates, causing the kubelet to re-request a new certificate ever few seconds.

We'll be rolling out a control plane fix to resolve this issue for 1.12 clusters shortly. If this is impacting kubelet operations for you, as a mitigation you can either use the 1.11 AMI, or run your nodes in non-public subnets and you should see this behavior stop.

If you are using Prometheus for monitoring and you're scraping the kube API servers, you can use the following PromQL query to monitor activity:

rate(
  apiserver_request_count{
    client=~"kubelet.*", 
    resource="certificatesigningrequests",
    verb="POST"
  }[5m]
)

And if you want to compare the total rate of CSRs for 1.11 vs 1.12 kubelets in your cluster:

sum(rate(
  apiserver_request_count{
    client=~"kubelet.*", 
    resource="certificatesigningrequests",
    verb="POST"
  }[5m]
)) by (client)
picture micahhausler on 11 Apr 2019
👍8

All 13 comments

We also encounter this problem. amazon-eks-node-1.12-v20190329 (ami-08716b70cac884aaa)

janlau1111 picture janlau1111 on 11 Apr 2019

Hi, thanks for reporting this. This issue affects 1.12 and is a result of the EKS certificate signer not adding public DNS names and public IPs to kubelet certificates, causing the kubelet to re-request a new certificate ever few seconds.

We'll be rolling out a control plane fix to resolve this issue for 1.12 clusters shortly. If this is impacting kubelet operations for you, as a mitigation you can either use the 1.11 AMI, or run your nodes in non-public subnets and you should see this behavior stop.

If you are using Prometheus for monitoring and you're scraping the kube API servers, you can use the following PromQL query to monitor activity:

rate(
  apiserver_request_count{
    client=~"kubelet.*", 
    resource="certificatesigningrequests",
    verb="POST"
  }[5m]
)

And if you want to compare the total rate of CSRs for 1.11 vs 1.12 kubelets in your cluster:

sum(rate(
  apiserver_request_count{
    client=~"kubelet.*", 
    resource="certificatesigningrequests",
    verb="POST"
  }[5m]
)) by (client)
micahhausler picture micahhausler on 11 Apr 2019
👍8

Hi,
I'm running a 1.12 EKS cluster with nodes on private subnet and I am facing the same issue. I guess this is because I have a Route53 private zone in the cluster.
Also, I can reproduce this behavior using AMS optimized AMI and custom (Debian based) AMI.

Do you have an ETA for a patch on this issue ?

pichouk picture pichouk on 29 Apr 2019
👍5

Hi,

Whats the status of this issue?
Do you when you will release an update ?

aviramradai picture aviramradai on 16 May 2019
👍3

Hi all, this has been resolved globally on the EKS side and no customer action is required. This is also fixed for instances in VPCs with custom DHCP option sets.

If you are using a custom non-EC2 assigned hostname, you'll need to configure the kubelet using the --hostname-override flag to an EC2-identifiable hostname (Ex: the instance's private DNS name) so that our certificate signer can verify that the kubelet certificate is for a legitimate target.

If you're still experiencing issues, please let us know.

micahhausler picture micahhausler on 20 May 2019
🎉3

@micahhausler i think --hostname-override is currently ignored when --cloud-provider is specified.

erks picture erks on 21 May 2019

Hi @micahhausler ;
It seems that the eks.2 update didn't address this issue: https://github.com/kubernetes/kubernetes/issues/77092
Do you have any insight on this ?
Thanks,

eviln1 picture eviln1 on 22 May 2019

@eviln1 can you post the contents of one of your CSRs that is not getting approved?

kubectl get --raw /apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=1 \
    | jq -r .items[0].spec.request \
    | base64 --decode
micahhausler picture micahhausler on 22 May 2019

The request:

$ kubectl get csr vault-74975fcd67-ht6qg-vault-4iz5n -o jsonpath='{.spec.request}' | base64 --decode | openssl req -text -noout -in /dev/stdin
Certificate Request:
    Data:
        Version: 1 (0x0)
        Subject: CN = vault
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (3072 bit)
                Modulus:
                    00:cd:d0:9f:b7:2f:23:70:68:f4:8e:8f:03:75:e0:
                    cc:e2:11:68:36:a2:dd:d4:8f:91:97:05:9d:fd:ff:
                    08:9e:d5:79:df:a8:b1:f1:41:96:86:8a:43:43:55:
                    36:1d:cf:be:ec:b1:1a:cb:67:5e:04:16:8f:d4:25:
                    50:c8:92:8a:65:68:2f:02:d6:37:e3:b8:7e:05:d3:
                    18:ac:54:d3:c8:04:b3:9f:ea:70:db:e3:93:c8:de:
                    1c:93:bd:f9:d6:b8:5c:3c:1f:26:fd:f4:2d:af:25:
                    06:0b:9c:96:38:bc:e2:8e:a8:1d:43:58:4c:fa:e6:
                    75:c8:c4:95:2c:b4:78:85:7f:7b:d1:18:ce:6e:c2:
                    74:6a:8b:e1:7c:19:9b:76:f7:50:a9:6a:41:28:0c:
                    35:66:0d:f4:6d:69:e3:98:79:6a:5c:80:b3:3e:47:
                    cb:53:77:34:00:ee:a9:3e:22:41:b1:25:e1:6d:e5:
                    8c:ee:82:f8:b9:c0:98:5e:86:6f:9c:75:42:9a:e8:
                    2c:c6:56:f2:ef:14:5c:83:56:f9:ba:bd:a7:d8:ee:
                    38:f2:e2:57:79:66:e7:4a:51:ef:a9:ab:bc:4c:e0:
                    92:b1:91:c3:38:a0:ed:65:47:d4:64:4c:ec:42:20:
                    fc:d8:13:f7:5a:24:8c:b6:9d:0d:bf:93:ba:fb:ee:
                    bd:1b:be:12:9e:98:b8:e1:7f:07:06:fc:3a:e4:06:
                    43:5e:23:e7:ee:8b:60:c5:c0:38:85:e0:55:dd:f0:
                    a8:20:5d:3f:ed:7f:08:25:90:ff:da:99:ac:10:e3:
                    a5:70:bd:12:3b:c2:f7:de:51:3a:44:b3:7b:fd:c9:
                    94:be:78:97:a3:0e:53:86:38:a6:4f:8c:88:c0:e2:
                    58:54:20:22:1b:c8:07:6b:ce:f5:ba:46:a5:75:f6:
                    8e:bc:0e:fa:37:f4:4e:35:89:a5:1f:d0:67:12:bd:
                    2e:12:f0:c7:62:0b:e8:e8:76:67:83:2e:e6:49:c5:
                    40:84:83:71:c9:af:ca:f0:6c:9d
                Exponent: 65537 (0x10001)
        Attributes:
        Requested Extensions:
            X509v3 Subject Alternative Name: 
                DNS:vault, DNS:vault.vault, DNS:vault.vault.svc.cluster.local, IP Address:172.24.18.208
    Signature Algorithm: sha256WithRSAEncryption
         bb:1d:e4:36:29:c9:86:b9:32:2a:47:6c:4d:16:89:1c:f5:1e:
         a2:e8:26:a5:cc:e7:f6:6b:44:ec:2c:cf:a7:88:2c:f3:21:15:
         14:d3:3c:d0:9c:16:9e:2d:50:7d:08:e9:db:79:44:de:b2:1a:
         02:65:2e:44:4a:be:9e:96:df:a9:f2:05:d5:8d:b7:65:a3:fb:
         74:cc:23:9f:ff:79:b0:8f:04:ad:c9:70:c5:af:fd:18:06:e8:
         d4:74:36:a1:d1:d7:6e:da:05:1d:c3:2f:38:34:55:b5:17:5e:
         51:e0:f5:09:6b:89:a5:8b:dd:f3:90:3d:64:e6:6a:ce:37:a2:
         c8:b5:ee:5f:2c:9a:9a:cb:aa:9b:52:be:e5:58:76:63:bf:bc:
         bd:bb:1a:ce:e6:f5:90:5d:02:66:9c:1d:c9:f0:2c:87:6e:ef:
         e4:8a:c5:21:55:54:b5:65:cb:e0:10:61:79:da:a0:74:4a:8b:
         bc:b9:4f:aa:31:c4:08:b9:ea:05:bc:4d:3f:d2:64:e3:c3:b7:
         f0:31:d8:b0:36:7a:d0:93:94:19:a9:a9:a6:43:57:9f:fd:41:
         ba:4f:48:8e:1c:24:48:47:1e:98:61:0b:bf:ed:2e:b6:a7:d3:
         02:0b:e6:5d:6c:c1:96:b8:4f:09:b7:e7:a6:08:7f:34:7f:38:
         0a:c4:58:00:b7:fa:03:fa:76:e7:34:fb:17:d2:c8:e1:02:5a:
         22:3c:fb:03:38:92:5b:6c:a4:80:c4:0d:7d:14:c1:91:4b:cd:
         b7:e1:9a:cc:3b:ae:b8:bc:25:9e:a8:95:1b:4e:e5:07:af:d8:
         8f:d8:0d:bc:34:84:16:e2:03:1d:99:eb:6b:60:1b:24:ee:d4:
         48:0a:5c:f7:6d:cf:7b:00:62:3b:6f:10:1b:a1:b5:5a:db:c7:
         ac:03:83:02:0e:1d:a0:14:ab:7f:99:bf:fb:c6:2d:a9:a3:18:
         96:9c:62:9d:a3:0d:5e:16:5a:58:c2:ed:42:d3:c1:5d:84:85:
         da:ff:c6:38:7e:78

The certificate:

$ kubectl get csr vault-74975fcd67-ht6qg-vault-4iz5n -o jsonpath='{.status.certificate}' | base64 --decode | openssl x509 -text -noout -in /dev/stdin 
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            7c:6a:14:3a:07:8c:bf:21:55:08:62:11:7e:2c:ca:aa:f0:ea:ee:fa
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = kubernetes
        Validity
            Not Before: May 22 16:56:00 2019 GMT
            Not After : May 21 16:56:00 2020 GMT
        Subject: CN = vault
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (3072 bit)
                Modulus:
                    00:cd:d0:9f:b7:2f:23:70:68:f4:8e:8f:03:75:e0:
                    cc:e2:11:68:36:a2:dd:d4:8f:91:97:05:9d:fd:ff:
                    08:9e:d5:79:df:a8:b1:f1:41:96:86:8a:43:43:55:
                    36:1d:cf:be:ec:b1:1a:cb:67:5e:04:16:8f:d4:25:
                    50:c8:92:8a:65:68:2f:02:d6:37:e3:b8:7e:05:d3:
                    18:ac:54:d3:c8:04:b3:9f:ea:70:db:e3:93:c8:de:
                    1c:93:bd:f9:d6:b8:5c:3c:1f:26:fd:f4:2d:af:25:
                    06:0b:9c:96:38:bc:e2:8e:a8:1d:43:58:4c:fa:e6:
                    75:c8:c4:95:2c:b4:78:85:7f:7b:d1:18:ce:6e:c2:
                    74:6a:8b:e1:7c:19:9b:76:f7:50:a9:6a:41:28:0c:
                    35:66:0d:f4:6d:69:e3:98:79:6a:5c:80:b3:3e:47:
                    cb:53:77:34:00:ee:a9:3e:22:41:b1:25:e1:6d:e5:
                    8c:ee:82:f8:b9:c0:98:5e:86:6f:9c:75:42:9a:e8:
                    2c:c6:56:f2:ef:14:5c:83:56:f9:ba:bd:a7:d8:ee:
                    38:f2:e2:57:79:66:e7:4a:51:ef:a9:ab:bc:4c:e0:
                    92:b1:91:c3:38:a0:ed:65:47:d4:64:4c:ec:42:20:
                    fc:d8:13:f7:5a:24:8c:b6:9d:0d:bf:93:ba:fb:ee:
                    bd:1b:be:12:9e:98:b8:e1:7f:07:06:fc:3a:e4:06:
                    43:5e:23:e7:ee:8b:60:c5:c0:38:85:e0:55:dd:f0:
                    a8:20:5d:3f:ed:7f:08:25:90:ff:da:99:ac:10:e3:
                    a5:70:bd:12:3b:c2:f7:de:51:3a:44:b3:7b:fd:c9:
                    94:be:78:97:a3:0e:53:86:38:a6:4f:8c:88:c0:e2:
                    58:54:20:22:1b:c8:07:6b:ce:f5:ba:46:a5:75:f6:
                    8e:bc:0e:fa:37:f4:4e:35:89:a5:1f:d0:67:12:bd:
                    2e:12:f0:c7:62:0b:e8:e8:76:67:83:2e:e6:49:c5:
                    40:84:83:71:c9:af:ca:f0:6c:9d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                35:30:06:9D:BE:82:12:AF:84:E2:06:88:C5:64:3F:29:FD:56:BE:D8
    Signature Algorithm: sha256WithRSAEncryption
         82:8b:ab:ce:e7:51:dc:6d:a1:11:4a:e8:7e:f8:13:73:24:aa:
         b1:73:ec:9b:c0:57:f6:06:29:3f:9c:c8:38:90:c3:8e:f2:15:
         d9:b3:81:ca:b8:db:59:09:ca:f9:83:8a:50:dc:d9:6b:32:87:
         33:a4:68:bf:cf:b9:1a:49:b3:ba:d2:f9:ab:9f:22:b9:ab:7f:
         76:b7:4b:18:ca:2e:51:51:69:3f:6b:69:90:9d:0c:31:9b:63:
         28:60:77:74:d4:b6:f0:02:b8:32:a1:fe:af:94:cf:8c:a4:86:
         ec:20:58:5a:53:9e:c3:47:f0:9d:fa:6b:01:6a:46:c5:93:2d:
         60:ef:ee:00:78:76:6a:5c:ef:4b:77:99:05:88:9a:8d:52:db:
         35:c3:ea:3f:ef:10:63:a0:37:f6:0d:e7:1c:cf:a5:12:38:bb:
         69:6d:9b:03:70:1b:0e:98:f5:8d:c5:d4:74:34:51:09:6c:c8:
         cd:68:3d:3f:ff:cd:e7:0a:5c:d5:16:e8:ba:f6:cd:5a:9c:34:
         35:bd:7c:f4:07:ae:49:c1:d3:f1:76:3e:c1:0f:73:9b:c4:80:
         57:1e:26:3a:4e:fe:24:29:57:12:e3:54:bb:bb:d9:d2:c2:49:
         66:44:28:2f:26:90:8f:a4:cc:4e:fb:88:f8:06:99:9a:7c:ff:
         cf:c9:af:b9

Additional info: 

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.6-eks-d69f1b", GitCommit:"d69f1bf3669bf00b7f4a758e978e0e7a1e3a68f7", GitTreeState:"clean", BuildDate:"2019-02-28T20:26:10Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
eviln1 picture eviln1 on 22 May 2019

Ok, so you're attempting to use the signer for a non-node server. Did this work previously for you in 1.11?

micahhausler picture micahhausler on 22 May 2019

Haven't tried it in 1.11 ;
As far as I understand, https://github.com/kubernetes/kubernetes/issues/77092 wasn't about kubelet certificates; the example CSR requested the following SAN : 

Requested Extensions:
  Subject Alternative Name:
    DNS:my-service, DNS:my-service.default, DNS:my-service.default.svc, DNS:my-service.default.svc.cluster, DNS:my-service.default.svc.cluster.local, DNS:localhost
eviln1 picture eviln1 on 23 May 2019
👍1

Hello,
do you have any updates @micahhausler ?
Thanks

eviln1 picture eviln1 on 28 May 2019
👍1

hi @micahhausler met the same problem like eviln1 (not honor Subject Alternative Name in the originating CSR)
have any updates?

WPH95 picture WPH95 on 5 Jun 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

mogren picture mogren  路  7Comments
abdennour picture abdennour  路  4Comments
tupacalypse187 picture tupacalypse187  路  3Comments
cdenneen picture cdenneen  路  6Comments
jhoblitt picture jhoblitt  路  4Comments