Openshift-ansible: OKD 3.11: Control plane pods didn't come up

[ctron]

Description

Provide a brief description of your issue here. For example:

On a fresh installation the ansible script fails with:

~~~
Failure summary:

1. Hosts: 159.69.154.239
   Play: Configure masters
   Task: Report control plane errors
   Message: Control plane pods didn't come up
   ~~~

Version

Please put the following version information in the code block
indicated below.

~
ansible 2.6.7
config file = /etc/ansible/ansible.cfg
configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/site-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.5 (default, Jul 13 2018, 13:06:57) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]
~

If you're operating from a git clone:

• The output of git describe

~~~

git describe

openshift-ansible-3.11.39-1
~~~

If you're running from playbooks installed via RPM

Steps To Reproduce

1. Create new machine with CentOS 7.5
2. Run ansible playbooks

Expected Results

Describe what you expected to happen.

Installation succeeds.

Observed Results

Describe what is actually happening.

Installation fails with the message above.

Additional Information

Provide any additional information which may help us diagnose the
issue.

~~~

cat /etc/redhat-release

CentOS Linux release 7.5.1804 (Core)
~~~

Inventory (worked with 3.10):

~~~
[OSEv3:children]
masters
nodes
nfs
etcd

[OSEv3:vars]
deployment_type=origin
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}]

openshift_install_examples=false

Our name

openshift_master_default_subdomain=cluster00.amazing.iot-playground.org
openshift_master_cluster_hostname=cluster00.amazing.iot-playground.org
openshift_master_cluster_public_hostname=cluster00.amazing.iot-playground.org

for NFS

openshift_enable_unsupported_configurations=true

openshift version

openshift_image_tag=v3.11
openshift_release=v3.11

Registry

openshift_hosted_registry_storage_kind=nfs
openshift_hosted_registry_storage_access_modes=['ReadWriteMany']
openshift_hosted_registry_storage_nfs_directory=/exports
openshift_hosted_registry_storage_nfs_options='*(rw,root_squash)'
openshift_hosted_registry_storage_volume_name=registry
openshift_hosted_registry_storage_volume_size=40Gi

Certificates

openshift_master_overwrite_named_certificates=true
openshift_master_named_certificates=[{"certfile": "/root/certs/fullchain.cer", "keyfile": "/root/certs/server.key", "cafile": "/root/certs/fullchain.cer", "names": ["cluster00.amazing.iot-playground.org"]}]
openshift_hosted_router_certificate={"certfile": "/root/certs/server.cer", "keyfile": "/root/certs/server.key", "cafile": "/root/certs/fullchain.cer"}

[masters]
159.69.154.239

[etcd]
159.69.154.239

[nodes]
159.69.154.239 openshift_ip=159.69.154.239 openshift_public_ip=159.69.154.239 openshift_hostname=cluster00.amazing.iot-playground.org openshift_public_hostname=cluster00.amazing.iot-playground.org openshift_node_group_name='node-config-all-in-one'

[nfs]
159.69.154.239
~~~

Running journalctl I can see the following errors repeating all over the place:

System log

~~~
Nov 05 18:30:48 cluster00 origin-node[28217]: E1105 18:30:48.368410 28217 kubelet_node_status.go:103] Unable to register node "cluster00" with API server: Post https://cluster00.amazing.iot-playground.org:8443/api/v1/nodes: dial tcp 159.69.154.239:8443: connect: connection refused
Nov 05 18:30:49 cluster00 origin-node[28217]: E1105 18:30:49.158068 28217 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://cluster00.amazing.iot-playground.org:8443/api/v1/nodes?fieldSelector=metadata.name%3Dcluster00&limit=500&resourceVersion=0: dial tcp 159.69.154.239:8443: connect: connection refused
Nov 05 18:30:49 cluster00 origin-node[28217]: E1105 18:30:49.168701 28217 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://cluster00.amazing.iot-playground.org:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 159.69.154.239:8443: connect: connection refused
Nov 05 18:30:49 cluster00 origin-node[28217]: E1105 18:30:49.168751 28217 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://cluster00.amazing.iot-playground.org:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dcluster00&limit=500&resourceVersion=0: dial tcp 159.69.154.239:8443: connect: connection refused
Nov 05 18:30:50 cluster00 origin-node[28217]: E1105 18:30:50.161028 28217 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://cluster00.amazing.iot-playground.org:8443/api/v1/nodes?fieldSelector=metadata.name%3Dcluster00&limit=500&resourceVersion=0: dial tcp 159.69.154.239:8443: connect: connection refused
Nov 05 18:30:50 cluster00 origin-node[28217]: E1105 18:30:50.171466 28217 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://cluster00.amazing.iot-playground.org:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 159.69.154.239:8443: connect: connection refused
Nov 05 18:30:50 cluster00 origin-node[28217]: E1105 18:30:50.171506 28217 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://cluster00.amazing.iot-playground.org:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dcluster00&limit=500&resourceVersion=0: dial tcp 159.69.154.239:8443: connect: connection refused
Nov 05 18:30:51 cluster00 origin-node[28217]: E1105 18:30:51.163475 28217 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://cluster00.amazing.iot-playground.org:8443/api/v1/nodes?fieldSelector=metadata.name%3Dcluster00&limit=500&resourceVersion=0: dial tcp 159.69.154.239:8443: connect: connection refused
Nov 05 18:30:51 cluster00 origin-node[28217]: E1105 18:30:51.173784 28217 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://cluster00.amazing.iot-playground.org:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dcluster00&limit=500&resourceVersion=0: dial tcp 159.69.154.239:8443: connect: connection refused
Nov 05 18:30:51 cluster00 origin-node[28217]: E1105 18:30:51.173822 28217 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://cluster00.amazing.iot-playground.org:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 159.69.154.239:8443: connect: connection refused
Nov 05 18:30:51 cluster00 origin-node[28217]: W1105 18:30:51.470194 28217 cni.go:172] Unable to update cni config: No networks found in /etc/cni/net.d
Nov 05 18:30:51 cluster00 origin-node[28217]: E1105 18:30:51.471406 28217 kubelet.go:2101] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Nov 05 18:30:52 cluster00 origin-node[28217]: E1105 18:30:52.166390 28217 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://cluster00.amazing.iot-playground.org:8443/api/v1/nodes?fieldSelector=metadata.name%3Dcluster00&limit=500&resourceVersion=0: dial tcp 159.69.154.239:8443: connect: connection refused
Nov 05 18:30:52 cluster00 origin-node[28217]: E1105 18:30:52.175886 28217 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://cluster00.amazing.iot-playground.org:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dcluster00&limit=500&resourceVersion=0: dial tcp 159.69.154.239:8443: connect: connection refused
Nov 05 18:30:52 cluster00 origin-node[28217]: E1105 18:30:52.175937 28217 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://cluster00.amazing.iot-playground.org:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 159.69.154.239:8443: connect: connection refused
Nov 05 18:30:53 cluster00 origin-node[28217]: E1105 18:30:53.170245 28217 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://cluster00.amazing.iot-playground.org:8443/api/v1/nodes?fieldSelector=metadata.name%3Dcluster00&limit=500&resourceVersion=0: dial tcp 159.69.154.239:8443: connect: connection refused
Nov 05 18:30:53 cluster00 origin-node[28217]: E1105 18:30:53.178079 28217 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://cluster00.amazing.iot-playground.org:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 159.69.154.239:8443: connect: connection refused
Nov 05 18:30:53 cluster00 origin-node[28217]: E1105 18:30:53.178226 28217 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://cluster00.amazing.iot-playground.org:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dcluster00&limit=500&resourceVersion=0: dial tcp 159.69.154.239:8443: connect: connection refused
Nov 05 18:30:53 cluster00 origin-node[28217]: E1105 18:30:53.402762 28217 event.go:212] Unable to write event: 'Patch https://cluster00.amazing.iot-playground.org:8443/api/v1/namespaces/default/events/cluster00.15644924d716acb6: dial tcp 159.69.154.239:8443: connect: connection refused' (may retry after sleeping)

~~~

[ctron]

Digging more into this issue it looks like the api server didn't properly start up. In the API server logs I can see the following fatal error:

~
F1106 10:42:38.185809 1 start_api.go:68] dial tcp [::1]:2379: connect: connection refused
~

Full API server log

~~~ E1106 10:42:08.168321 1 helpers.go:134] Encountered config error json: unknown field "masterCount" in object *config.MasterConfig, raw JSON: {"admissionConfig":{"pluginConfig":{"BuildDefaults":{"configuration":{"apiVersion":"v1","env":[],"kind":"BuildDefaultsConfig","resources":{"limits":{},"requests":{}}}},"BuildOverrides":{"configuration":{"apiVersion":"v1","kind":"BuildOverridesConfig"}},"openshift.io/ImagePolicy":{"configuration":{"apiVersion":"v1","executionRules":[{"matchImageAnnotations":[{"key":"images.openshift.io/deny-execution","value":"true"}],"name":"execution-denied","onResources":[{"resource":"pods"},{"resource":"builds"}],"reject":true,"skipOnResolutionFailure":true}],"kind":"ImagePolicyConfig"}}}},"aggregatorConfig":{"proxyClientInfo":{"certFile":"aggregator-front-proxy.crt","keyFile":"aggregator-front-proxy.key"}},"apiLevels":["v1"],"apiVersion":"v1","authConfig":{"requestHeader":{"clientCA":"front-proxy-ca.crt","clientCommonNames":["aggregator-front-proxy"],"extraHeaderPrefixes":["X-Remote-Extra-"],"groupHeaders":["X-Remote-Group"],"usernameHeaders":["X-Remote-User"]}},"controllerConfig":{"election":{"lockName":"openshift-master-controllers"},"serviceServingCert":{"signer":{"certFile":"service-signer.crt","keyFile":"service-signer.key"}}},"controllers":"*","corsAllowedOrigins":["(?i)//127\\.0\\.0\\.1(:|\\z)","(?i)//localhost(:|\\z)","(?i)//159\\.69\\.154\\.239(:|\\z)","(?i)//kubernetes\\.default(:|\\z)","(?i)//kubernetes\\.default\\.svc\\.cluster\\.local(:|\\z)","(?i)//kubernetes(:|\\z)","(?i)//openshift\\.default(:|\\z)","(?i)//cluster00(:|\\z)","(?i)//openshift\\.default\\.svc(:|\\z)","(?i)//172\\.30\\.0\\.1(:|\\z)","(?i)//openshift\\.default\\.svc\\.cluster\\.local(:|\\z)","(?i)//kubernetes\\.default\\.svc(:|\\z)","(?i)//openshift(:|\\z)","(?i)//cluster00\\.amazing\\.iot\\-playground\\.org(:|\\z)"],"dnsConfig":{"bindAddress":"0.0.0.0:8053","bindNetwork":"tcp4"},"etcdClientInfo":{"ca":"master.etcd-ca.crt","certFile":"master.etcd-client.crt","keyFile":"master.etcd-client.key","urls":["https://cluster00:2379"]},"etcdStorageConfig":{"kubernetesStoragePrefix":"kubernetes.io","kubernetesStorageVersion":"v1","openShiftStoragePrefix":"openshift.io","openShiftStorageVersion":"v1"},"imageConfig":{"format":"docker.io/openshift/origin-${component}:${version}","latest":false},"imagePolicyConfig":{"internalRegistryHostname":"docker-registry.default.svc:5000"},"kind":"MasterConfig","kubeletClientInfo":{"ca":"ca-bundle.crt","certFile":"master.kubelet-client.crt","keyFile":"master.kubelet-client.key","port":10250},"kubernetesMasterConfig":{"apiServerArguments":{"storage-backend":["etcd3"],"storage-media-type":["application/vnd.kubernetes.protobuf"]},"controllerArguments":{"cluster-signing-cert-file":["/etc/origin/master/ca.crt"],"cluster-signing-key-file":["/etc/origin/master/ca.key"],"pv-recycler-pod-template-filepath-hostpath":["/etc/origin/master/recycler_pod.yaml"],"pv-recycler-pod-template-filepath-nfs":["/etc/origin/master/recycler_pod.yaml"]},"masterCount":1,"masterIP":"159.69.154.239","podEvictionTimeout":null,"proxyClientInfo":{"certFile":"master.proxy-client.crt","keyFile":"master.proxy-client.key"},"schedulerArguments":null,"schedulerConfigFile":"/etc/origin/master/scheduler.json","servicesNodePortRange":"","servicesSubnet":"172.30.0.0/16","staticNodeNames":[]},"masterClients":{"externalKubernetesClientConnectionOverrides":{"acceptContentTypes":"application/vnd.kubernetes.protobuf,application/json","burst":400,"contentType":"application/vnd.kubernetes.protobuf","qps":200},"externalKubernetesKubeConfig":"","openshiftLoopbackClientConnectionOverrides":{"acceptContentTypes":"application/vnd.kubernetes.protobuf,application/json","burst":600,"contentType":"application/vnd.kubernetes.protobuf","qps":300},"openshiftLoopbackKubeConfig":"openshift-master.kubeconfig"},"masterPublicURL":"https://cluster00.amazing.iot-playground.org:8443","networkConfig":{"clusterNetworks":[{"cidr":"10.128.0.0/14","hostSubnetLength":9}],"externalIPNetworkCIDRs":["0.0.0.0/0"],"networkPluginName":"redhat/openshift-ovs-subnet","serviceNetworkCIDR":"172.30.0.0/16"},"oauthConfig":{"assetPublicURL":"https://cluster00.amazing.iot-playground.org:8443/console/","grantConfig":{"method":"auto"},"identityProviders":[{"challenge":true,"login":true,"mappingMethod":"claim","name":"htpasswd_auth","provider":{"apiVersion":"v1","file":"/etc/origin/master/htpasswd","kind":"HTPasswdPasswordIdentityProvider"}}],"masterCA":"ca-bundle.crt","masterPublicURL":"https://cluster00.amazing.iot-playground.org:8443","masterURL":"https://cluster00.amazing.iot-playground.org:8443","sessionConfig":{"sessionMaxAgeSeconds":3600,"sessionName":"ssn","sessionSecretsFile":"/etc/origin/master/session-secrets.yaml"},"tokenConfig":{"accessTokenMaxAgeSeconds":86400,"authorizeTokenMaxAgeSeconds":500}},"pauseControllers":false,"policyConfig":{"bootstrapPolicyFile":"/etc/origin/master/policy.json","openshiftInfrastructureNamespace":"openshift-infra","openshiftSharedResourcesNamespace":"openshift"},"projectConfig":{"defaultNodeSelector":"node-role.kubernetes.io/compute=true","projectRequestMessage":"","projectRequestTemplate":"","securityAllocator":{"mcsAllocatorRange":"s0:/2","mcsLabelsPerProject":5,"uidAllocatorRange":"1000000000-1999999999/10000"}},"routingConfig":{"subdomain":"cluster00.amazing.iot-playground.org"},"serviceAccountConfig":{"limitSecretReferences":false,"managedNames":["default","builder","deployer"],"masterCA":"ca-bundle.crt","privateKeyFile":"serviceaccounts.private.key","publicKeyFiles":["serviceaccounts.public.key"]},"servingInfo":{"bindAddress":"0.0.0.0:8443","bindNetwork":"tcp4","certFile":"master.server.crt","clientCA":"ca.crt","keyFile":"master.server.key","maxRequestsInFlight":500,"namedCertificates":[{"certFile":"/etc/origin/master/named_certificates/fullchain.cer","keyFile":"/etc/origin/master/named_certificates/server.key","names":["cluster00.amazing.iot-playground.org"]}],"requestTimeoutSeconds":3600},"volumeConfig":{"dynamicProvisioningEnabled":true}} I1106 10:42:08.171170 1 plugins.go:84] Registered admission plugin "NamespaceLifecycle" I1106 10:42:08.171196 1 plugins.go:84] Registered admission plugin "Initializers" I1106 10:42:08.171212 1 plugins.go:84] Registered admission plugin "ValidatingAdmissionWebhook" I1106 10:42:08.171225 1 plugins.go:84] Registered admission plugin "MutatingAdmissionWebhook" I1106 10:42:08.171234 1 plugins.go:84] Registered admission plugin "AlwaysAdmit" I1106 10:42:08.171240 1 plugins.go:84] Registered admission plugin "AlwaysPullImages" I1106 10:42:08.171253 1 plugins.go:84] Registered admission plugin "LimitPodHardAntiAffinityTopology" I1106 10:42:08.171260 1 plugins.go:84] Registered admission plugin "DefaultTolerationSeconds" I1106 10:42:08.171270 1 plugins.go:84] Registered admission plugin "AlwaysDeny" I1106 10:42:08.171285 1 plugins.go:84] Registered admission plugin "EventRateLimit" I1106 10:42:08.171303 1 plugins.go:84] Registered admission plugin "DenyEscalatingExec" I1106 10:42:08.171313 1 plugins.go:84] Registered admission plugin "DenyExecOnPrivileged" I1106 10:42:08.171323 1 plugins.go:84] Registered admission plugin "ExtendedResourceToleration" I1106 10:42:08.171334 1 plugins.go:84] Registered admission plugin "OwnerReferencesPermissionEnforcement" I1106 10:42:08.171368 1 plugins.go:84] Registered admission plugin "ImagePolicyWebhook" I1106 10:42:08.171383 1 plugins.go:84] Registered admission plugin "LimitRanger" I1106 10:42:08.171403 1 plugins.go:84] Registered admission plugin "NamespaceAutoProvision" I1106 10:42:08.171413 1 plugins.go:84] Registered admission plugin "NamespaceExists" I1106 10:42:08.171424 1 plugins.go:84] Registered admission plugin "NodeRestriction" I1106 10:42:08.171439 1 plugins.go:84] Registered admission plugin "PersistentVolumeLabel" I1106 10:42:08.171451 1 plugins.go:84] Registered admission plugin "PodNodeSelector" I1106 10:42:08.171466 1 plugins.go:84] Registered admission plugin "PodPreset" I1106 10:42:08.171478 1 plugins.go:84] Registered admission plugin "PodTolerationRestriction" I1106 10:42:08.171496 1 plugins.go:84] Registered admission plugin "ResourceQuota" I1106 10:42:08.171512 1 plugins.go:84] Registered admission plugin "PodSecurityPolicy" I1106 10:42:08.171528 1 plugins.go:84] Registered admission plugin "Priority" I1106 10:42:08.171540 1 plugins.go:84] Registered admission plugin "SecurityContextDeny" I1106 10:42:08.171552 1 plugins.go:84] Registered admission plugin "ServiceAccount" I1106 10:42:08.171567 1 plugins.go:84] Registered admission plugin "DefaultStorageClass" I1106 10:42:08.171577 1 plugins.go:84] Registered admission plugin "PersistentVolumeClaimResize" I1106 10:42:08.171591 1 plugins.go:84] Registered admission plugin "StorageObjectInUseProtection" F1106 10:42:38.185809 1 start_api.go:68] dial tcp [::1]:2379: connect: connection refused ~~~

[ctron]

Checking if the etcd process is up, it seems so, but it also seems that etcd isn't bound to localhost, but to the actual IP address of the machine:

~
2018-11-06 10:27:36.093928 I | embed: ready to serve client requests
2018-11-06 10:27:36.111267 I | embed: serving client requests on 159.69.154.239:2379
~

~
LISTEN 0 128 159.69.154.239:2379 :
~

[ctron]

So it turns out that the problem was caused by the initial hosts file of this machine containing the following entry:

~
127.0.0.1 cluster00
~

Which maps the hostname (cluster00) to 127.0.0.1 and seems to trigger this issue. Replacing 127.0.0.1 with the machine's IP address did solve the problem.

However, this did work with OKD 3.10. So to me, this looks like a regression in 3.11, as the internal and public IP was properly specified in the Ansible inventory file. So there should be no reason to use 127.0.0.1 instead what was specified inventory.

[ntfs32]

I found this problem related to multiple nodes etcd.
I tried it many times and found: the master-api container is restarting and restarting...
and etcd pod logs connect others etcd node failed.
in ansible hosts file
add [etcd] whit openshift_ip:
[etcd]
master1 openshift_ip=x.x.x.x

then uninstall and deploy again success

[rajith77]

I found this problem related to multiple nodes etcd.
I tried it many times and found: the master-api container is restarting and restarting...
and etcd pod logs connect others etcd node failed.
in ansible hosts file
add [etcd] whit openshift_ip:
[etcd]
master1 openshift_ip=x.x.x.x

then uninstall and deploy again success

I had the same issue and your suggestion worked.
Can you please let me know where you found this information? I thought openshift_ip was deprecated from 3.10 onwards... so not sure how it fixed the issue. I was curious to know what the issue is and why setting openshift_ip fixed it.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci-robot]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.