Openshift-ansible: Installation of containerized OpenShift 3.9 origin fails on Verify API Server

[MarWestermann]

Description

Installation of OpenShift Origin fails on Phase: Verify API Server

I tried to install the containerized version of openshift 3.9 on CentOS. But that fails as it cannot connect to the api server.

Version

lsb_release -d

Description:    CentOS Linux release 7.5.1804 (Core)

ansible-version:

ansible 2.5.3
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Apr 11 2018, 07:36:10) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

git describe:

openshift-ansible-3.9.30-1-6-gf76a396

Steps To Reproduce

use hosts file provided under additional informations. checkout openshift-ansible, move to branch release-3.9.
run the prerequisites playbook followed by deploy_cluster.yml playbook.

Expected Results

Installation succeeds

Observed Results

installation fails on phase: Verify API Server

This is because the origin-master-api docker container cannot start successfully. And is restarted again and again with error message start_api.go:67] dial tcp [::1]:2379: getsockopt: connection refused (see additional information for complete logs)

Additional Information

on master:

docker ps:

CONTAINER ID        IMAGE                                    COMMAND                  CREATED             STATUS              PORTS               NAMES
9d4e338691c2        openshift/origin:v3.9.0                  "/usr/bin/openshif..."   24 seconds ago      Up 24 seconds                           origin-master-api
7e2244ac3b67        openshift/origin:v3.9.0                  "/usr/bin/openshif..."   About an hour ago   Up About an hour                        origin-master-controllers
cbc58eebc7e7        registry.fedoraproject.org/latest/etcd   "/usr/bin/etcd"          About an hour ago   Up About an hour                        etcd_container

docker logs origin-master-api

I0604 08:26:12.344663       1 start_api.go:104] Using a listen address override "0.0.0.0:443"
I0604 08:26:12.344956       1 plugins.go:83] Registered admission plugin "NamespaceLifecycle"
I0604 08:26:12.344966       1 plugins.go:83] Registered admission plugin "Initializers"
I0604 08:26:12.344979       1 plugins.go:83] Registered admission plugin "ValidatingAdmissionWebhook"
I0604 08:26:12.344982       1 plugins.go:83] Registered admission plugin "MutatingAdmissionWebhook"
I0604 08:26:12.345033       1 plugins.go:83] Registered admission plugin "AlwaysAdmit"
I0604 08:26:12.345040       1 plugins.go:83] Registered admission plugin "AlwaysPullImages"
I0604 08:26:12.345043       1 plugins.go:83] Registered admission plugin "LimitPodHardAntiAffinityTopology"
I0604 08:26:12.345049       1 plugins.go:83] Registered admission plugin "DefaultTolerationSeconds"
I0604 08:26:12.345056       1 plugins.go:83] Registered admission plugin "AlwaysDeny"
I0604 08:26:12.345064       1 plugins.go:83] Registered admission plugin "EventRateLimit"
I0604 08:26:12.345069       1 plugins.go:83] Registered admission plugin "DenyEscalatingExec"
I0604 08:26:12.345069       1 plugins.go:83] Registered admission plugin "DenyExecOnPrivileged"
I0604 08:26:12.345077       1 plugins.go:83] Registered admission plugin "ExtendedResourceToleration"
I0604 08:26:12.345087       1 plugins.go:83] Registered admission plugin "OwnerReferencesPermissionEnforcement"
I0604 08:26:12.345099       1 plugins.go:83] Registered admission plugin "ImagePolicyWebhook"
I0604 08:26:12.345106       1 plugins.go:83] Registered admission plugin "InitialResources"
I0604 08:26:12.345109       1 plugins.go:83] Registered admission plugin "LimitRanger"
I0604 08:26:12.345117       1 plugins.go:83] Registered admission plugin "NamespaceAutoProvision"
I0604 08:26:12.345126       1 plugins.go:83] Registered admission plugin "NamespaceExists"
I0604 08:26:12.345133       1 plugins.go:83] Registered admission plugin "NodeRestriction"
I0604 08:26:12.345138       1 plugins.go:83] Registered admission plugin "PersistentVolumeLabel"
I0604 08:26:12.345145       1 plugins.go:83] Registered admission plugin "PodNodeSelector"
I0604 08:26:12.345151       1 plugins.go:83] Registered admission plugin "PodPreset"
I0604 08:26:12.345156       1 plugins.go:83] Registered admission plugin "PodTolerationRestriction"
I0604 08:26:12.345162       1 plugins.go:83] Registered admission plugin "ResourceQuota"
I0604 08:26:12.345172       1 plugins.go:83] Registered admission plugin "PodSecurityPolicy"
I0604 08:26:12.345175       1 plugins.go:83] Registered admission plugin "Priority"
I0604 08:26:12.345184       1 plugins.go:83] Registered admission plugin "SecurityContextDeny"
I0604 08:26:12.345193       1 plugins.go:83] Registered admission plugin "ServiceAccount"
I0604 08:26:12.345199       1 plugins.go:83] Registered admission plugin "DefaultStorageClass"
I0604 08:26:12.345205       1 plugins.go:83] Registered admission plugin "PersistentVolumeClaimResize"
I0604 08:26:12.345211       1 plugins.go:83] Registered admission plugin "PVCProtection"
W0604 08:26:12.346205       1 start_master.go:287] Warning: aggregatorConfig.proxyClientInfo: Invalid value: "": if no client certificate is specified, the aggregator will be unable to proxy to remote servers, master start will continue.
I0604 08:26:12.350394       1 plugins.go:83] Registered admission plugin "NamespaceLifecycle"
I0604 08:26:12.350482       1 plugins.go:83] Registered admission plugin "Initializers"
I0604 08:26:12.350488       1 plugins.go:83] Registered admission plugin "ValidatingAdmissionWebhook"
I0604 08:26:12.350488       1 plugins.go:83] Registered admission plugin "MutatingAdmissionWebhook"
I0604 08:26:12.350525       1 plugins.go:83] Registered admission plugin "AlwaysAdmit"
I0604 08:26:12.350525       1 plugins.go:83] Registered admission plugin "AlwaysPullImages"
I0604 08:26:12.350549       1 plugins.go:83] Registered admission plugin "LimitPodHardAntiAffinityTopology"
I0604 08:26:12.350549       1 plugins.go:83] Registered admission plugin "DefaultTolerationSeconds"
I0604 08:26:12.350549       1 plugins.go:83] Registered admission plugin "AlwaysDeny"
I0604 08:26:12.350549       1 plugins.go:83] Registered admission plugin "EventRateLimit"
I0604 08:26:12.350565       1 plugins.go:83] Registered admission plugin "DenyEscalatingExec"
I0604 08:26:12.350565       1 plugins.go:83] Registered admission plugin "DenyExecOnPrivileged"
I0604 08:26:12.350565       1 plugins.go:83] Registered admission plugin "ExtendedResourceToleration"
I0604 08:26:12.350586       1 plugins.go:83] Registered admission plugin "OwnerReferencesPermissionEnforcement"
I0604 08:26:12.350591       1 plugins.go:83] Registered admission plugin "ImagePolicyWebhook"
I0604 08:26:12.350596       1 plugins.go:83] Registered admission plugin "InitialResources"
I0604 08:26:12.350596       1 plugins.go:83] Registered admission plugin "LimitRanger"
I0604 08:26:12.350597       1 plugins.go:83] Registered admission plugin "NamespaceAutoProvision"
I0604 08:26:12.350597       1 plugins.go:83] Registered admission plugin "NamespaceExists"
I0604 08:26:12.350615       1 plugins.go:83] Registered admission plugin "NodeRestriction"
I0604 08:26:12.350615       1 plugins.go:83] Registered admission plugin "PersistentVolumeLabel"
I0604 08:26:12.350615       1 plugins.go:83] Registered admission plugin "PodNodeSelector"
I0604 08:26:12.350632       1 plugins.go:83] Registered admission plugin "PodPreset"
I0604 08:26:12.350638       1 plugins.go:83] Registered admission plugin "PodTolerationRestriction"
I0604 08:26:12.350642       1 plugins.go:83] Registered admission plugin "ResourceQuota"
I0604 08:26:12.350647       1 plugins.go:83] Registered admission plugin "PodSecurityPolicy"
I0604 08:26:12.350652       1 plugins.go:83] Registered admission plugin "Priority"
I0604 08:26:12.350656       1 plugins.go:83] Registered admission plugin "SecurityContextDeny"
I0604 08:26:12.350661       1 plugins.go:83] Registered admission plugin "ServiceAccount"
I0604 08:26:12.350665       1 plugins.go:83] Registered admission plugin "DefaultStorageClass"
I0604 08:26:12.350670       1 plugins.go:83] Registered admission plugin "PersistentVolumeClaimResize"
I0604 08:26:12.350675       1 plugins.go:83] Registered admission plugin "PVCProtection"
I0604 08:26:12.350979       1 configgetter.go:53] Initializing cache sizes based on 0MB limit
I0604 08:26:12.357843       1 plugins.go:83] Registered admission plugin "NamespaceLifecycle"
I0604 08:26:12.357882       1 plugins.go:83] Registered admission plugin "Initializers"
I0604 08:26:12.357891       1 plugins.go:83] Registered admission plugin "ValidatingAdmissionWebhook"
I0604 08:26:12.357903       1 plugins.go:83] Registered admission plugin "MutatingAdmissionWebhook"
I0604 08:26:12.358009       1 plugins.go:83] Registered admission plugin "AlwaysAdmit"
I0604 08:26:12.358009       1 plugins.go:83] Registered admission plugin "AlwaysPullImages"
I0604 08:26:12.358009       1 plugins.go:83] Registered admission plugin "LimitPodHardAntiAffinityTopology"
I0604 08:26:12.358034       1 plugins.go:83] Registered admission plugin "DefaultTolerationSeconds"
I0604 08:26:12.358034       1 plugins.go:83] Registered admission plugin "AlwaysDeny"
I0604 08:26:12.358034       1 plugins.go:83] Registered admission plugin "EventRateLimit"
I0604 08:26:12.358034       1 plugins.go:83] Registered admission plugin "DenyEscalatingExec"
I0604 08:26:12.358069       1 plugins.go:83] Registered admission plugin "DenyExecOnPrivileged"
I0604 08:26:12.358069       1 plugins.go:83] Registered admission plugin "ExtendedResourceToleration"
I0604 08:26:12.358069       1 plugins.go:83] Registered admission plugin "OwnerReferencesPermissionEnforcement"
I0604 08:26:12.358107       1 plugins.go:83] Registered admission plugin "ImagePolicyWebhook"
I0604 08:26:12.358121       1 plugins.go:83] Registered admission plugin "InitialResources"
I0604 08:26:12.358129       1 plugins.go:83] Registered admission plugin "LimitRanger"
I0604 08:26:12.358129       1 plugins.go:83] Registered admission plugin "NamespaceAutoProvision"
I0604 08:26:12.358142       1 plugins.go:83] Registered admission plugin "NamespaceExists"
I0604 08:26:12.358142       1 plugins.go:83] Registered admission plugin "NodeRestriction"
I0604 08:26:12.358142       1 plugins.go:83] Registered admission plugin "PersistentVolumeLabel"
I0604 08:26:12.358142       1 plugins.go:83] Registered admission plugin "PodNodeSelector"
I0604 08:26:12.358172       1 plugins.go:83] Registered admission plugin "PodPreset"
I0604 08:26:12.358179       1 plugins.go:83] Registered admission plugin "PodTolerationRestriction"
I0604 08:26:12.358190       1 plugins.go:83] Registered admission plugin "ResourceQuota"
I0604 08:26:12.358196       1 plugins.go:83] Registered admission plugin "PodSecurityPolicy"
I0604 08:26:12.358206       1 plugins.go:83] Registered admission plugin "Priority"
I0604 08:26:12.358241       1 plugins.go:83] Registered admission plugin "SecurityContextDeny"
I0604 08:26:12.358241       1 plugins.go:83] Registered admission plugin "ServiceAccount"
I0604 08:26:12.358242       1 plugins.go:83] Registered admission plugin "DefaultStorageClass"
I0604 08:26:12.358249       1 plugins.go:83] Registered admission plugin "PersistentVolumeClaimResize"
I0604 08:26:12.358257       1 plugins.go:83] Registered admission plugin "PVCProtection"
I0604 08:26:12.360942       1 register.go:154] Admission plugin AlwaysAdmit is not enabled.  It will not be started.
I0604 08:26:12.361359       1 register.go:154] Admission plugin NamespaceAutoProvision is not enabled.  It will not be started.
I0604 08:26:12.361810       1 register.go:154] Admission plugin NamespaceExists is not enabled.  It will not be started.
I0604 08:26:12.362719       1 register.go:154] Admission plugin EventRateLimit is not enabled.  It will not be started.
I0604 08:26:12.363229       1 register.go:154] Admission plugin ProjectRequestLimit is not enabled.  It will not be started.
I0604 08:26:12.363997       1 register.go:154] Admission plugin openshift.io/RestrictSubjectBindings is not enabled.  It will not be started.
I0604 08:26:12.365945       1 register.go:154] Admission plugin RunOnceDuration is not enabled.  It will not be started.
I0604 08:26:12.366321       1 register.go:154] Admission plugin PodNodeConstraints is not enabled.  It will not be started.
I0604 08:26:12.367380       1 register.go:154] Admission plugin ClusterResourceOverride is not enabled.  It will not be started.
I0604 08:26:12.368861       1 register.go:154] Admission plugin ImagePolicyWebhook is not enabled.  It will not be started.
I0604 08:26:12.369238       1 register.go:154] Admission plugin PodPreset is not enabled.  It will not be started.
I0604 08:26:12.369620       1 register.go:154] Admission plugin InitialResources is not enabled.  It will not be started.
I0604 08:26:12.370842       1 register.go:154] Admission plugin SecurityContextDeny is not enabled.  It will not be started.
I0604 08:26:12.371601       1 register.go:154] Admission plugin PodSecurityPolicy is not enabled.  It will not be started.
I0604 08:26:12.371970       1 register.go:154] Admission plugin DenyEscalatingExec is not enabled.  It will not be started.
I0604 08:26:12.372393       1 register.go:154] Admission plugin DenyExecOnPrivileged is not enabled.  It will not be started.
I0604 08:26:12.373175       1 register.go:154] Admission plugin PersistentVolumeClaimResize is not enabled.  It will not be started.
I0604 08:26:12.373630       1 register.go:154] Admission plugin AlwaysPullImages is not enabled.  It will not be started.
I0604 08:26:12.374047       1 register.go:154] Admission plugin LimitPodHardAntiAffinityTopology is not enabled.  It will not be started.
W0604 08:26:12.374968       1 admission.go:66] PersistentVolumeLabel admission controller is deprecated. Please remove this controller from your configuration files and scripts.
I0604 08:26:12.376143       1 register.go:154] Admission plugin Priority is not enabled.  It will not be started.
I0604 08:26:12.376548       1 register.go:154] Admission plugin ExtendedResourceToleration is not enabled.  It will not be started.
I0604 08:26:12.377032       1 register.go:154] Admission plugin DefaultTolerationSeconds is not enabled.  It will not be started.
I0604 08:26:12.377387       1 register.go:154] Admission plugin PVCProtection is not enabled.  It will not be started.
I0604 08:26:12.377800       1 register.go:154] Admission plugin Initializers is not enabled.  It will not be started.
I0604 08:26:12.378128       1 register.go:154] Admission plugin MutatingAdmissionWebhook is not enabled.  It will not be started.
I0604 08:26:12.378531       1 register.go:154] Admission plugin ValidatingAdmissionWebhook is not enabled.  It will not be started.
I0604 08:26:12.379208       1 register.go:154] Admission plugin AlwaysDeny is not enabled.  It will not be started.
I0604 08:26:12.380248       1 plugins.go:83] Registered admission plugin "NamespaceLifecycle"
I0604 08:26:12.380347       1 plugins.go:83] Registered admission plugin "Initializers"
I0604 08:26:12.380347       1 plugins.go:83] Registered admission plugin "ValidatingAdmissionWebhook"
I0604 08:26:12.380358       1 plugins.go:83] Registered admission plugin "MutatingAdmissionWebhook"
I0604 08:26:12.380430       1 plugins.go:83] Registered admission plugin "AlwaysAdmit"
I0604 08:26:12.380449       1 plugins.go:83] Registered admission plugin "AlwaysPullImages"
I0604 08:26:12.380459       1 plugins.go:83] Registered admission plugin "LimitPodHardAntiAffinityTopology"
I0604 08:26:12.380465       1 plugins.go:83] Registered admission plugin "DefaultTolerationSeconds"
I0604 08:26:12.380468       1 plugins.go:83] Registered admission plugin "AlwaysDeny"
I0604 08:26:12.380487       1 plugins.go:83] Registered admission plugin "EventRateLimit"
I0604 08:26:12.380489       1 plugins.go:83] Registered admission plugin "DenyEscalatingExec"
I0604 08:26:12.380502       1 plugins.go:83] Registered admission plugin "DenyExecOnPrivileged"
I0604 08:26:12.380508       1 plugins.go:83] Registered admission plugin "ExtendedResourceToleration"
I0604 08:26:12.380509       1 plugins.go:83] Registered admission plugin "OwnerReferencesPermissionEnforcement"
I0604 08:26:12.380530       1 plugins.go:83] Registered admission plugin "ImagePolicyWebhook"
I0604 08:26:12.380530       1 plugins.go:83] Registered admission plugin "InitialResources"
I0604 08:26:12.380548       1 plugins.go:83] Registered admission plugin "LimitRanger"
I0604 08:26:12.380556       1 plugins.go:83] Registered admission plugin "NamespaceAutoProvision"
I0604 08:26:12.380566       1 plugins.go:83] Registered admission plugin "NamespaceExists"
I0604 08:26:12.380590       1 plugins.go:83] Registered admission plugin "NodeRestriction"
I0604 08:26:12.380615       1 plugins.go:83] Registered admission plugin "PersistentVolumeLabel"
I0604 08:26:12.380645       1 plugins.go:83] Registered admission plugin "PodNodeSelector"
I0604 08:26:12.380678       1 plugins.go:83] Registered admission plugin "PodPreset"
I0604 08:26:12.380689       1 plugins.go:83] Registered admission plugin "PodTolerationRestriction"
I0604 08:26:12.380694       1 plugins.go:83] Registered admission plugin "ResourceQuota"
I0604 08:26:12.380704       1 plugins.go:83] Registered admission plugin "PodSecurityPolicy"
I0604 08:26:12.380712       1 plugins.go:83] Registered admission plugin "Priority"
I0604 08:26:12.380731       1 plugins.go:83] Registered admission plugin "SecurityContextDeny"
I0604 08:26:12.380746       1 plugins.go:83] Registered admission plugin "ServiceAccount"
I0604 08:26:12.380753       1 plugins.go:83] Registered admission plugin "DefaultStorageClass"
I0604 08:26:12.380757       1 plugins.go:83] Registered admission plugin "PersistentVolumeClaimResize"
I0604 08:26:12.380769       1 plugins.go:83] Registered admission plugin "PVCProtection"
I0604 08:26:12.529715       1 master_config.go:445] Initializing cache sizes based on 0MB limit
I0604 08:26:12.529846       1 master_config.go:509] Using the lease endpoint reconciler with TTL=15s and interval=10s
I0604 08:26:12.531479       1 start_master.go:558] Starting master on 0.0.0.0:443 (v3.9.0+ee4d1e7-20)
I0604 08:26:12.531522       1 start_master.go:559] Public master address is https://master1.testcloud.mycompany.de
I0604 08:26:12.531541       1 start_master.go:566] Using images from "openshift/origin-<component>:v3.9.0"
F0604 08:26:42.532434       1 start_api.go:67] dial tcp [::1]:2379: getsockopt: connection refused

docker logs etcd_container

2018-06-04 07:06:25.222436 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=https://80.240.27.61:2379
2018-06-04 07:06:25.222604 I | pkg/flags: recognized and used environment variable ETCD_CERT_FILE=/etc/etcd/server.crt
2018-06-04 07:06:25.222622 I | pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=true
2018-06-04 07:06:25.222636 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd/
2018-06-04 07:06:25.222647 I | pkg/flags: recognized and used environment variable ETCD_DEBUG=False
2018-06-04 07:06:25.222659 I | pkg/flags: recognized and used environment variable ETCD_ELECTION_TIMEOUT=2500
2018-06-04 07:06:25.222679 I | pkg/flags: recognized and used environment variable ETCD_HEARTBEAT_INTERVAL=500
2018-06-04 07:06:25.222714 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=https://80.240.27.61:2380
2018-06-04 07:06:25.222741 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=master1.testcloud.mycompany.de=https://80.240.27.61:2380
2018-06-04 07:06:25.222808 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
2018-06-04 07:06:25.222822 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-1
2018-06-04 07:06:25.222833 I | pkg/flags: recognized and used environment variable ETCD_KEY_FILE=/etc/etcd/server.key
2018-06-04 07:06:25.222847 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=https://80.240.27.61:2379
2018-06-04 07:06:25.222873 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=https://80.240.27.61:2380
2018-06-04 07:06:25.225400 I | pkg/flags: recognized and used environment variable ETCD_NAME=master1.testcloud.mycompany.de
2018-06-04 07:06:25.225432 I | pkg/flags: recognized and used environment variable ETCD_PEER_CERT_FILE=/etc/etcd/peer.crt
2018-06-04 07:06:25.225445 I | pkg/flags: recognized and used environment variable ETCD_PEER_CLIENT_CERT_AUTH=true
2018-06-04 07:06:25.225457 I | pkg/flags: recognized and used environment variable ETCD_PEER_KEY_FILE=/etc/etcd/peer.key
2018-06-04 07:06:25.225474 I | pkg/flags: recognized and used environment variable ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/ca.crt
2018-06-04 07:06:25.225493 I | pkg/flags: recognized and used environment variable ETCD_QUOTA_BACKEND_BYTES=4294967296
2018-06-04 07:06:25.225511 I | pkg/flags: recognized and used environment variable ETCD_TRUSTED_CA_FILE=/etc/etcd/ca.crt
2018-06-04 07:06:25.225573 I | etcdmain: etcd Version: 3.2.7
2018-06-04 07:06:25.226719 I | etcdmain: Git SHA: bb66589
2018-06-04 07:06:25.226736 I | etcdmain: Go Version: go1.9beta2
2018-06-04 07:06:25.226748 I | etcdmain: Go OS/Arch: linux/amd64
2018-06-04 07:06:25.226758 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4
2018-06-04 07:06:25.227916 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2018-06-04 07:06:25.227979 I | embed: peerTLS: cert = /etc/etcd/peer.crt, key = /etc/etcd/peer.key, ca = , trusted-ca = /etc/etcd/ca.crt, client-cert-auth = true
2018-06-04 07:06:25.230176 I | embed: listening for peers on https://80.240.27.61:2380
2018-06-04 07:06:25.230297 I | embed: listening for client requests on 80.240.27.61:2379
2018-06-04 07:06:25.247817 I | etcdserver: name = master1.testcloud.mycompany.de
2018-06-04 07:06:25.247847 I | etcdserver: data dir = /var/lib/etcd/
2018-06-04 07:06:25.247863 I | etcdserver: member dir = /var/lib/etcd/member
2018-06-04 07:06:25.247863 I | etcdserver: heartbeat = 500ms
2018-06-04 07:06:25.247863 I | etcdserver: election = 2500ms
2018-06-04 07:06:25.247898 I | etcdserver: snapshot count = 100000
2018-06-04 07:06:25.247918 I | etcdserver: advertise client URLs = https://80.240.27.61:2379
2018-06-04 07:06:25.248371 I | etcdserver: restarting member 6486934f8042dfa3 in cluster 415c958bf1a3eec at commit index 4
2018-06-04 07:06:25.248459 I | raft: 6486934f8042dfa3 became follower at term 2
2018-06-04 07:06:25.248483 I | raft: newRaft 6486934f8042dfa3 [peers: [], term: 2, commit: 4, applied: 0, lastindex: 4, lastterm: 2]
2018-06-04 07:06:25.251827 W | auth: simple token is not cryptographically signed
2018-06-04 07:06:25.254310 I | etcdserver: starting server... [version: 3.2.7, cluster version: to_be_decided]
2018-06-04 07:06:25.254424 I | embed: ClientTLS: cert = /etc/etcd/server.crt, key = /etc/etcd/server.key, ca = , trusted-ca = /etc/etcd/ca.crt, client-cert-auth = true
2018-06-04 07:06:25.255831 I | etcdserver/membership: added member 6486934f8042dfa3 [https://80.240.27.61:2380] to cluster 415c958bf1a3eec
2018-06-04 07:06:25.255998 N | etcdserver/membership: set the initial cluster version to 3.2
2018-06-04 07:06:25.256047 I | etcdserver/api: enabled capabilities for version 3.2
2018-06-04 07:06:25.749020 I | raft: 6486934f8042dfa3 is starting a new election at term 2
2018-06-04 07:06:25.749129 I | raft: 6486934f8042dfa3 became candidate at term 3
2018-06-04 07:06:25.749166 I | raft: 6486934f8042dfa3 received MsgVoteResp from 6486934f8042dfa3 at term 3
2018-06-04 07:06:25.749266 I | raft: 6486934f8042dfa3 became leader at term 3
2018-06-04 07:06:25.749311 I | raft: raft.node: 6486934f8042dfa3 elected leader 6486934f8042dfa3 at term 3
2018-06-04 07:06:25.750235 I | etcdserver: published {Name:master1.testcloud.mycompany.de ClientURLs:[https://80.240.27.61:2379]} to cluster 415c958bf1a3eec
2018-06-04 07:06:25.750327 I | embed: ready to serve client requests
2018-06-04 07:06:25.751320 I | embed: serving client requests on 80.240.27.61:2379
2018-06-04 07:06:25.759488 I | etcdserver/api/v3rpc: Failed to dial 80.240.27.61:2379: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.

Your inventory file (especially any non-standard configuration parameters)

[OSEv3:children]
masters
nodes
etcd
nfs

[OSEv3:vars]
ansible_ssh_user=root
openshift_deployment_type=origin
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]
containerized=true
openshift_release=v3.9.0
openshift_image_tag=v3.9.0
openshift_install_examples=true
enable_docker_excluder=false
enable_openshift_excluder=false
openshift_enable_unsupported_configurations=True

osm_use_cockpit=true
osm_cockpit_plugins=['cockpit-kubernetes']
openshift_master_cluster_method=native
osm_custom_cors_origins=['.*']


openshift_disable_check=disk_availability,docker_storage,memory_availability,docker_image_availability
openshift_master_cluster_hostname=master1.testcloud.mycompany.de
openshift_master_cluster_public_hostname=master1.testcloud.mycompany.de
openshift_master_default_subdomain=testapp.mycompany.de
openshift_clock_enabled=true
openshift_master_api_port=443
openshift_master_console_port=443

openshift_hosted_router_replicas=1

openshift_enable_service_catalog=false

#######
openshift_hosted_manage_registry=true
openshift_hosted_registry_storage_kind=nfs
openshift_hosted_registry_storage_access_modes=['ReadWriteMany']
openshift_hosted_registry_storage_nfs_directory=/exports
openshift_hosted_registry_storage_nfs_options='*(rw,root_squash)'
openshift_hosted_registry_storage_volume_name=registry
openshift_hosted_registry_storage_volume_size=10Gi
openshift_hosted_registry_replicas=1
openshift_hosted_registry_deploy=true

openshift_logging_install_logging=true
openshift_hosted_logging_enable_ops_cluster=True
openshift_logging_use_ops=true
openshift_logging_kibana_hostname=kibana.testapp.mycompany.de
openshift_logging_storage_kind=nfs
openshift_logging_storage_access_modes=['ReadWriteOnce']
openshift_logging_storage_nfs_directory=/exports
openshift_logging_storage_nfs_options='*(rw,root_squash)'
openshift_logging_storage_volume_name=logging
openshift_logging_storage_volume_size=10Gi
openshift_logging_storage_labels={'storage': 'logging'}
openshift_hosted_logging_deploy=true


openshift_metrics_install_metrics=true
openshift_metrics_storage_kind=nfs
openshift_metrics_storage_access_modes=['ReadWriteOnce']
openshift_metrics_storage_nfs_directory=/exports
openshift_metrics_storage_nfs_options='*(rw,root_squash)'
openshift_metrics_storage_volume_name=metrics
openshift_metrics_storage_volume_size=10Gi
openshift_metrics_storage_labels={'storage': 'metrics'}
openshift_metrics_hawkular_hostname=hawkular-metrics.testapp.mycompany.de
openshift_metrics_cassandra_storage_type=nfs
openshift_hosted_metrics_deploy=true

openshift_hosted_prometheus_deploy=false
openshift_prometheus_storage_kind=nfs
openshift_prometheus_storage_access_modes=['ReadWriteOnce']
openshift_prometheus_storage_nfs_directory=/exports
openshift_prometheus_storage_nfs_options='*(rw,root_squash)'
openshift_prometheus_storage_volume_name=prometheus
openshift_prometheus_storage_volume_size=10Gi
openshift_prometheus_storage_labels={'storage': 'prometheus'}

openshift_prometheus_storage_type='pvc'
openshift_prometheus_alertmanager_storage_kind=nfs
openshift_prometheus_alertmanager_storage_access_modes=['ReadWriteOnce']
openshift_prometheus_alertmanager_storage_nfs_directory=/exports
openshift_prometheus_alertmanager_storage_nfs_options='*(rw,root_squash)'
openshift_prometheus_alertmanager_storage_volume_name=prometheus-alertmanager
openshift_prometheus_alertmanager_storage_volume_size=10Gi
openshift_prometheus_alertmanager_storage_labels={'storage': 'prometheus-alertmanager'}
openshift_prometheus_alertmanager_storage_type='pvc'

openshift_prometheus_alertbuffer_storage_kind=nfs
openshift_prometheus_alertbuffer_storage_access_modes=['ReadWriteOnce']
openshift_prometheus_alertbuffer_storage_nfs_directory=/exports
openshift_prometheus_alertbuffer_storage_nfs_options='*(rw,root_squash)'
openshift_prometheus_alertbuffer_storage_volume_name=prometheus-alertbuffer
openshift_prometheus_alertbuffer_storage_volume_size=10Gi
openshift_prometheus_alertbuffer_storage_labels={'storage': 'prometheus-alertbuffer'}
openshift_prometheus_alertbuffer_storage_type='pvc'

#######

openshift_master_overwrite_named_certificates=true
openshift_set_hostname=True


[masters]
master1.testcloud.mycompany.de

[etcd]
master1.testcloud.mycompany.de

[nfs]
master1.testcloud.mycompany.de

[nodes]
master1.testcloud.mycompany.de
node1.testcloud.mycompany.de  openshift_node_labels="{'region': 'infra', 'zone': 'default', 'router': 'router'}" openshift_schedulable=true
node2.testcloud.mycompany.de  openshift_node_labels="{'region': 'infra', 'zone': 'default', 'router': 'router'}" openshift_schedulable=true
node3.testcloud.mycompany.de  openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_schedulable=true
node4.testcloud.mycompany.de  openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_schedulable=true

[MarWestermann]

I reinstalled all master and nodes and tested again with the same result.
So the actual issue seems to be, that the master-api container cannot connect to etcd container with error message: bad certificate (see last line of log from etcd container)

[stobias123]

Seeing this exact same issue.
oops - wrong person on cc.

[MarWestermann]

The issue seems to be that etcd binds to the wrong interface:

netstat -tulpn

tcp        0      0 80.XXX.XX.XX:2379       0.0.0.0:*               LISTEN      23413/etcd          
tcp        0      0 80.XXX.XX.XX:2380       0.0.0.0:*               LISTEN      23413/etcd

The etcd configuration looks the following:

ETCD_LISTEN_PEER_URLS=https://80.XXX.XX.XX:2380
ETCD_LISTEN_CLIENT_URLS=https://80.XXX.XX.XX:2379

But the openshift-api container resolves master1.testcloud.mycompany.de to localhost (::1)
This is because this host is listen in /etc/hosts (not sure if this was added by openshift ansible installation):

127.0.0.1 master1.testcloud.mycompany.de
::1       master1.testcloud.mycompany.de

And the master-configuration:

etcdClientInfo:
  ca: master.etcd-ca.crt
  certFile: master.etcd-client.crt
  keyFile: master.etcd-client.key
  urls:
    - https://master1.testcloud.mycompany.de:2379

I'm not a network guy and that is why I'm not sure what the correct way will be to resolv that.. So please help - thank you very much

[MarWestermann]

ok after thinking about it, I just removed the lines from /etc/hosts so that the domain resolves to the external IP and now it can connect to etcd ;-)
@stobias123 maybe that will help you too

[stobias123]

For the record... Changing /etc/hosts did not help me.

[camrossi]

I had the exact same issue, did a yum update and after that all deployed flawlessly.
I am not sure what was the root cause to be honest :-)