Openshift-ansible: openshift-ansible-service-broker

[DavidTinoco]

Description

Hi, im having problems with openshift-ansible-service-broker, i have a pod in a continuous CrashLoopBackOff. Im deploying

• 1 LB
• 2 Masters
• 2 Infra
• 2 Apps

Version

~
[root@openshift2-master1 ~]# ansible --version
ansible 2.5.3
config file = /etc/ansible/ansible.cfg
configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/site-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.5 (default, Apr 11 2018, 07:36:10) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]
~
~
[root@openshift2-master1 openshift-ansible]# git describe
openshift-ansible-3.9.29-1-22-gdc3bed5
~

Steps To Reproduce

~
ansible-playbook -i ~/openshift-ansible/despliegue ~/openshift-ansible/playbooks/prerequisites.yml
ansible-playbook -i ~/openshift-ansible/despliegue ~/openshift-ansible/playbooks/deploy_cluster.yml
~

Pods, pv, pvc, logs Status

~~~
[root@openshift2-master1 ~]# oc get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
etcd-vol2-volume 1G RWO Retain Bound openshift-ansible-service-broker/etcd 2h

[root@openshift2-master1 ~]# oc get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
etcd Bound etcd-vol2-volume 1G RWO 6m

[root@openshift2-master1 ~]# oc get pod
NAME READY STATUS RESTARTS AGE
asb-1-deploy 1/1 Running 0 7m
asb-1-g8d2f 0/1 CrashLoopBackOff 6 6m
asb-etcd-1-9764t 1/1 Running 0 6m

[root@openshift2-master1 ~]# oc logs asb-1-g8d2f

Using config file mounted to /etc/ansible-service-broker/config.yaml

== Starting Ansible Service Broker... ==

2018/06/12 13:38:22 Unable to get log.logfile from config
[2018-06-12T13:38:22.943Z] [NOTICE] - Initializing clients...
[2018-06-12T13:38:22.945Z] [INFO] - == ETCD CX ==
[2018-06-12T13:38:22.945Z] [INFO] - EtcdHost: asb-etcd.openshift-ansible-service-broker.svc
[2018-06-12T13:38:22.945Z] [INFO] - EtcdPort: 2379
[2018-06-12T13:38:22.945Z] [INFO] - Endpoints: [https://asb-etcd.openshift-ansible-service-broker.svc:2379]
[2018-06-12T13:38:23.946Z] [ERROR] - client: etcd cluster is unavailable or misconfigured; error #0: client: endpoint https://asb-etcd.openshift-ansible-service-broker.svc:2379 exceeded header timeout

[root@openshift2-master1 ~]# oc logs asb-etcd-1-9764t
2018-06-12 13:31:48.445641 I | etcdmain: etcd Version: 3.3.7
2018-06-12 13:31:48.446872 I | etcdmain: Git SHA: 56536de55
2018-06-12 13:31:48.446897 I | etcdmain: Go Version: go1.9.6
2018-06-12 13:31:48.446912 I | etcdmain: Go OS/Arch: linux/amd64
2018-06-12 13:31:48.446925 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4
2018-06-12 13:31:48.448871 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2018-06-12 13:31:48.455529 I | embed: listening for peers on http://localhost:2380
2018-06-12 13:31:48.455846 I | embed: listening for client requests on 0.0.0.0:2379
2018-06-12 13:31:48.473948 I | etcdserver: name = default
2018-06-12 13:31:48.474046 I | etcdserver: data dir = /data
2018-06-12 13:31:48.474065 I | etcdserver: member dir = /data/member
2018-06-12 13:31:48.474082 I | etcdserver: heartbeat = 100ms
2018-06-12 13:31:48.474099 I | etcdserver: election = 1000ms
2018-06-12 13:31:48.474109 I | etcdserver: snapshot count = 100000
2018-06-12 13:31:48.474132 I | etcdserver: advertise client URLs = https://asb-etcd.openshift-ansible-service-broker.svc:2379
2018-06-12 13:31:48.480390 I | etcdserver: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 16
2018-06-12 13:31:48.480488 I | raft: 8e9e05c52164694d became follower at term 8
2018-06-12 13:31:48.480533 I | raft: newRaft 8e9e05c52164694d [peers: [], term: 8, commit: 16, applied: 0, lastindex: 16, lastterm: 8]
2018-06-12 13:31:48.495716 W | auth: simple token is not cryptographically signed
2018-06-12 13:31:48.498521 I | etcdserver: starting server... [version: 3.3.7, cluster version: to_be_decided]
2018-06-12 13:31:48.502330 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
2018-06-12 13:31:48.502736 N | etcdserver/membership: set the initial cluster version to 3.3
2018-06-12 13:31:48.502966 I | etcdserver/api: enabled capabilities for version 3.3
2018-06-12 13:31:48.507406 I | embed: ClientTLS: cert = /etc/tls/private/tls.crt, key = /etc/tls/private/tls.key, ca = , trusted-ca = /var/run/etcd-auth-secret/ca.crt, client-cert-auth = true, crl-file =
2018-06-12 13:31:49.784276 I | raft: 8e9e05c52164694d is starting a new election at term 8
2018-06-12 13:31:49.784472 I | raft: 8e9e05c52164694d became candidate at term 9
2018-06-12 13:31:49.784569 I | raft: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 9
2018-06-12 13:31:49.784649 I | raft: 8e9e05c52164694d became leader at term 9
2018-06-12 13:31:49.784719 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 9
2018-06-12 13:31:49.787677 I | etcdserver: published {Name:default ClientURLs:[https://asb-etcd.openshift-ansible-service-broker.svc:2379]} to cluster cdf818194e3a8c32
2018-06-12 13:31:49.787881 I | embed: ready to serve client requests
2018-06-12 13:31:49.799202 I | embed: serving client requests on [::]:2379
2018-06-12 13:31:49.823573 I | embed: rejected connection from "127.0.0.1:52440" (error "tls: failed to verify client's certificate: x509: certificate signed by unknown authority", ServerName "")
WARNING: 2018/06/12 13:31:49 Failed to dial 0.0.0.0:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.
~
I was having problems with the apiserver and solved it editing (inside de containter) the options ndots:5 to options ndots:4
~
[root@openshift2-master1 ~]# oc project kube-service-catalog
Now using project "kube-service-catalog" on server "https://openshift2-masterha.**.and:8443".
[root@openshift2-master1 ~]# oc rsh apiserver-8tpd6
sh-4.2# cat /etc/resolv.conf
nameserver 192.168.0.36
search kube-service-catalog.svc.cluster.local svc.cluster.local cluster.local
options ndots:4
sh-4.2# exit
~~~

[Turbots]

Im having the same error now in the asb etcd pod:

2018-09-26 11:18:42.074028 I | etcdserver: published {Name:default ClientURLs:[https://asb-etcd.openshift-ansible-service-broker.svc:2379 ]} to cluster cdf818194e3a8c32
2018-09-26 11:18:42.082306 I | embed: serving client requests on [::]:2379
2018-09-26 11:18:42.140644 I | embed: rejected connection from "127.0.0.1:38350" (error "tls: failed to verify client's certificate: x509: certificate signed by unknown authority", ServerName "")
WARNING: 2018/09/26 11:18:42 Failed to dial 0.0.0.0:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.

[Jmainguy]

I have the same problem.

openshift-ansible-service-broker    asb-1-2p8lh                                                       0/1       CrashLoopBackOff    6427       56d

 oc logs -f asb-1-2p8lh -n openshift-ansible-service-broker
Using config file mounted to /etc/ansible-service-broker/config.yaml
2018/09/27 19:10:05 Unable to get log.logfile from config
============================================================
==           Starting Ansible Service Broker...           ==
============================================================
[2018-09-27T19:10:05.918Z] [NOTICE] - Initializing clients...
[2018-09-27T19:10:05.965Z] [INFO] - == ETCD CX ==
[2018-09-27T19:10:05.965Z] [INFO] - EtcdHost: asb-etcd.openshift-ansible-service-broker.svc
[2018-09-27T19:10:05.965Z] [INFO] - EtcdPort: 2379
[2018-09-27T19:10:05.965Z] [INFO] - Endpoints: [https://asb-etcd.openshift-ansible-service-broker.svc:2379]
[2018-09-27T19:10:06.114Z] [ERROR] - client: etcd cluster is unavailable or misconfigured; error #0: x509: certificate signed by unknown authority

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci-robot]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.