Openshift-ansible: call to 'certificates_to_synchronize' causes openshift_master_certificates to abend with 'ERROR! |failed expects hostvars is a dict'

[AllanCano]

Description

Provide a brief description of your issue here. For example:

On a multi master install deploy_cluster.yml terminates after task
"openshift_master_certificates : Generate the loopback master client config"
with the error message
"ERROR! |failed expects hostvars is a dict"

Version

• oc verions

# oc version
oc v3.7.1+c2ce2c0-1
kubernetes v1.7.6+a08f5eeb62
features: Basic-Auth GSSAPI Kerberos SPNEGO

• Ansible verions

# ansible --version
ansible 2.6.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Aug  4 2017, 00:39:18) [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]

• Ansible Openshift Playbooks

# cd /etc/ansible/
# mkdir playbooks
# cd playbooks
# git clone https://github.com/openshift/openshift-ansible

• CentOS version

# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)

• Openshift version

# openshift version
openshift v3.7.1+c2ce2c0-1
kubernetes v1.7.6+a08f5eeb62
etcd 3.2.8

Inventory Hosts Files

#---- Set variables common for all OSEv3 hosts
[OSEv3:vars]
ansible_ssh_user=root
debug_level='2'
containerized='false'
openshift_deployment_type=origin
os_firewall_use_firewalld='true'

#-- Configure networking
#openshift_master_default_subdomain=fossil.com
openshift_use_openshift_sdn=true
#openshift_use_flannel=true
#flannel_interface=bond0

# Uncomment the following to enable htpasswd authentication; defaults to DenyAllPasswordIdentityProvider.
#openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]

# Native high availbility cluster method with optional load balancer.
# If no lb group is defined installer assumes that a load balancer has
# been preconfigured. For installation the value of
# openshift_master_cluster_hostname must resolve to the load balancer
# or to one or all of the masters defined in the inventory if no load
# balancer is present.
#-- When we get to production use an external LB
openshift_master_cluster_method=native
#openshift_master_cluster_hostname=uusrcdkub00.fossil.com
#openshift_master_cluster_public_hostname=uusrcdkub00.fossil.com

# apply updated node defaults
openshift_node_kubelet_args={'pods-per-core': ['10'], 'max-pods': ['250'], 'image-gc-high-threshold': ['90'], 'image-gc-low-threshold': ['80']}

# enable ntp on masters to ensure proper failover
openshift_clock_enabled=true

# Setup rolling updates at the service level
openshift_rolling_restart_mode='services'

# Change all the certificates to last for 10 years.
openshift_hosted_registry_cert_expire_days='720'
openshift_ca_cert_expire_days='720'
openshift_node_cert_expire_days='720'
openshift_master_cert_expire_days='720'
etcd_ca_default_days='720'


#---- host group for masters
[masters]
uusrcdkub01.fossil.com
uusrcdkub02.fossil.com
uusrcdkub03.fossil.com

#---- host group for etcd
[etcd]
uusrcdkub01.fossil.com
uusrcdkub02.fossil.com
uusrcdkub03.fossil.com

#---- Specify load balancer host
#[lb]
#uusrcdkub00.fossil.com

#---- host group for nodes, includes region info
[nodes]
uusrcdkub01.fossil.com openshift_schedulable=true       openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
uusrcdkub02.fossil.com openshift_schedulable=true       openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
uusrcdkub03.fossil.com openshift_schedulable=true       openshift_node_labels="{'region': 'infra', 'zone': 'default'}"


# Create an OSEv3 group that contains the master, nodes, etcd, and lb groups.
# The lb group lets Ansible configure HAProxy as the load balancing solution.
# Comment lb out if your load balancer is pre-configured.
[OSEv3:children]
masters
nodes
etcd
#lb

Steps To Reproduce

• After base CentOS 7.3 install

1. Install pre-reqs and update the OS

yum -y install wget git net-tools bind-utils iptables-services bridge-utils bash-completion kexec-tools sos psacct curl docker etcd atomic pyOpenSSL.x86_64 python-cryptography python-lxml
yum update
systemctl reboot

1. Increase the root volume group to use the complete disk

fdisk /dev/sda
d
2
n
primary
2
default
default

systemctl reboot
pvresize /dev/sda2

1. Update the docker config to use a LV for the docker storage

systemctl stop docker
rm -rf /var/lib/docker
vi /etc/sysconfig/docker-storage-setup
# File contents below
/bin/docker-storage-setup
systemctl start docker
systemctl enbale docker
docker info
lvs

File: /etc/sysconfig/docker-storage-setup

# Edit this file to override any configuration options specified in
# /usr/lib/docker-storage-setup/docker-storage-setup.
#
# For more details refer to "man docker-storage-setup"
#STORAGE_DRIVER
#DEVS=/dev/sda
VG=centos
GROWPART=enable
AUTO_EXTEND_POOL=enable
MIN_DATA_SIZE=8G
POOL_AUTOEXTEND_THRESHOLD=60
POOL_AUTOEXTEND_PERCENT=10

• Install Ansible on the first master
  We need at least Ansible 2.4.3. At the time of writing this doc the latest published was 2.4.2. So, we need to role our own RPM. To build Ansible we need:

    yum -y install asciidoc rpm-build python2-devel

• Build and install the latest Ansible

    mkdir Development
    git clone https://github.com/ansible/ansible.git
    cd ansible
    make rpm
    cd /root/Development/ansible/rpm-build
    rpm --install ansible-2.6.0-100.git201803142047.0cf2ecb.devel.el7.centos.noarch.rpm

• Pull the OpenShift Ansible Source

    cd /etc/ansible/
    mkdir playbooks
    cd playbooks
    git clone https://github.com/openshift/openshift-ansible

• Create the Ansible Host File

  vi /etc/ansbile/hosts

File contents listed above.

• Run the pre-reqs and deploy

  cd /etc/ansible/playbooks/openshift-ansible/playbooks/
  ansible-playbook -i /etc/ansible/hosts $PWD/prerequisites.yml

  cd /etc/ansible/playbooks/openshift-ansible/playbooks/openshift-checks
  vi roles/openshift_health_checker/openshift_checks/package_version.py

I updated the allowed docker versions for (3,7) to include 1.13 and 1.13.1

  ansible-playbook -i /etc/ansible/hosts $PWD/pre-install.yml

  cd /etc/ansible/playbooks/openshift-ansible/playbooks/
  ansible-playbook -vvv -i /etc/ansible/hosts $PWD/deploy_cluster.yml

Expected Results

Describe what you expected to happen.

I expected the install and configuration to complete

Observed Results

Everything is chugging along find. The verbose output before the abend is listed below. I don't see any other errors in the output until this.

TASK [openshift_master_certificates : Generate the loopback master client config] ***************************************************************************************
task path: /etc/ansible/playbooks/openshift-ansible/roles/openshift_master_certificates/tasks/main.yml:63
Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py
<uusrcdkub01.fossil.com> ESTABLISH SSH CONNECTION FOR USER: root
<uusrcdkub01.fossil.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 uusrcdkub01.fossil.com '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<uusrcdkub01.fossil.com> (0, '/root\n', '')
<uusrcdkub01.fossil.com> ESTABLISH SSH CONNECTION FOR USER: root
<uusrcdkub01.fossil.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 uusrcdkub01.fossil.com '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1521658222.4-93760209595884 `" && echo ansible-tmp-1521658222.4-93760209595884="` echo /root/.ansible/tmp/ansible-tmp-1521658222.4-93760209595884 `" ) && sleep 0'"'"''
<uusrcdkub01.fossil.com> (0, 'ansible-tmp-1521658222.4-93760209595884=/root/.ansible/tmp/ansible-tmp-1521658222.4-93760209595884\n', '')
<uusrcdkub01.fossil.com> PUT /root/.ansible/tmp/ansible-local-20837_E4d2Q/tmpbS7CiZ TO /root/.ansible/tmp/ansible-tmp-1521658222.4-93760209595884/command.py
<uusrcdkub01.fossil.com> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 '[uusrcdkub01.fossil.com]'
...
Skip a lot
...
Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py
<uusrcdkub01.fossil.com> ESTABLISH SSH CONNECTION FOR USER: root
<uusrcdkub01.fossil.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 uusrcdkub01.fossil.com '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<uusrcdkub01.fossil.com> (0, '/root\n', '')
<uusrcdkub01.fossil.com> ESTABLISH SSH CONNECTION FOR USER: root
<uusrcdkub01.fossil.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 uusrcdkub01.fossil.com '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726 `" && echo ansible-tmp-1521656686.1-214052715328726="` echo /root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726 `" ) && sleep 0'"'"''
<uusrcdkub01.fossil.com> (0, 'ansible-tmp-1521656686.1-214052715328726=/root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726\n', '')
<uusrcdkub01.fossil.com> PUT /root/.ansible/tmp/ansible-local-115241x1n7s/tmp4R_seg TO /root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726/command.py
<uusrcdkub01.fossil.com> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 '[uusrcdkub01.fossil.com]'
<uusrcdkub01.fossil.com> (0, 'sftp> put /root/.ansible/tmp/ansible-local-115241x1n7s/tmp4R_seg /root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726/command.py\n', '')
<uusrcdkub01.fossil.com> ESTABLISH SSH CONNECTION FOR USER: root
<uusrcdkub01.fossil.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 uusrcdkub01.fossil.com '/bin/sh -c '"'"'chmod u+x /root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726/ /root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726/command.py && sleep 0'"'"''
<uusrcdkub01.fossil.com> (0, '', '')
<uusrcdkub01.fossil.com> ESTABLISH SSH CONNECTION FOR USER: root
<uusrcdkub01.fossil.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 -tt uusrcdkub01.fossil.com '/bin/sh -c '"'"'/usr/bin/python /root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726/command.py && sleep 0'"'"''
<uusrcdkub01.fossil.com> (0, '\r\n{"invocation": {"module_args": {"warn": true, "executable": null, "_uses_shell": false, "_raw_params": "oc adm create-api-client-config\\n --certificate-authority=/etc/origin/master/ca.crt\\n  --client-dir=/etc/origin/generated-configs/master-uusrcdkub03.fossil.com\\n --groups=system:masters,system:openshift-master\\n --master=https://uusrcdkub03.fossil.com:8443\\n --public-master=https://uusrcdkub03.fossil.com:8443\\n --signer-cert=/etc/origin/master/ca.crt\\n --signer-key=/etc/origin/master/ca.key\\n --signer-serial=/etc/origin/master/ca.serial.txt\\n --user=system:openshift-master\\n --basename=openshift-master\\n --expire-days=720", "removes": null, "creates": "/etc/origin/generated-configs/master-uusrcdkub03.fossil.com/openshift-master.kubeconfig", "chdir": null, "stdin": null}}, "cmd": "oc adm create-api-client-config\\n --certificate-authority=/etc/origin/master/ca.crt\\n  --client-dir=/etc/origin/generated-configs/master-uusrcdkub03.fossil.com\\n --groups=system:masters,system:openshift-master\\n --master=https://uusrcdkub03.fossil.com:8443\\n --public-master=https://uusrcdkub03.fossil.com:8443\\n --signer-cert=/etc/origin/master/ca.crt\\n --signer-key=/etc/origin/master/ca.key\\n --signer-serial=/etc/origin/master/ca.serial.txt\\n --user=system:openshift-master\\n --basename=openshift-master\\n --expire-days=720", "stdout": "skipped, since /etc/origin/generated-configs/master-uusrcdkub03.fossil.com/openshift-master.kubeconfig exists", "rc": 0, "changed": false}\r\n', 'Shared connection to uusrcdkub01.fossil.com closed.\r\n')
<uusrcdkub01.fossil.com> ESTABLISH SSH CONNECTION FOR USER: root
<uusrcdkub01.fossil.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 uusrcdkub01.fossil.com '/bin/sh -c '"'"'rm -f -r /root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726/ > /dev/null 2>&1 && sleep 0'"'"''
<uusrcdkub01.fossil.com> (0, '', '')
ok: [uusrcdkub01.fossil.com -> uusrcdkub01.fossil.com] => (item=uusrcdkub03.fossil.com) => {
    "changed": false,
    "cmd": "oc adm create-api-client-config\n --certificate-authority=/etc/origin/master/ca.crt\n  --client-dir=/etc/origin/generated-configs/master-uusrcdkub03.fossil.com\n --groups=system:masters,system:openshift-master\n --master=https://uusrcdkub03.fossil.com:8443\n --public-master=https://uusrcdkub03.fossil.com:8443\n --signer-cert=/etc/origin/master/ca.crt\n --signer-key=/etc/origin/master/ca.key\n --signer-serial=/etc/origin/master/ca.serial.txt\n --user=system:openshift-master\n --basename=openshift-master\n --expire-days=720",
    "invocation": {
        "module_args": {
            "_raw_params": "oc adm create-api-client-config\n --certificate-authority=/etc/origin/master/ca.crt\n  --client-dir=/etc/origin/generated-configs/master-uusrcdkub03.fossil.com\n --groups=system:masters,system:openshift-master\n --master=https://uusrcdkub03.fossil.com:8443\n --public-master=https://uusrcdkub03.fossil.com:8443\n --signer-cert=/etc/origin/master/ca.crt\n --signer-key=/etc/origin/master/ca.key\n --signer-serial=/etc/origin/master/ca.serial.txt\n --user=system:openshift-master\n --basename=openshift-master\n --expire-days=720",
            "_uses_shell": false,
            "chdir": null,
            "creates": "/etc/origin/generated-configs/master-uusrcdkub03.fossil.com/openshift-master.kubeconfig",
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "item": "uusrcdkub03.fossil.com",
    "rc": 0,
    "stdout": "skipped, since /etc/origin/generated-configs/master-uusrcdkub03.fossil.com/openshift-master.kubeconfig exists",
    "stdout_lines": [
        "skipped, since /etc/origin/generated-configs/master-uusrcdkub03.fossil.com/openshift-master.kubeconfig exists"
    ]
}
ERROR! |failed expects hostvars is a dict

[AllanCano]

Ansible -vvv output

[AllanCano]

Non-verbose output

[AllanCano]

Found that the error is coming from:

/etc/ansible/playbooks/openshift-ansible/roles/lib_utils/filter_plugins/openshift_master.py line 487.

_hostvars[inventory_hostname]_ is being passed in to the custom filter _certificates_to_synchronize_

I did a little googling and found that the type for __hostvars[inventory_hostname]__ is returned as a _class_ instead of a _dict_. I'm not a python scripter so this is taking longer than I hoped to work through anyone with an idea here on what to do?

debug code via altering line 487:

raise errors.AnsibleFilterError("|failed expects hostvars is a dict, found %s" % type(hostvars) )

[AllanCano]

OK. I think I understand what's happening and how I got around the problem, though, I'm at least a week away from knowing enough python to prove/disprove anything.

In _/etc/ansible/playbooks/openshift-ansible/roles/openshift_master_certificates/tasks/main.yml_ around line 63 I simply changed the _with_items_ clause from
- "{{ hostvars[inventory_hostname] | certificates_to_synchronize }}"
to
- "{{ hostvars[inventory_hostname]['ansible_facts'] | certificates_to_synchronize }}"
and everything progressed.

I think since the _hostvars[inventory]_ is magic it doesn't exist as a dictionary until accessed. So, accessing a dictionary variable _ansible_facts_ solves 2 problems of instantiating the variable and passing an actual dictionary to the customer filter.

Don't know how to raise this as a bug (possible), but if someone in the know ever reads this please let me know if I guessed right as to why the work-a-round worked and how to place this as a bug report.

[Pectojin]

@AllanCano, thanks for the detective work.

I got the same error in openshift_master.py (although it's on line 516 in my case).
Making your change to openshift_master_certificates/tasks/main.yml (now on line 108) also helped me past the failure.

For the sake of completeness, here's my openshift version settings:

openshift_deployment_type=origin
openshift_release=v3.7
openshift_pkg_version=-3.7.1
openshift_image_tag=v3.7.1

And running ansible 2.6.0.

[Pectojin]

Just FYI

After fixing this issue manually I continued and ran into #4985, where it was suggested to work under ansible 2.4.2, so I downgraded.

After downgrading the above fix did not work, so I had to un-do it to continue installation under ansible 2.4.2.

[onknows]

This fix works for me for deploy_cluster but not for
playbooks/openshift-master/redeploy-certificates.yml

This will again fail with this message.

To successfully run redeploy this fix needs to be reversed I think. So two separate 3.9 branches are necessary to successful run 3.9 branch. This looks like a mess.

[djfoley01]

@AllanCano Not sure if anyone confirmed this, but your resolution of modifying /etc/ansible/playbooks/openshift-ansible/roles/openshift_master_certificates/tasks/main.yml has fixed our issue. This was line 95 for us.

[damiandragowski]

Hi @all,

I have the same issue with deploying openshift, but after apply this patch above everythink work fine.
Patch was made in origin/release-3.9 branch. I hope this will solve yours problems :).

Greatings,
Damian

diff --git a/playbooks/init/validate_hostnames.yml b/playbooks/init/validate_hostnames.yml
index b49f7dd..88b275f 100644
--- a/playbooks/init/validate_hostnames.yml
+++ b/playbooks/init/validate_hostnames.yml
@@ -40,4 +40,4 @@
     when:
     - openshift_ip is defined
     - openshift_ip not in ansible_all_ipv4_addresses
-    - openshift_ip_check | default(true) | bool
+    - openshift_ip_check | default(false) | bool
diff --git a/roles/container_runtime/defaults/main.yml b/roles/container_runtime/defaults/main.yml
index 4e27541..e54ed1f 100644
--- a/roles/container_runtime/defaults/main.yml
+++ b/roles/container_runtime/defaults/main.yml
@@ -87,7 +87,7 @@ openshift_use_crio_only: False
 l_openshift_image_tag_default: "{{ 'v' ~ openshift_release if openshift_release is defined else 'latest' }}"
 l_openshift_image_tag: "{{ openshift_image_tag | default(l_openshift_image_tag_default) | string}}"

-l_required_docker_version: '1.12'
+l_required_docker_version: '1.13'

diff --git a/roles/lib_utils/filter_plugins/openshift_master.py b/roles/lib_utils/filter_plugins/openshift_master.py
index e67b19c..062b8bf 100644
--- a/roles/lib_utils/filter_plugins/openshift_master.py
+++ b/roles/lib_utils/filter_plugins/openshift_master.py
@@ -483,8 +483,6 @@ class FilterModule(object):
     @staticmethod
     def certificates_to_synchronize(hostvars, include_keys=True, include_ca=True):
         ''' Return certificates to synchronize based on facts. '''
-        if not issubclass(type(hostvars), dict):
-            raise errors.AnsibleFilterError("|failed expects hostvars is a dict")
         certs = ['admin.crt',
                  'admin.key',
                  'admin.kubeconfig',

[s-u-b-h-a-k-a-r]

I am also facing the same error while installing openshift 3.9.0. What is the actual fix that needs to be made to fix this issue 'ERROR! |failed expects hostvars '

[IRobL]

Things have been going no where for me for days :(

[Tomelin]

I am also facing the same error while installing openshift 3.9.0.

[Tomelin]

My error is:

ok: [master.rafaeltomelin.co -> master.rafaeltomelin.co] => {
"changed": false,
"examined": 0,
"files": [],
"invocation": {
"module_args": {
"age": null,
"age_stamp": "mtime",
"contains": null,
"depth": null,
"excludes": null,
"file_type": "file",
"follow": false,
"get_checksum": false,
"hidden": false,
"paths": [
"/etc/origin/master/legacy-ca/"
],
"patterns": [
".*-ca.crt"
],
"recurse": false,
"size": null,
"use_regex": true
}
},
"matched": 0,
"msg": "/etc/origin/master/legacy-ca/ was skipped as it does not seem to be a valid directory or it cannot be accessed\n"
}

TASK [openshift_master_certificates : Create the master server certificate] ***************************
task path: /root/openshift-ansible/roles/openshift_master_certificates/tasks/main.yml:39

TASK [openshift_master_certificates : Generate the loopback master client config] *************************
task path: /root/openshift-ansible/roles/openshift_master_certificates/tasks/main.yml:63
ERROR! |failed expects hostvars is a dict

[Tomelin]

I change line 95 in openshift-ansible/roles/openshift_master_certificates/tasks/main.yml and success:

Line 95:
# certificates_to_synchronize is a custom filter in lib_utils
#- "{{ hostvars[inventory_hostname] | certificates_to_synchronize }}"
to

• "{{ hostvars[inventory_hostname]['ansible_facts'] | certificates_to_synchronize }}"

[plaimbock]

For release-3.9 and Ansible 2.6.0.0 this solved it for me: https://github.com/abn/openshift-ansible/commit/b20684b6560ecbe5c68b4412069beb2f24960504

[IRobL]

Thanks @plaimbock! Using the patch branch got passed the error!

But did you happen to hit an error on task

openshift_web_console : Verify that the console is running

wherein the /usr/local/bin/oc get deployment webconsole -o json -n openshift-web-console command fails after 60 retries?

Manually running said command yields the below output (note that the hostname centos7 routes to it's local VBox NIC IP):

$  /usr/local/bin/oc get deployment webconsole -o json -n openshift-web-console
The connection to the server centos7:8443 was refused - did you specify the right host or port

[s-u-b-h-a-k-a-r]

By changing the below lines it solved for me
In /etc/ansible/playbooks/openshift-ansible/roles/openshift_master_certificates/tasks/main.yml around line 63 I simply changed the with_items clause from

• "{{ hostvars[inventory_hostname] | certificates_to_synchronize }}"
  to
• "{{ hostvars[inventory_hostname]['ansible_facts'] | certificates_to_synchronize }}"
  and everything progressed.

[vrutkovs]

So no modifications are required for ansible 2.4, right? This seems to be caused by ansible 2.6 changes

[IRobL]

You ask a very good question, @vrutkovs. The readme indicates that 2.5.x is out of the question, and that anything lower than 2.4.3.0 won't work either. Here's what I have available in Centos7's yum packages:

$ yum --showduplicates list ansible
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.centos.org
 * extras: mirror.centos.org
 * updates: mirror.centos.org
Installed Packages
ansible.noarch                                                                             2.6.0-1.el7                                                                               @epel
Available Packages
ansible.noarch                                                                             2.3.0.0-3.el7                                                                             centos-openshift-origin
ansible.noarch                                                                             2.4.2.0-2.el7                                                                             extras

I have a feeling 2.3.0.0 wouldn't work since it's technically lower than 2.4.3.0. I'm testing with immutable infrastructure and would be happy to test out other version of Ansible. What's an ideal way to get ansible 2.4.3.0 installed? Also, can validations be written into the code base to ensure that a compatible version of Ansible is being used, I have a feeling that would spare many people the frustration I felt last week.

[vrutkovs]

2.4.2 would work, although its known to leak memory. Not sure why would CentOS not update to 2.4.3 - but that's a bug in CentOS anyway.

[lbrigman124]

I'm running from the EPEL package on CentOS. I have version 2.5.5 installed. No building of packages required.
Anyway. I also ran into this error. If the error has been there since March, I'm not sure why I haven't seen it. We are typically doing one or two installs a week.

[lbrigmanArris]

BTW, I tested the patch. It worked for me.

[damiandragowski]

@lbrigman124 Funny thing is that I had installed openshift on centos 7 last thursday and it went fine, when I tried to install on other machine ( same OS ) at saturday, the issue came out :).

[lbrigman124]

@damiandragowski I had the exact same experience. I have been using a fixed commit(c264b57) since about the middle of April without any issues including multiple changes in the version of Ansible. Friday of last week(July 6), I started seeing the above failure. Something else in the environment changed to trigger this failure.

[xrow]

For release-3.9 and Ansible 2.6.0.0 this solved it for me: abn/openshift-ansible@b20684b

Will one merge it?

[j00p34]

The openshift/origin-ansible container v3.9 has the same issue (3.9.33)
tag v3.9.28 does work.

[vrutkovs]

3.9 now supports ansible 2.6, fixed by #9125 (thanks @abn). 3.10 should work fine as well. 3.11 would require 2.6.

Other versions would still need 2.4 as support for both 2.4 and 2.6 would be too complex.

[sailinnthu]

By changing the below lines it solved for me
In /etc/ansible/playbooks/openshift-ansible/roles/openshift_master_certificates/tasks/main.yml

"{{ hostvars[inventory_hostname] | certificates_to_synchronize }}"
to
"{{ hostvars[inventory_hostname]['ansible_facts'] | certificates_to_synchronize }}"
and everything progressed.

[nstrug]

This is https://bugzilla.redhat.com/show_bug.cgi?id=1605046
The fix is on QA and should be released soon.
In the meantime you can apply the patch from https://github.com/openshift/openshift-ansible/pull/9125

[karthik101]

I have something similar issue while redeploying master certificates and ended up with errors below:

ASK [Remove generated certificates] ****************************************
fatal: [master1.com]: FAILED! => {
"failed": true
}

MSG:

Unexpected templating type error occurred on ({{ hostvars[inventory_hostname] | certificates_to_synchronize(include_keys=false, include_ca=false) }}): certificates_to_synchronize() got an unexpected keyword argument 'include_ca'

openshift-ansible version : openshift-ansible-3.4.90-1
oc version: oc v1.4.1
Ansible: ansible-2.3.2.0-2.el7.noarch

10265