Kubespray: Failed Kargo deploy at 'etcd : Check_certs | Set 'sync_certs' to true'

[nikos9742]

Hi :)
This a BUG REPORT

I'm trying to use kargo to deploy kubernetes in a azure stack environnement (basically azure) on VM running ubuntu 16.04 LTS.

I tried kargo prepare --nodes node1[ansible_ssh_host=10.3.0.4] node2[ansible_ssh_host=10.3.0.5] node3[ansible_ssh_host=10.3.0.6] [--etcds N+] [--masters N+]
then
kargo prepare --nodes node1[ansible_ssh_host=10.3.0.4] node2[ansible_ssh_host=10.3.0.5] node3[ansible_ssh_host=10.3.0.6]
but the scripts seems to fail when the checks happens for the etcs certs.

Environment:

• Cloud provider or hardware configuration:
  Microsoft Azure Stack
• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):

printf "$(uname -srm)\n$(cat /etc/os-release)\n"
Linux 4.4.0-70-generic x86_64
NAME="Ubuntu"
VERSION="16.04.2 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.2 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

• Version of Ansible (ansible --version):
  ansible --version

Kargo version (commit) (git rev-parse --short HEAD):
git rev-parse --short HEAD c2c334d

Network plugin used:
default : calico

Copy of your inventory file:

`cat .kargo/inventory/inventory.cfg
[kube-master]
node1
node2

[all]
node1           ansible_ssh_host=10.3.0.4
node2           ansible_ssh_host=10.3.0.5
node3           ansible_ssh_host=10.3.0.6

[k8s-cluster:children]
kube-node
kube-master

[kube-node]
node1
node2
node3

[etcd]
node1
node2
node3`

Command used to invoke ansible:
kargo deploy

Output of ansible run:

https://gist.github.com/nikos9742/313fd6eccff7016de8c721cd18a818f7

[cloudrkt]

It seems that the {{ certs.sync }} variable is not defined. The same code is found here:

https://github.com/kubernetes-incubator/kargo/blob/9624662bf6e7fd9dd3090cd87c47589fc482fc87/roles/kubernetes/secrets/tasks/check-certs.yml#L65

[adidenko]

Confirm the bug, see the same error on my labs

[jduhamel]

I'd like to reopen this. I get this failure when running it after a sucessful run.

fatal: [core-01]: FAILED! => {"failed": true, "msg": "The conditional check '{%- set certs = {'sync': False} -%} {% if gen_node_certs[inventory_hostname] or \n (not kubecert_node.results[0].stat.exists|default(False)) or\n (not kubecert_node.results[1].stat.exists|default(False)) or\n (kubecert_node.results[1].stat.checksum|default('') != kubecert_master.files|selectattr(\"path\", \"equalto\", kubecert_node.results[1].stat.path)|map(attribute=\"checksum\")|first|default('')) -%}\n {%- set _ = certs.update({'sync': True}) -%}\n{% endif %} {{ certs.sync }}' failed. The error was: Invalid conditional detected: unexpected indent (<unknown>, line 1)\n\nThe error appears to have been in '/root/kargo/roles/kubernetes/secrets/tasks/check-certs.yml': line 54, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: \"Check_certs | Set 'sync_certs' to true\"\n ^ here\n"} fatal: [core-02]: FAILED! => {"failed": true, "msg": "The conditional check '{%- set certs = {'sync': False} -%} {% if gen_node_certs[inventory_hostname] or \n (not kubecert_node.results[0].stat.exists|default(False)) or\n (not kubecert_node.results[1].stat.exists|default(False)) or\n (kubecert_node.results[1].stat.checksum|default('') != kubecert_master.files|selectattr(\"path\", \"equalto\", kubecert_node.results[1].stat.path)|map(attribute=\"checksum\")|first|default('')) -%}\n {%- set _ = certs.update({'sync': True}) -%}\n{% endif %} {{ certs.sync }}' failed. The error was: Invalid conditional detected: unexpected indent (<unknown>, line 1)\n\nThe error appears to have been in '/root/kargo/roles/kubernetes/secrets/tasks/check-certs.yml': line 54, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: \"Check_certs | Set 'sync_certs' to true\"\n ^ here\n"}

[mattymo]

@jduhamel did you check out the latest master? https://github.com/kubernetes-incubator/kargo/pull/1194 should have fixed this

[jduhamel]

I did. the first time I ran ansible-playbook ran through this task fine. Then I got stuck on the dns-masq bug so I re-ran ansible-playbook and got the error.

[adidenko]

@mattymo actually #1194 fixed only roles/etcd/tasks/check_certs.yml, this time it failed on roles/kubernetes/secrets/tasks/check-certs.yml. But I have not seen such error before, so that's why I fixed only etcd task.

[jduhamel]

That fixed it for me. on to the dnsmasq issue. Thanks @mattymo @adidenko

[karthik-ir]

I'm using the latest master and applied @mattymo changes of commit 5a57071 in my local. But then I get the same error
`fatal: [master-0]: FAILED! => {"failed": true, "msg": "The conditional check '{%- set certs = {'sync': False} -%}n{% if gen_node_certs[inventory_hostname] orn (not etcdcert_node.results[0].stat.exists|default(False)) orn (not etcdcert_node.results[1].stat.exists|default(False)) orn (etcdcert_node.results[1].stat.checksum|default('') != etcdcert_master.files|selectattr(\"path\", \"equalto\", etcdcert_node.results[1].stat.path)|map(attribute=\"checksum\")|first|default('')) -%}n {%- set _ = certs.update({'sync': True}) -%}n{% endif %}n{{ certs.sync }}' failed. The error was: no test named 'equalto'nnThe error appears to have been in '/home/vymo-sandbox-user/kargo/roles/etcd/tasks/check_certs.yml': line 57, column 3, but maynbe elsewhere in the file depending on the exact syntax problem.nnThe offending line appears to be:nnn- name: \"Check_certs | Set 'sync_certs' to true\"n ^ heren"}

fatal: [master-1]: FAILED! => {"failed": true, "msg": "The conditional check '{%- set certs = {'sync': False} -%}n{% if gen_node_certs[inventory_hostname] orn (not etcdcert_node.results[0].stat.exists|default(False)) orn (not etcdcert_node.results[1].stat.exists|default(False)) orn (etcdcert_node.results[1].stat.checksum|default('') != etcdcert_master.files|selectattr(\"path\", \"equalto\", etcdcert_node.results[1].stat.path)|map(attribute=\"checksum\")|first|default('')) -%}n {%- set _ = certs.update({'sync': True}) -%}n{% endif %}n{{ certs.sync }}' failed. The error was: no test named 'equalto'nnThe error appears to have been in '/home/vymo-sandbox-user/kargo/roles/etcd/tasks/check_certs.yml': line 57, column 3, but maynbe elsewhere in the file depending on the exact syntax problem.nnThe offending line appears to be:nnn- name: \"Check_certs | Set 'sync_certs' to true\"n ^ heren"}

fatal: [master-2]: FAILED! => {"failed": true, "msg": "The conditional check '{%- set certs = {'sync': False} -%}n{% if gen_node_certs[inventory_hostname] orn (not etcdcert_node.results[0].stat.exists|default(False)) orn (not etcdcert_node.results[1].stat.exists|default(False)) orn (etcdcert_node.results[1].stat.checksum|default('') != etcdcert_master.files|selectattr(\"path\", \"equalto\", etcdcert_node.results[1].stat.path)|map(attribute=\"checksum\")|first|default('')) -%}n {%- set _ = certs.update({'sync': True}) -%}n{% endif %}n{{ certs.sync }}' failed. The error was: no test named 'equalto'nnThe error appears to have been in '/home/vymo-sandbox-user/kargo/roles/etcd/tasks/check_certs.yml': line 57, column 3, but maynbe elsewhere in the file depending on the exact syntax problem.nnThe offending line appears to be:nnn- name: \"Check_certs | Set 'sync_certs' to true\"n ^ heren"}`

[karthik-ir]

Upgrading the Jinja2 version solved this error.

[frederic-loui]

Hi, I ran into the sam problem and run ubuntu 16.04 LTS. Which jinja version did you use ?
Thanks in advane for your answer !

[ilterpehlivan]

Hi,

I am also getting this error in RHEL7:

FAILED! => {"failed": true, "msg": "The conditional check '{%- set certs = {'sync': False} -%}n{% if gen_node_certs[inventory_hostname] orn (not etcdcert_node.results[0].stat.exists|default(False)) orn (not etcdcert_node.results[1].stat.exists|default(False)) orn (etcdcert_node.results[1].stat.checksum|default('') != etcdcert_master.files|selectattr(\"path\", \"equalto\", etcdcert_node.results[1].stat.path)|map(attribute=\"checksum\")|first|default('')) -%}n {%- set _ = certs.update({'sync': True}) -%}n{% endif %}n{{ certs.sync }}' failed. The error was: no test named 'equalto'nnThe error appears to have been in '/root/kargo/roles/etcd/tasks/check_certs.yml': line 57, column 3, but maynbe elsewhere in the file depending on the exact syntax problem.nnThe offending line appears to be:nnn- name: \"Check_certs | Set 'sync_certs' to true\"n ^ heren"}

I upgraded to Jinja2 version 2.9, but still this error showing up! Is this related to my inventory file since i am using IPs instead of hostnames ?

Thnaks

[mattymo]

Try jinja2 version 2.8

[ilterpehlivan]

ok that was the trick, thanks mattymo :)
Here i am putting the steps for installing Jinja2 2.8 version for RHEL
1- Install Pip
yum -y install python-pip
2- Install 2.2.8 version
pip install https://pypi.python.org/packages/2.7/J/Jinja2/Jinja2-2.8-py2.py3-none-any.whl

[wtli]

try: yum install -y python-devel
i solve this problem

[vhosakot]

pip install --upgrade Jinja2 resolved this error for me.

[vlokesh526]

Jinja2 upgrade worked for me to cover the cert errors