I can't deploy simple cluster (1 master, 2 minions) via current kargo master (commit ea874899c78d3f3210d565686a82f23c6db62b98).
OS: Ubuntu 16.04
Inventory:
gklab-97a-182 ip=10.91.97.182
gklab-97a-181 ip=10.91.97.181
gklab-97a-180 ip=10.91.97.180
[kube-master]
gklab-97a-182
[etcd]
gklab-97a-182
gklab-97a-181
gklab-97a-180
[kube-node]
gklab-97a-181
gklab-97a-180
[k8s-cluster:children]
kube-node
kube-master
Errors:
fatal: [gklab-97a-181]: FAILED! => {"failed": true, "msg": "ERROR! 'dict object' has no attribute 'ansible_default_ipv4'"}
fatal: [gklab-97a-180]: FAILED! => {"failed": true, "msg": "ERROR! 'dict object' has no attribute 'ansible_default_ipv4'"}
fatal: [gklab-97a-182]: FAILED! => {"failed": true, "msg": "ERROR! 'dict object' has no attribute 'ansible_default_ipv4'"}
gklab-97a-180 : ok=228 changed=8 unreachable=0 failed=1
gklab-97a-181 : ok=254 changed=19 unreachable=0 failed=1
gklab-97a-182 : ok=248 changed=11 unreachable=0 failed=1
@pskrzyns @bogdando @dkrzyszczyk
(similar error: https://github.com/kubernetes-incubator/kargo/issues/212, but doesnt look related)
can you show at which tasks it failed. It should not look at the default_ipv4 at all with the ip var configured
TASK [kubernetes/preinstall : Hosts | populate inventory into hosts file]
Any progress or workaround here? This still blocks me.
@mzylowski Please include complete ansible-playbook output and the command specified to run ansible.
Also, please try this:
ansible -i inventory/inventory.cfg -m debug -a "var=hostvars[inventory_hostname]" all
I'm interested to see if it gathers ansible_default_ipv4. If not, then there's something wrong with Ansible compatibility on your hosts.
Sorry for the late answer (holidays etc..)
Inventory:
cat inventory/inventory.cfg
gklab-97a-182 ip=10.91.97.182 ansible_ssh_host=10.91.97.182
gklab-97a-181 ip=10.91.97.181 ansible_ssh_host=10.91.97.181
gklab-97a-180 ip=10.91.97.180 ansible_ssh_host=10.91.97.180
[kube-master]
gklab-97a-182
[etcd]
gklab-97a-182
[kube-node]
gklab-97a-181
gklab-97a-180
[k8s-cluster:children]
kube-node
kube-master
Deployment:
ansible-playbook -u root --become-user=root -i /home/mzylowski/Repos/kargo/inventory/inventory.cfg /home/mzylowski/Repos/kargo/cluster.yml
PLAY ***************************************************************************
TASK [bastion-ssh-config : set_fact] *******************************************
ok: [localhost]
TASK [bastion-ssh-config : set_fact] *******************************************
skipping: [localhost]
TASK [bastion-ssh-config : set_fact] *******************************************
skipping: [localhost]
TASK [bastion-ssh-config : create ssh bastion conf] ****************************
ok: [localhost]
PLAY ***************************************************************************
TASK [bootstrap-os : include] **************************************************
skipping: [gklab-97a-181]
skipping: [gklab-97a-180]
skipping: [gklab-97a-182]
TASK [bootstrap-os : include] **************************************************
skipping: [gklab-97a-181]
skipping: [gklab-97a-180]
skipping: [gklab-97a-182]
TASK [bootstrap-os : include] **************************************************
skipping: [gklab-97a-181]
skipping: [gklab-97a-180]
skipping: [gklab-97a-182]
TASK [bootstrap-os : include] **************************************************
included: /home/mzylowski/Repos/kargo/roles/bootstrap-os/tasks/setup-pipelining.yml for gklab-97a-181, gklab-97a-180, gklab-97a-182
TASK [bootstrap-os : Remove require tty] ***************************************
ok: [gklab-97a-182]
ok: [gklab-97a-180]
ok: [gklab-97a-181]
PLAY ***************************************************************************
PLAY ***************************************************************************
TASK [adduser : User | Create User Group] **************************************
ok: [gklab-97a-182]
ok: [gklab-97a-180]
ok: [gklab-97a-181]
TASK [adduser : User | Create User] ********************************************
ok: [gklab-97a-182]
ok: [gklab-97a-180]
ok: [gklab-97a-181]
TASK [kubernetes/preinstall : Force binaries directory for CoreOS] *************
skipping: [gklab-97a-181]
skipping: [gklab-97a-180]
skipping: [gklab-97a-182]
TASK [kubernetes/preinstall : check bin dir exists] ****************************
ok: [gklab-97a-182]
ok: [gklab-97a-181]
ok: [gklab-97a-180]
TASK [kubernetes/preinstall : include] *****************************************
skipping: [gklab-97a-181]
skipping: [gklab-97a-180]
skipping: [gklab-97a-182]
TASK [kubernetes/preinstall : include] *****************************************
included: /home/mzylowski/Repos/kargo/roles/kubernetes/preinstall/tasks/set_facts.yml for gklab-97a-181, gklab-97a-180, gklab-97a-182
TASK [kubernetes/preinstall : set_fact] ****************************************
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : set_fact] ****************************************
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : set_fact] ****************************************
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : set_fact] ****************************************
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : set_fact] ****************************************
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : set_fact] ****************************************
skipping: [gklab-97a-181]
skipping: [gklab-97a-180]
skipping: [gklab-97a-182]
TASK [kubernetes/preinstall : set_fact] ****************************************
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : set_fact] ****************************************
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : set_fact] ****************************************
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : set_fact] ****************************************
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : set_fact] ****************************************
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : set_fact] ****************************************
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : set_fact] ****************************************
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : set_fact] ****************************************
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : set_fact] ****************************************
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : set_fact] ****************************************
ok: [gklab-97a-181]
ok: [gklab-97a-182]
ok: [gklab-97a-180]
TASK [kubernetes/preinstall : set_fact] ****************************************
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : set_fact] ****************************************
ok: [gklab-97a-181]
ok: [gklab-97a-182]
ok: [gklab-97a-180]
TASK [kubernetes/preinstall : set_fact] ****************************************
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : set_fact] ****************************************
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : set_fact] ****************************************
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : include] *****************************************
included: /home/mzylowski/Repos/kargo/roles/kubernetes/preinstall/tasks/set_resolv_facts.yml for gklab-97a-181, gklab-97a-180, gklab-97a-182
TASK [kubernetes/preinstall : check resolvconf] ********************************
ok: [gklab-97a-182]
ok: [gklab-97a-181]
ok: [gklab-97a-180]
TASK [kubernetes/preinstall : set_fact] ****************************************
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : set_fact] ****************************************
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : check kubelet] ***********************************
ok: [gklab-97a-182]
ok: [gklab-97a-181]
ok: [gklab-97a-180]
TASK [kubernetes/preinstall : check if early DNS configuration stage] **********
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : target resolv.conf files] ************************
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : target temporary resolvconf cloud init file (CoreOS)] ***
skipping: [gklab-97a-181]
skipping: [gklab-97a-180]
skipping: [gklab-97a-182]
TASK [kubernetes/preinstall : target dhclient conf/hook files for Red Hat family] ***
skipping: [gklab-97a-181]
skipping: [gklab-97a-180]
skipping: [gklab-97a-182]
TASK [kubernetes/preinstall : target dhclient conf/hook files for Debian family] ***
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : generate search domains to resolvconf] ***********
ok: [gklab-97a-180]
ok: [gklab-97a-182]
ok: [gklab-97a-181]
TASK [kubernetes/preinstall : pick dnsmasq cluster IP or default resolver] *****
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : generate nameservers to resolvconf] **************
ok: [gklab-97a-181]
ok: [gklab-97a-180]
ok: [gklab-97a-182]
TASK [kubernetes/preinstall : gather os specific variables] ********************
ok: [gklab-97a-181] => (item=/home/mzylowski/Repos/kargo/roles/kubernetes/preinstall/templates/../vars/debian.yml)
ok: [gklab-97a-180] => (item=/home/mzylowski/Repos/kargo/roles/kubernetes/preinstall/templates/../vars/debian.yml)
ok: [gklab-97a-182] => (item=/home/mzylowski/Repos/kargo/roles/kubernetes/preinstall/templates/../vars/debian.yml)
TASK [kubernetes/preinstall : Create kubernetes config directory] **************
ok: [gklab-97a-182]
ok: [gklab-97a-180]
ok: [gklab-97a-181]
TASK [kubernetes/preinstall : Create kubernetes script directory] **************
ok: [gklab-97a-182]
ok: [gklab-97a-180]
ok: [gklab-97a-181]
TASK [kubernetes/preinstall : Create kubernetes manifests directory] ***********
ok: [gklab-97a-182]
ok: [gklab-97a-181]
ok: [gklab-97a-180]
TASK [kubernetes/preinstall : Create kubernetes logs directory] ****************
skipping: [gklab-97a-181]
skipping: [gklab-97a-180]
skipping: [gklab-97a-182]
TASK [kubernetes/preinstall : check cloud_provider value] **********************
skipping: [gklab-97a-181]
skipping: [gklab-97a-180]
skipping: [gklab-97a-182]
TASK [kubernetes/preinstall : include] *****************************************
skipping: [gklab-97a-181]
skipping: [gklab-97a-180]
skipping: [gklab-97a-182]
TASK [kubernetes/preinstall : include] *****************************************
skipping: [gklab-97a-181]
skipping: [gklab-97a-180]
skipping: [gklab-97a-182]
TASK [kubernetes/preinstall : Enable ip forwarding] ****************************
ok: [gklab-97a-182]
ok: [gklab-97a-180]
ok: [gklab-97a-181]
TASK [kubernetes/preinstall : Create cni directories] **************************
skipping: [gklab-97a-181] => (item=/etc/cni/net.d)
skipping: [gklab-97a-181] => (item=/opt/cni/bin)
skipping: [gklab-97a-180] => (item=/etc/cni/net.d)
skipping: [gklab-97a-180] => (item=/opt/cni/bin)
skipping: [gklab-97a-182] => (item=/etc/cni/net.d)
skipping: [gklab-97a-182] => (item=/opt/cni/bin)
TASK [kubernetes/preinstall : Update package management cache (YUM)] ***********
skipping: [gklab-97a-181]
skipping: [gklab-97a-180]
skipping: [gklab-97a-182]
TASK [kubernetes/preinstall : Install latest version of python-apt for Debian distribs] ***
ok: [gklab-97a-182]
ok: [gklab-97a-180]
ok: [gklab-97a-181]
TASK [kubernetes/preinstall : Install python-dnf for latest RedHat versions] ***
skipping: [gklab-97a-181]
skipping: [gklab-97a-180]
skipping: [gklab-97a-182]
TASK [kubernetes/preinstall : Install epel-release on RedHat/CentOS] ***********
skipping: [gklab-97a-181]
skipping: [gklab-97a-180]
skipping: [gklab-97a-182]
TASK [kubernetes/preinstall : Install packages requirements] *******************
ok: [gklab-97a-182] => (item=[u'python-apt', u'aufs-tools', u'apt-transport-https', u'software-properties-common', u'python-httplib2', u'openssl', u'curl', u'rsync', u'bash-completion', u'socat'])
ok: [gklab-97a-180] => (item=[u'python-apt', u'aufs-tools', u'apt-transport-https', u'software-properties-common', u'python-httplib2', u'openssl', u'curl', u'rsync', u'bash-completion', u'socat'])
ok: [gklab-97a-181] => (item=[u'python-apt', u'aufs-tools', u'apt-transport-https', u'software-properties-common', u'python-httplib2', u'openssl', u'curl', u'rsync', u'bash-completion', u'socat'])
TASK [kubernetes/preinstall : Disable IPv6 DNS lookup] *************************
skipping: [gklab-97a-181]
skipping: [gklab-97a-180]
skipping: [gklab-97a-182]
TASK [kubernetes/preinstall : Set selinux policy to permissive] ****************
skipping: [gklab-97a-181]
skipping: [gklab-97a-180]
skipping: [gklab-97a-182]
TASK [kubernetes/preinstall : Write openstack cloud-config] ********************
skipping: [gklab-97a-181]
skipping: [gklab-97a-180]
skipping: [gklab-97a-182]
TASK [kubernetes/preinstall : Write azure cloud-config] ************************
skipping: [gklab-97a-181]
skipping: [gklab-97a-180]
skipping: [gklab-97a-182]
TASK [kubernetes/preinstall : include] *****************************************
included: /home/mzylowski/Repos/kargo/roles/kubernetes/preinstall/tasks/etchosts.yml for gklab-97a-181, gklab-97a-180, gklab-97a-182
TASK [kubernetes/preinstall : Hosts | populate inventory into hosts file] ******
fatal: [gklab-97a-181]: FAILED! => {"failed": true, "msg": "ERROR! 'dict object' has no attribute 'ansible_default_ipv4'"}
fatal: [gklab-97a-180]: FAILED! => {"failed": true, "msg": "ERROR! 'dict object' has no attribute 'ansible_default_ipv4'"}
fatal: [gklab-97a-182]: FAILED! => {"failed": true, "msg": "ERROR! 'dict object' has no attribute 'ansible_default_ipv4'"}
PLAY RECAP *********************************************************************
gklab-97a-180 : ok=45 changed=0 unreachable=0 failed=1
gklab-97a-181 : ok=45 changed=0 unreachable=0 failed=1
gklab-97a-182 : ok=45 changed=0 unreachable=0 failed=1
localhost : ok=2 changed=0 unreachable=0 failed=0
Debug:
ansible -i inventory/inventory.cfg -m debug -a "var=hostvars[inventory_hostname]" all | grep -A 5 ipv4
"ansible_all_ipv4_addresses": [
"10.233.117.1",
"10.91.97.182",
"10.233.117.0"
],
"ansible_all_ipv6_addresses": [
--
"ansible_default_ipv4": {
"address": "10.91.97.182",
"alias": "enp1s0f0",
"broadcast": "10.91.97.255",
"gateway": "10.91.96.1",
"interface": "enp1s0f0",
--
"ipv4": {
"address": "10.233.117.1",
"broadcast": "global",
"netmask": "255.255.255.0",
"network": "10.233.117.0"
},
--
"ipv4": {
"address": "10.91.97.182",
"broadcast": "10.91.97.255",
"netmask": "255.255.254.0",
"network": "10.91.96.0"
},
--
"ipv4": {
"address": "10.233.117.0",
"broadcast": "global",
"netmask": "255.255.192.0",
"network": "10.233.64.0"
},
--
"ipv4": {
"address": "127.0.0.1",
"broadcast": "host",
"netmask": "255.0.0.0",
"network": "127.0.0.0"
},
--
"ansible_all_ipv4_addresses": [
"10.233.73.0",
"10.91.97.181",
"10.233.73.1"
],
"ansible_all_ipv6_addresses": [
--
"ansible_default_ipv4": {
"address": "10.91.97.181",
"alias": "enp4s0f1",
"broadcast": "10.91.97.255",
"gateway": "10.91.96.1",
"interface": "enp4s0f1",
--
"ipv4": {
"address": "10.233.73.1",
"broadcast": "global",
"netmask": "255.255.255.0",
"network": "10.233.73.0"
},
--
"ipv4": {
"address": "10.91.97.181",
"broadcast": "10.91.97.255",
"netmask": "255.255.254.0",
"network": "10.91.96.0"
},
--
"ipv4": {
"address": "10.233.73.0",
"broadcast": "global",
"netmask": "255.255.192.0",
"network": "10.233.64.0"
},
--
"ipv4": {
"address": "127.0.0.1",
"broadcast": "host",
"netmask": "255.0.0.0",
"network": "127.0.0.0"
},
--
"ansible_all_ipv4_addresses": [
"10.233.104.0",
"10.91.97.180",
"10.233.104.1"
],
"ansible_all_ipv6_addresses": [
--
"ansible_default_ipv4": {
"address": "10.91.97.180",
"alias": "enp4s0f1",
"broadcast": "10.91.97.255",
"gateway": "10.91.96.1",
"interface": "enp4s0f1",
--
"ipv4": {
"address": "10.233.104.1",
"broadcast": "global",
"netmask": "255.255.255.0",
"network": "10.233.104.0"
},
--
"ipv4": {
"address": "10.91.97.180",
"broadcast": "10.91.97.255",
"netmask": "255.255.254.0",
"network": "10.91.96.0"
},
--
"ipv4": {
"address": "10.233.104.0",
"broadcast": "global",
"netmask": "255.255.192.0",
"network": "10.233.64.0"
},
--
"ipv4": {
"address": "127.0.0.1",
"broadcast": "host",
"netmask": "255.0.0.0",
"network": "127.0.0.0"
},
So it looks like ansible_default_ipv4 should be set...
@mzylowski please try with the patch
problem exists on ansible 2.0.0.2. with fresh ansible 2.2.0.0 that works fine.
You need Ansible 2.1.0 or newer to use Kargo
Provisioning node in this network was miss-configured and indeed the problem was in ansible version. After upgrading to 2.2.0 I finished my deployment. Thanks @mattymo and @theundefined, I forgot to check this simple thing in new environment, sorry..
@bogdando Thanks for the path. Before ansible upgrading I'v checked it and my deployment failed few task later on something similar. It helped but the problem was in my ansible version.
@mzylowski : We are having same issue with ansible 2.3.1.0. Do you know what exactly was miss-configured during node provisioning?
I am facing the same error with a fresh install of Ansible 2.3.
But the variable is in fact set, I can confirm with the following command:
$ ansible node1 -i inventory/inventory -m debug -a "var=ansible_default_ipv4" -b
node1 | SUCCESS => {
"ansible_default_ipv4": {
"address": "**REDACTED**",
"alias": "**REDACTED**",
"broadcast": "**REDACTED**",
"gateway": "**REDACTED**",
"interface": "**REDACTED**",
"macaddress": "00:50:56:b8:5c:f6",
"mtu": 1500,
"netmask": "255.255.255.0",
"network": "**REDACTED**",
"type": "ether"
}
}
Be sure to check all nodes can access internet as well so they can install packages, even though you can ssh to it.
Most helpful comment
@mzylowski : We are having same issue with ansible 2.3.1.0. Do you know what exactly was miss-configured during node provisioning?