Kubespray: ERROR! 'dict object' has no attribute 'pkg_mgr'

[cchongXD]

I tried to run ansible-playbook -b -i inventory/mycluster/inventory.ini cluster.yml and faced this error:

ERROR! 'dict object' has no attribute 'pkg_mgr'

[floryut]

Can you please provide a bit more context ?

[levin-kelevra]

when I deploy a new cluster I see the same error.
Target OS: Ubuntu 19.10 x64
Ansible OS: Ubuntu 20.04 x64
KubeSpray: commit aba63f0
Kubernetes: 1.19.2
Containers: docker
Azure: not used

TASK [kubernetes/preinstall : Check if we are running inside a Azure VM] *****************************************************************************************
ok: [k8s-master1]
ok: [k8s-master3]
ok: [k8s-wnode1]
ok: [k8s-wnode2]
ok: [k8s-master2]
ok: [k8s-wnode4]
ok: [k8s-wnode3]
ok: [k8s-wnode5]
Tuesday 29 September 2020  14:54:56 +0300 (0:00:00.522)       0:01:12.306 ***** 
Tuesday 29 September 2020  14:54:56 +0300 (0:00:00.076)       0:01:12.382 ***** 
Tuesday 29 September 2020  14:54:56 +0300 (0:00:00.065)       0:01:12.447 ***** 
Tuesday 29 September 2020  14:54:56 +0300 (0:00:00.066)       0:01:12.514 ***** 
Tuesday 29 September 2020  14:54:56 +0300 (0:00:00.062)       0:01:12.577 ***** 
Tuesday 29 September 2020  14:54:56 +0300 (0:00:00.059)       0:01:12.636 ***** 
Tuesday 29 September 2020  14:54:56 +0300 (0:00:00.058)       0:01:12.695 ***** 
Tuesday 29 September 2020  14:54:56 +0300 (0:00:00.060)       0:01:12.755 ***** 
Tuesday 29 September 2020  14:54:56 +0300 (0:00:00.069)       0:01:12.825 ***** 
Tuesday 29 September 2020  14:54:57 +0300 (0:00:00.070)       0:01:12.896 ***** 
Tuesday 29 September 2020  14:54:57 +0300 (0:00:00.064)       0:01:12.960 ***** 
Tuesday 29 September 2020  14:54:57 +0300 (0:00:00.062)       0:01:13.023 ***** 
Tuesday 29 September 2020  14:54:57 +0300 (0:00:00.077)       0:01:13.100 ***** 
Tuesday 29 September 2020  14:54:57 +0300 (0:00:00.063)       0:01:13.164 ***** 
Tuesday 29 September 2020  14:54:57 +0300 (0:00:00.059)       0:01:13.224 ***** 
Tuesday 29 September 2020  14:54:57 +0300 (0:00:00.061)       0:01:13.285 ***** 
Tuesday 29 September 2020  14:54:57 +0300 (0:00:00.061)       0:01:13.347 ***** 
Tuesday 29 September 2020  14:54:57 +0300 (0:00:00.059)       0:01:13.406 ***** 
Tuesday 29 September 2020  14:54:57 +0300 (0:00:00.076)       0:01:13.483 ***** 
Tuesday 29 September 2020  14:54:57 +0300 (0:00:00.064)       0:01:13.548 ***** 
Tuesday 29 September 2020  14:54:57 +0300 (0:00:00.064)       0:01:13.613 ***** 
Tuesday 29 September 2020  14:54:57 +0300 (0:00:00.063)       0:01:13.676 ***** 
Tuesday 29 September 2020  14:54:57 +0300 (0:00:00.088)       0:01:13.765 ***** 
Tuesday 29 September 2020  14:54:57 +0300 (0:00:00.060)       0:01:13.825 ***** 
Tuesday 29 September 2020  14:54:58 +0300 (0:00:00.063)       0:01:13.889 ***** 
Tuesday 29 September 2020  14:54:58 +0300 (0:00:00.092)       0:01:13.982 ***** 
Tuesday 29 September 2020  14:54:58 +0300 (0:00:00.060)       0:01:14.042 ***** 
Tuesday 29 September 2020  14:54:58 +0300 (0:00:00.059)       0:01:14.102 ***** 
Tuesday 29 September 2020  14:54:58 +0300 (0:00:00.059)       0:01:14.161 ***** 
Tuesday 29 September 2020  14:54:58 +0300 (0:00:00.060)       0:01:14.221 ***** 
Tuesday 29 September 2020  14:54:58 +0300 (0:00:00.060)       0:01:14.282 ***** 
Tuesday 29 September 2020  14:54:58 +0300 (0:00:00.060)       0:01:14.343 ***** 
Tuesday 29 September 2020  14:54:58 +0300 (0:00:00.082)       0:01:14.426 ***** 
Tuesday 29 September 2020  14:54:58 +0300 (0:00:00.063)       0:01:14.490 ***** 
Tuesday 29 September 2020  14:54:58 +0300 (0:00:00.079)       0:01:14.569 ***** 
Tuesday 29 September 2020  14:54:58 +0300 (0:00:00.061)       0:01:14.631 ***** 
Tuesday 29 September 2020  14:54:58 +0300 (0:00:00.060)       0:01:14.691 ***** 
Tuesday 29 September 2020  14:54:58 +0300 (0:00:00.061)       0:01:14.752 ***** 
Tuesday 29 September 2020  14:54:59 +0300 (0:00:00.087)       0:01:14.840 ***** 
Tuesday 29 September 2020  14:54:59 +0300 (0:00:00.064)       0:01:14.905 ***** 
Tuesday 29 September 2020  14:54:59 +0300 (0:00:00.063)       0:01:14.968 ***** 
Tuesday 29 September 2020  14:54:59 +0300 (0:00:00.064)       0:01:15.032 ***** 
Tuesday 29 September 2020  14:54:59 +0300 (0:00:00.083)       0:01:15.116 ***** 
Tuesday 29 September 2020  14:54:59 +0300 (0:00:00.061)       0:01:15.177 ***** 
Tuesday 29 September 2020  14:54:59 +0300 (0:00:00.059)       0:01:15.237 ***** 
Tuesday 29 September 2020  14:54:59 +0300 (0:00:00.062)       0:01:15.300 ***** 
Tuesday 29 September 2020  14:54:59 +0300 (0:00:00.065)       0:01:15.365 ***** 
Tuesday 29 September 2020  14:54:59 +0300 (0:00:00.061)       0:01:15.426 ***** 
Tuesday 29 September 2020  14:54:59 +0300 (0:00:00.064)       0:01:15.491 ***** 
Tuesday 29 September 2020  14:54:59 +0300 (0:00:00.069)       0:01:15.561 ***** 
ERROR! 'dict object' has no attribute 'pkg_mgr'

[floryut]

Can you run your ansible log with a bit more verbosity ? hard to tell which play/task failed :)

[levin-kelevra]

Can you run your ansible log with a bit more verbosity ? hard to tell which play/task failed :)

sure
ansible.log

[tumbl3w33d]

Happens to me at this task:

TASK [container-engine/containerd : Set containerd pin priority to apt_preferences on Debian family] ***
Tuesday 29 September 2020  17:56:49 +0000 (0:00:00.300)       0:06:20.262 ***** 
skipping: [k8s-prod-master-01]
skipping: [k8s-prod-master-02]
skipping: [k8s-prod-master-03]
skipping: [k8s-prod-worker-01]
skipping: [k8s-prod-worker-02]
skipping: [k8s-prod-worker-03]
skipping: [k8s-prod-worker-04]
skipping: [k8s-prod-worker-05]
skipping: [k8s-prod-worker-06]
skipping: [k8s-prod-worker-07]
ERROR! 'dict object' has no attribute 'pkg_mgr'

running with

ansible 2.10.1
  config file = /my-repo/ansible.cfg
  configured module search path = ['/my-repo/kubespray/library']
  ansible python module location = /usr/lib/python3.8/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.8.5 (default, Jul 20 2020, 23:11:29) [GCC 9.3.0]

Edit: this was while running in a dedicated alpine-based container that loaded the latest and greatest pip module versions of a requirements.txt. After this failure I retried the same run outside the container on a debian-based virtualenv (which comes with older module versions) and it completed successfully. Makes me wonder if this is about a certain depedency constellation that's incompatible.
broken.txt
working.txt

But simply downgrading ansible inside the alpine container didn't do the trick before. So it's probably about something else.

[jaurestchoffo]

Meme Erreur avec CentOS7 et Ansible 2.10.1

[levin-kelevra]

The problem is Ansible 2.10.1
After downgrade to 2.9.13 all works fine.
When will support Ansible 2.10?

[floryut]

The problem is Ansible 2.10.1
After downgrade to 2.9.13 all works fine.
When will support Ansible 2.10?

Wasn't aware of that.
@cchongXD could you change the issue title with something including the Ansible 2.10 issue?

[piotrkochan]

@cchongXD @levin-kelevra I have same situation, it looks like pip installs good version from requirements (2.9) but mine (and yours) environment still uses global-installed ansible so just make sure you use correct one

╰─$ python-env/bin/ansible-playbook --version
ansible-playbook 2.9.6

╰─$ ansible-playbook --version                                                                      
ansible-playbook 2.10.1

[mischenkovn]

Same behavior.
It works with ansible 2.9.14, and not ansible 2.10.2.

[ledroide]

Same error with Ansible 2.10.1
After downgrading to Ansible 2.9.14, the problem is solved.
Note : I had to uninstall ansible and ansible-base, and reinstall ansible, because of python errors.

pip3 uninstall ansible ansible-base
pip3 install ansible==2.9.14

[Learloj]

I also ran into this issue. The downgrade indeed worked. I could also work around it by updating kubespray/roles/container-engine/containerd/defaults/main.yml adding 'pkg_mgr' to 'containerd_package_info':

  containerd_package_info:
+   pkg_mgr:
    pkgs:

However, the playbook then breaks at a later stage as well.

That said, the requirements.txt clearly state it needs ansible version 2.7.16.

[piotrkochan]

That said, the requirements.txt clearly state it needs ansible version 2.7.16.

run ansible --version and verify you're using this version

[hakoerber]

Hey, I have the same problem and might be able to provide a bit more context / debugging:

For me, it happens in the containerd role (roles/container-engine/containerd). The following tasks seems to be the root of the problem: Link

It uses a common pattern: Load variables from a file specific to a certain OS. In my setup, the containerd role is never used (I use container_manager: docker). Therefore, all tasks in the containerd role are skipped, including the tasks I mentioned.

This leaves containerd_package_info unitialized (using the bare default). This should not be a problem, because all tasks requiring containerd_package_info are of course skipped. But it seems that since Ansible 2.10, Jinja expressions are also evaluated in skipped tasks. So this task breaks.

Because the pattern of reading OS-specific variables is used more or less everywhere, this will break in a lot of places with Ansible 2.10.

As far as I can see, Ansible 2.10 is not released yet and still in development: Link. I did not find anything in the changelog at first glance that would point to this change being intentional. I'd classify this as an Ansible bug. If you agree, I'd open an issue with Ansible (if you did not already).

[piotrkochan]

@hakoerber just read what I've posted above, maybe it will fix your problem

[hma23]

same error on ansible 2.10.2 version
TASK [kubernetes/preinstall : Check if we are running inside a Azure VM] *************************************
ok: [node1]
ok: [node2]
ok: [node3]
ok: [node4]
ok: [node5]
ok: [node6]
Monday 02 November 2020 17:05:01 +0000 (0:00:00.454) 0:02:31.751 *
Monday 02 November 2020 17:05:02 +0000 (0:00:00.113) 0:02:31.865 *
Monday 02 November 2020 17:05:02 +0000 (0:00:00.116) 0:02:31.981 *
Monday 02 November 2020 17:05:02 +0000 (0:00:00.116) 0:02:32.097 *
Monday 02 November 2020 17:05:02 +0000 (0:00:00.112) 0:02:32.210 *
Monday 02 November 2020 17:05:02 +0000 (0:00:00.120) 0:02:32.330 *
Monday 02 November 2020 17:05:02 +0000 (0:00:00.116) 0:02:32.447 *
Monday 02 November 2020 17:05:02 +0000 (0:00:00.113) 0:02:32.561 *
Monday 02 November 2020 17:05:02 +0000 (0:00:00.127) 0:02:32.689 *
Monday 02 November 2020 17:05:03 +0000 (0:00:00.128) 0:02:32.817 *
Monday 02 November 2020 17:05:03 +0000 (0:00:00.117) 0:02:32.935 *
Monday 02 November 2020 17:05:03 +0000 (0:00:00.122) 0:02:33.058 *
Monday 02 November 2020 17:05:03 +0000 (0:00:00.139) 0:02:33.198 *
Monday 02 November 2020 17:05:03 +0000 (0:00:00.119) 0:02:33.317 *
Monday 02 November 2020 17:05:03 +0000 (0:00:00.124) 0:02:33.441 *
Monday 02 November 2020 17:05:03 +0000 (0:00:00.125) 0:02:33.567 *
Monday 02 November 2020 17:05:03 +0000 (0:00:00.113) 0:02:33.681 *
Monday 02 November 2020 17:05:04 +0000 (0:00:00.120) 0:02:33.801 *
Monday 02 November 2020 17:05:04 +0000 (0:00:00.112) 0:02:33.914 *
Monday 02 November 2020 17:05:04 +0000 (0:00:00.150) 0:02:34.064 *
Monday 02 November 2020 17:05:04 +0000 (0:00:00.121) 0:02:34.186 *
Monday 02 November 2020 17:05:04 +0000 (0:00:00.121) 0:02:34.308 *
Monday 02 November 2020 17:05:04 +0000 (0:00:00.128) 0:02:34.437 *
Monday 02 November 2020 17:05:04 +0000 (0:00:00.172) 0:02:34.610 *
Monday 02 November 2020 17:05:04 +0000 (0:00:00.123) 0:02:34.733 *
Monday 02 November 2020 17:05:05 +0000 (0:00:00.117) 0:02:34.850 *
Monday 02 November 2020 17:05:05 +0000 (0:00:00.118) 0:02:34.969 *
Monday 02 November 2020 17:05:05 +0000 (0:00:00.155) 0:02:35.125 *
Monday 02 November 2020 17:05:05 +0000 (0:00:00.120) 0:02:35.245 *
Monday 02 November 2020 17:05:05 +0000 (0:00:00.116) 0:02:35.362 *
Monday 02 November 2020 17:05:05 +0000 (0:00:00.116) 0:02:35.479 *
Monday 02 November 2020 17:05:05 +0000 (0:00:00.119) 0:02:35.598 *
Monday 02 November 2020 17:05:05 +0000 (0:00:00.117) 0:02:35.716 *
Monday 02 November 2020 17:05:06 +0000 (0:00:00.139) 0:02:35.855 *
Monday 02 November 2020 17:05:06 +0000 (0:00:00.111) 0:02:35.967 *
Monday 02 November 2020 17:05:06 +0000 (0:00:00.109) 0:02:36.077 *
Monday 02 November 2020 17:05:06 +0000 (0:00:00.122) 0:02:36.199 *
Monday 02 November 2020 17:05:06 +0000 (0:00:00.120) 0:02:36.319 *
Monday 02 November 2020 17:05:06 +0000 (0:00:00.116) 0:02:36.436 *
Monday 02 November 2020 17:05:06 +0000 (0:00:00.123) 0:02:36.560 *
Monday 02 November 2020 17:05:06 +0000 (0:00:00.117) 0:02:36.678 *
Monday 02 November 2020 17:05:07 +0000 (0:00:00.150) 0:02:36.828 *
Monday 02 November 2020 17:05:07 +0000 (0:00:00.119) 0:02:36.947 *
Monday 02 November 2020 17:05:07 +0000 (0:00:00.129) 0:02:37.077 *
Monday 02 November 2020 17:05:07 +0000 (0:00:00.112) 0:02:37.189 *
Monday 02 November 2020 17:05:07 +0000 (0:00:00.149) 0:02:37.339 *
Monday 02 November 2020 17:05:07 +0000 (0:00:00.120) 0:02:37.460 *
Monday 02 November 2020 17:05:07 +0000 (0:00:00.121) 0:02:37.581 *
Monday 02 November 2020 17:05:07 +0000 (0:00:00.118) 0:02:37.699 *
Monday 02 November 2020 17:05:08 +0000 (0:00:00.118) 0:02:37.818 *
Monday 02 November 2020 17:05:08 +0000 (0:00:00.139) 0:02:37.958 **
ERROR! 'dict object' has no attribute 'pkg_mgr'

[hakoerber]

Hey @piotrkochan,

I forgot to mention that using ansible <2.10 fixes the problem for me. My comment was rather meant as a more technical analysis of the problem and to get a solution.