Salt: Unexpected error while connecting to salt (master) KeyError: 'os_family'

Created on 30 Apr 2020  路  8Comments  路  Source: saltstack/salt

Description
Hi Guys, I understand that the official repos don't yet support Ubuntu 20.04 with Py3 implementation, so please feeel free to close if this is expected behaviour. However, when trying to connect a Ubuntu 20.04 to a Ubuntu 18.04 master (both using Salt version 3000.2) I get an "Unexpected error while connecting to salt (master) KeyError: 'os_family'" on the minion.

Setup

  • Existing salt-master (Ubuntu 18.04, 3000.2).
  • New salt-minion (Ubuntu 20.04, 3000.2).
  • Minion's key is seen by and accepted by master.

Steps to Reproduce the behavior
Attempt to run highstate on the minion after accepting key on master; or simply run systemctl restart salt-minion.service; the output of /var/log/salt/minion will be flooded with these messages (several per second):

2020-04-30 15:49:47,480 [salt.minion      :1058][CRITICAL][17150] Unexpected error while connecting to salt
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 1037, in _connect_minion
    yield minion.connect_master(failed=failed)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 239, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 1236, in connect_master
    yield self._post_master_init(master)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 239, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1070, in run
    yielded = self.gen.send(value)
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 1279, in _post_master_init
    self.schedule = salt.utils.schedule.Schedule(
  File "/usr/lib/python3/dist-packages/salt/utils/schedule.py", line 99, in __new__
    instance.__singleton_init__(opts, functions,
  File "/usr/lib/python3/dist-packages/salt/utils/schedule.py", line 152, in __singleton_init__
    self.time_offset = self.functions.get('timezone.get_offset', lambda: '0000')()
  File "/usr/lib/python3/dist-packages/salt/modules/timezone.py", line 234, in get_offset
    if 'AIX' not in __grains__['os_family']:
  File "/usr/lib/python3/dist-packages/salt/utils/context.py", line 236, in __getitem__
    return self._dict()[key]
KeyError: 'os_family'

Expected behavior
Connection to the salt-master, allowing highstate to be run.

Versions Report

salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) 

salt-minion, running on Ubuntu 20.04 server via the "deb http://repo.saltstack.com/py3/ubuntu/18.04/amd64/latest bionic main" repo.

Salt Version:
           Salt: 3000.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.7.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.10.1
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: 1.4.4
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.8.2 (default, Mar 13 2020, 10:14:16)
   python-gnupg: 0.4.5
         PyYAML: 5.3.1
          PyZMQ: 18.1.1
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.2

System Versions:
           dist: ubuntu 20.04 focal
         locale: utf-8
        machine: x86_64
        release: 5.4.0-28-generic
         system: Linux
        version: Ubuntu 20.04 focal


salt-master, running on Ubuntu 18.04 server:

Salt Version:
           Salt: 3000.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
         Jinja2: 2.10
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.9 (default, Apr 18 2020, 01:56:04)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-96-generic
         system: Linux
        version: Ubuntu 18.04 bionic

Bug Core Duplicate Grains Z Release Sodium severity-high
Source
picture oekelly
👍1

Most helpful comment

For now, I'm hacking around it with this patch on the minion (tailor to your needs):

--- /usr/lib/python3/dist-packages/salt/utils/context.py.org       2020-04-20 23:14:41.000000000 +0200
+++ /usr/lib/python3/dist-packages/salt/utils/context.py        2020-05-07 15:25:17.984684060 +0200
@@ -217,6 +217,16 @@
             )
         super(NamespacedDictWrapper, self).__init__(self._dict())

+        # Getting it to work on Ubuntu 20.04
+        self._dict()['os'] = "Ubuntu"
+        self._dict()['os_family'] = "Debian"
+        self._dict()['oscodename'] = "focal"
+        self._dict()['osfullname'] = "Ubuntu"
+        self._dict()['osmajorrelease'] = 20
+        self._dict()['lsb_distrib_release'] = "20.04"
+        self._dict()['lsb_distrib_codename'] = "focal"
+        self._dict()['lsb_distrib_description'] = "Ubuntu 20.04 LTS"
+        self._dict()['lsb_distrib_id'] = "Ubuntu"
+        self._dict()['fqdn'] = "servername.domain.com"
+        self._dict()['host'] = "servername"
+        self._dict()['virtual'] = "xen"
+        self._dict()['nodename'] = "servername"
+
     def _dict(self):
         r = self.__dict
         for k in self.pre_keys:
picture wiebeytec on 7 May 2020
👍4

All 8 comments

This is happening to me too on Fedora 32. Just upgraded my OS to that version, which deprecated Python 2.x versions. All Fedora 32 minions are showing the following: 

Apr 30 09:38:54 f32 salt-minion[22268]:     self.schedule = salt.utils.schedule.Schedule(
Apr 30 09:38:54 f32 salt-minion[22268]:   File "/usr/lib/python3.8/site-packages/salt/utils/schedule.py", line 99, in >
Apr 30 09:38:54 f32 salt-minion[22268]:     instance.__singleton_init__(opts, functions,
Apr 30 09:38:54 f32 salt-minion[22268]:   File "/usr/lib/python3.8/site-packages/salt/utils/schedule.py", line 152, in>
Apr 30 09:38:54 f32 salt-minion[22268]:     self.time_offset = self.functions.get('timezone.get_offset', lambda: '0000>
Apr 30 09:38:54 f32 salt-minion[22268]:   File "/usr/lib/python3.8/site-packages/salt/modules/timezone.py", line 234, >
Apr 30 09:38:54 f32 salt-minion[22268]:     if 'AIX' not in __grains__['os_family']:
Apr 30 09:38:54 f32 salt-minion[22268]:   File "/usr/lib/python3.8/site-packages/salt/utils/context.py", line 236, in >
Apr 30 09:38:54 f32 salt-minion[22268]:     return self._dict()[key]
Apr 30 09:38:54 f32 salt-minion[22268]: KeyError: 'os_family'
RooneyMcNibNug picture RooneyMcNibNug on 30 Apr 2020

Salt will not officially support 3.8 until Sodium. That's targetted for June.

twangboy picture twangboy on 1 May 2020

any update on this? 20.04 is out and as it's an LTS version lots of people (including myself) would like to move...

ThisGuyCodes picture ThisGuyCodes on 4 May 2020
👍4

For now, I'm hacking around it with this patch on the minion (tailor to your needs):

--- /usr/lib/python3/dist-packages/salt/utils/context.py.org       2020-04-20 23:14:41.000000000 +0200
+++ /usr/lib/python3/dist-packages/salt/utils/context.py        2020-05-07 15:25:17.984684060 +0200
@@ -217,6 +217,16 @@
             )
         super(NamespacedDictWrapper, self).__init__(self._dict())

+        # Getting it to work on Ubuntu 20.04
+        self._dict()['os'] = "Ubuntu"
+        self._dict()['os_family'] = "Debian"
+        self._dict()['oscodename'] = "focal"
+        self._dict()['osfullname'] = "Ubuntu"
+        self._dict()['osmajorrelease'] = 20
+        self._dict()['lsb_distrib_release'] = "20.04"
+        self._dict()['lsb_distrib_codename'] = "focal"
+        self._dict()['lsb_distrib_description'] = "Ubuntu 20.04 LTS"
+        self._dict()['lsb_distrib_id'] = "Ubuntu"
+        self._dict()['fqdn'] = "servername.domain.com"
+        self._dict()['host'] = "servername"
+        self._dict()['virtual'] = "xen"
+        self._dict()['nodename'] = "servername"
+
     def _dict(self):
         r = self.__dict
         for k in self.pre_keys:
wiebeytec picture wiebeytec on 7 May 2020
👍4

Work seems to be progressing here: https://github.com/saltstack/salt/issues/55310

twangboy picture twangboy on 7 May 2020

I am not sure if this is a python3.8 issue. I upgraded from 2019.2 to 3000.2 on Oracle Linux 7. I am getting the same issue with python3.6

2020-05-07 16:59:30,682 [salt.minion :1060][CRITICAL][31223] Unexpected error while connecting to salt
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/minion.py", line 1037, in _connect_minion
yield minion.connect_master(failed=failed)
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 239, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/usr/lib/python3.6/site-packages/salt/minion.py", line 1236, in connect_master
yield self._post_master_init(master)
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 239, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1070, in run
yielded = self.gen.send(value)
File "/usr/lib/python3.6/site-packages/salt/minion.py", line 1283, in _post_master_init
cleanup=[master_event(type='alive')])
File "/usr/lib/python3.6/site-packages/salt/utils/schedule.py", line 105, in __new__
utils=utils)
File "/usr/lib/python3.6/site-packages/salt/utils/schedule.py", line 152, in __singleton_init__
self.time_offset = self.functions.get('timezone.get_offset', lambda: '0000')()
File "/usr/lib/python3.6/site-packages/salt/modules/timezone.py", line 234, in get_offset
if 'AIX' not in __grains__['os_family']:
File "/usr/lib/python3.6/site-packages/salt/utils/context.py", line 236, in __getitem__
return self._dict()[key]
KeyError: 'os_family'

cingeyedog picture cingeyedog on 7 May 2020

`salt --versions-report
Salt Version:
Salt: 3000.2

Dependency Versions:
cffi: Not Installed
cherrypy: unknown
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.8.1
libgit2: Not Installed
M2Crypto: 0.35.2
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.6.2
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: 3.9.7
pygit2: Not Installed
Python: 3.6.8 (default, Aug 7 2019, 08:02:28)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 15.3.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.1.4

System Versions:
dist: oracle 7.7
locale: UTF-8
machine: x86_64
release: 3.10.0-1062.12.1.el7.x86_64
system: Linux
version: Oracle Linux Server 7.7`

cingeyedog picture cingeyedog on 7 May 2020

Sorry. I got it figured out. The problem was that when I upgraded salt via rpms, the salt cache wasn't cleared. To fix, stop salt-minion, delete the cache files, start salt-minion

cingeyedog picture cingeyedog on 8 May 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

chrismoos picture chrismoos  路  54Comments
Jiaion picture Jiaion  路  52Comments
sivann picture sivann  路  73Comments
Zogg picture Zogg  路  80Comments
kevich picture kevich  路  66Comments