Salt: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 22: invalid start byte

Created on 23 May 2020 · 8Comments · Source: saltstack/salt

Description of Issue

Nearly every salt command gives an exception.

root@master:/prj/salt# salt -l debug v test.versions_report
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/bind.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/bind.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/cli_summary.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/cli_summary.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/file_roots.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/file_roots.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/gitfs.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/gitfs.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/master_roots.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/master_roots.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/peer_run.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/peer_run.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/pidfile.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/pidfile.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/pillar_roots.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/pillar_roots.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/publisher_acl.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/publisher_acl.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/saltapi.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/saltapi.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/socket.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/socket.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/syndic_pidfile.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/syndic_pidfile.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/timeout_salt_api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/timeout_salt_api.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/vault.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/vault.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: master
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/bind.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/bind.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/cli_summary.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/cli_summary.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/file_roots.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/file_roots.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/gitfs.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/gitfs.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/master_roots.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/master_roots.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/peer_run.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/peer_run.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/pidfile.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/pidfile.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/pillar_roots.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/pillar_roots.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/publisher_acl.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/publisher_acl.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/saltapi.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/saltapi.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/socket.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/socket.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/syndic_pidfile.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/syndic_pidfile.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/timeout_salt_api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/timeout_salt_api.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/vault.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/vault.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: master
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'master_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Closing AsyncZeroMQReqChannel instance
[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] Reading minion list from /var/cache/salt/master/jobs/f3/3bd06730eccf8be9286990534dae35a8ece0439be7efc715b91e30a2d2d770/.minions.p
[DEBUG   ] get_iter_returns for jid 20200522214502387287 sent to {'v'} will timeout at 23:45:17.445519
[ERROR   ] Exception occurred in Subscriber while handling stream: 'utf-8' codec can't decode byte 0x82 in position 22: invalid start byte
[DEBUG   ] Closing IPCMessageSubscriber instance
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 22: invalid start byte
Traceback (most recent call last):
  File "/usr/local/bin/salt", line 10, in <module>
    sys.exit(salt_main())
  File "/usr/local/lib/python3.7/dist-packages/salt/scripts.py", line 530, in salt_main
    client.run()
  File "/usr/local/lib/python3.7/dist-packages/salt/cli/salt.py", line 194, in run
    for full_ret in cmd_func(**kwargs):
  File "/usr/local/lib/python3.7/dist-packages/salt/client/__init__.py", line 775, in cmd_cli
    **kwargs):
  File "/usr/local/lib/python3.7/dist-packages/salt/client/__init__.py", line 1525, in get_cli_event_returns
    **kwargs
  File "/usr/local/lib/python3.7/dist-packages/salt/client/__init__.py", line 1102, in get_iter_returns
    for raw in ret_iter:
  File "/usr/local/lib/python3.7/dist-packages/salt/client/__init__.py", line 1036, in get_returns_no_block
    no_block=True, auto_reconnect=self.auto_reconnect)
  File "/usr/local/lib/python3.7/dist-packages/salt/utils/event.py", line 638, in get_event
    ret = self._get_event(wait, tag, match_func, no_block)
  File "/usr/local/lib/python3.7/dist-packages/salt/utils/event.py", line 544, in _get_event
    raw = self.subscriber.read_sync(timeout=wait)
  File "/usr/local/lib/python3.7/dist-packages/salt/transport/ipc.py", line 693, in read_sync
    return self.io_loop.run_sync(lambda: self._read(timeout))
  File "/usr/local/lib/python3.7/dist-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync
    return future_cell[0].result()
  File "/usr/local/lib/python3.7/dist-packages/salt/ext/tornado/concurrent.py", line 239, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/local/lib/python3.7/dist-packages/salt/ext/tornado/gen.py", line 1070, in run
    yielded = self.gen.send(value)
  File "/usr/local/lib/python3.7/dist-packages/salt/transport/ipc.py", line 678, in _read
    raise exc_to_raise  # pylint: disable=E0702
  File "/usr/local/lib/python3.7/dist-packages/salt/transport/ipc.py", line 651, in _read
    for framed_msg in self.unpacker:
  File "msgpack/_unpacker.pyx", line 562, in msgpack._cmsgpack.Unpacker.__next__
  File "msgpack/_unpacker.pyx", line 493, in msgpack._cmsgpack.Unpacker._unpack
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 22: invalid start byte
Traceback (most recent call last):
  File "/usr/local/bin/salt", line 10, in <module>
    sys.exit(salt_main())
  File "/usr/local/lib/python3.7/dist-packages/salt/scripts.py", line 530, in salt_main
    client.run()
  File "/usr/local/lib/python3.7/dist-packages/salt/cli/salt.py", line 194, in run
    for full_ret in cmd_func(**kwargs):
  File "/usr/local/lib/python3.7/dist-packages/salt/client/__init__.py", line 775, in cmd_cli
    **kwargs):
  File "/usr/local/lib/python3.7/dist-packages/salt/client/__init__.py", line 1525, in get_cli_event_returns
    **kwargs
  File "/usr/local/lib/python3.7/dist-packages/salt/client/__init__.py", line 1102, in get_iter_returns
    for raw in ret_iter:
  File "/usr/local/lib/python3.7/dist-packages/salt/client/__init__.py", line 1036, in get_returns_no_block
    no_block=True, auto_reconnect=self.auto_reconnect)
  File "/usr/local/lib/python3.7/dist-packages/salt/utils/event.py", line 638, in get_event
    ret = self._get_event(wait, tag, match_func, no_block)
  File "/usr/local/lib/python3.7/dist-packages/salt/utils/event.py", line 544, in _get_event
    raw = self.subscriber.read_sync(timeout=wait)
  File "/usr/local/lib/python3.7/dist-packages/salt/transport/ipc.py", line 693, in read_sync
    return self.io_loop.run_sync(lambda: self._read(timeout))
  File "/usr/local/lib/python3.7/dist-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync
    return future_cell[0].result()
  File "/usr/local/lib/python3.7/dist-packages/salt/ext/tornado/concurrent.py", line 239, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/local/lib/python3.7/dist-packages/salt/ext/tornado/gen.py", line 1070, in run
    yielded = self.gen.send(value)
  File "/usr/local/lib/python3.7/dist-packages/salt/transport/ipc.py", line 678, in _read
    raise exc_to_raise  # pylint: disable=E0702
  File "/usr/local/lib/python3.7/dist-packages/salt/transport/ipc.py", line 651, in _read
    for framed_msg in self.unpacker:
  File "msgpack/_unpacker.pyx", line 562, in msgpack._cmsgpack.Unpacker.__next__
  File "msgpack/_unpacker.pyx", line 493, in msgpack._cmsgpack.Unpacker._unpack
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 22: invalid start byte

Setup

No changes in the default setup.
On de master i did a upgrade to 3000.3 from a working environment

On the client i did a plane install by apt install salt-minion

Steps to Reproduce Issue

On a raspberry pi running raspbian buster (10.4) run:

salt -l debug v test.versions_report

Versions Report

root@master:/prj/salt# salt --versions-report
Salt Version:
           Salt: 3000.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.7.3 (default, Dec 20 2019, 18:57:59)
   python-gnupg: Not Installed
         PyYAML: 5.3.1
          PyZMQ: 19.0.1
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.2

System Versions:
           dist: debian 10.4 
         locale: UTF-8
        machine: armv7l
        release: 4.19.66-v7+
         system: Linux
        version: debian 10.4 



root@v:/home/me# salt-minion -l debug --versions-report
/usr/lib/python2.7/dist-packages/salt/scripts.py:212: DeprecationWarning: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date.  Salt will drop support for Python 2.7 in the Sodium release or later.
Salt Version:
           Salt: 3000.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.7.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.16 (default, Oct 10 2019, 22:02:15)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 17.1.2
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1

System Versions:
           dist: debian 10.4 
         locale: UTF-8
        machine: armv7l
        release: 4.19.97-v7l+
         system: Linux
        version: debian 10.4

Bug Core severity-high

Source

plReynaerde

All 8 comments

@plReynaerde python2 has been deprecated, have you tried installing salt with python3?

Akm0d on 26 May 2020

@plReynaerde python2 has been deprecated, have you tried installing salt with python3?

Yes, i have. By the way salt-minion depends on the python2 apps.

apt-cache depends salt-minion

salt-minion
Depends: lsb-base
Depends: bsdmainutils
Depends: dctrl-tools
Depends: python-crypto
Depends: python-zmq
Depends: salt-common
Depends: init-system-helpers
Depends:
python
Recommends: debconf-utils
Recommends: dmidecode
Recommends:
Recommends:
Suggests: python-augeas
It pulls all python 2 packages in, instead of python3

The python3 packages are in the repository:

apt-cache search python|grep \-zmq

pypy-zmq - PyPy bindings for 0MQ library
python-zmq - Python bindings for 0MQ library
python-zmq-dbg - Python bindings for 0MQ library - debugging files
python3-zmq - Python3 bindings for 0MQ library
python3-zmq-dbg - Python3 bindings for 0MQ library - debugging files

plReynaerde on 26 May 2020

👍1

Was this an upgrade that took salt from py2 to py3 with a dist-upgrade or is this a raw first-time install of salt on the pi?

Akm0d on 27 May 2020

The master was an upgrade from a 3000.1 version (I believe it was already a python3 based one) and the minion was a fresh install.

plReynaerde on 27 May 2020

@plReynaerde It is likely that you had Py 2.7 already installed on the Raspberry Pi since SaltStack does not support Py 2 with Buster. It has only ever been built with Python 3 support. Debian 9 shipped with Debian's own fork of Salt (ran out of /usr/local/bin and /usr/local/lib) which conflicted with the version supplied by SaltStack (runs out of /usr/bin and /usr/lib). Is it possible that you performed a dist-upgrade from Debian 9 to Debian 10 and still have the older Debian fork of Salt present - this would account for the problems you are now seeing.

Search for salt-call in your system, it should be present in path /usr/bin

root@raspberrypi:/home/pi# apt-cache policy salt-common
salt-common:
  Installed: 3000.3+ds-1
  Candidate: 3000.3+ds-1
  Version table:
 *** 3000.3+ds-1 500
        500 https://repo.saltstack.com/py3/debian/10/armhf/latest buster/main armhf Packages
        100 /var/lib/dpkg/status
     2018.3.4+dfsg1-6+deb10u1 500
        500 http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages
root@raspberrypi:/home/pi# apt-cache depends salt-common
salt-common
  Depends: python3-apt
  Depends: python3-dateutil
  Depends: python3-jinja2
  Depends: python3-msgpack
  Depends: python3-pkg-resources
  Depends: python3-requests
  Depends: python3-yaml
  Depends: python3-systemd
  Depends: python3-psutil
  Depends: python3-distro
  Depends: python3-crypto
  Depends: python3-markupsafe
  Depends: python3-zmq
  Depends: <python3:any>
    python3
  Breaks: python3-mako
  Recommends: lsb-release
  Recommends: python3-croniter
  Suggests: python3-pycurl
  Suggests: python3-twisted
root@raspberrypi:/home/pi#

Note: to upgrade Salt form Py 2 to Py 3, it is currently recommended to remove and re-install with the latest Py 3 supported version.

dmurphy18 on 27 May 2020

Reviewing the errors at the start of the issue, I believe that the Debian fork of Salt is active on your Pi

File "/usr/local/lib/python3.7/dist-packages/salt/client/__init__.py

The correct path for Salt's version of the same file

root@raspberrypi:/home/pi# ls -al /usr/lib/python3/dist-packages/salt/client/__init__.py 
-rw-r--r-- 1 root root 79021 May  7 08:38 /usr/lib/python3/dist-packages/salt/client/__init__.py
root@raspberrypi:/home/pi#

dmurphy18 on 27 May 2020

👍1

Well probably, but i install salt from:
/etc/apt/sources.list.d$ cat salt.list
deb http://repo.saltstack.com/apt/debian/9/armhf/latest stretch main
(There is was no 'buster', yet)

But you triggered me... There was still a salt dir in the /usr/lib/python2.7/dist-packages that was the 3000.1 version. So i removed that one. So i guess the apt update; apt dist-upgrade did not go well. An upgrade from stretch to buster leaves python2.7 as default. And it looks like that the upgrade of salt changed, between the 3000.1 (python2) and 3000.3 (python3), from location /usr/local to /usr and that the 2.7 packages where not removed.
What i did was remove the 2.7 dist-packages of salt and the salt commands from /usr/local/bin and changed the default python to python3 (ln -s) And that seems to have done the trick...

plReynaerde on 27 May 2020

And @dmurphy18 you were right. I missed the deinstall the 2.7 version and install the 3 version. And basically that's what i just have done... Thank for your help!!

plReynaerde on 28 May 2020

Was this page helpful?

0 / 5 - 0 ratings