Salt: Could "pip3 install salt" break the existing salt installation?

[wsw70]

Description of Issue/Question

I had a server and several minions running fine. Since I wanted to experience with the Python API, I installed the salt module (pip3 install salt).

It look it broke all salt commands (I am not sure but this is the only change I can think of).

Running salt '*' test.ping yields

[ERROR   ] Exception occurred in Subscriber while handling stream: 'utf-8' codec can't decode byte 0x82 in position 22: invalid start byte
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnpackValueError: 'utf-8' codec can't decode byte 0x82 in position 22: invalid start byte
Traceback (most recent call last):
  File "msgpack/_unpacker.pyx", line 499, in msgpack._unpacker.Unpacker._unpack
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 22: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/salt", line 11, in <module>
    sys.exit(salt_main())
  File "/usr/local/lib/python3.5/dist-packages/salt/scripts.py", line 476, in salt_main
    client.run()
  File "/usr/local/lib/python3.5/dist-packages/salt/cli/salt.py", line 173, in run
    for full_ret in cmd_func(**kwargs):
  File "/usr/local/lib/python3.5/dist-packages/salt/client/__init__.py", line 806, in cmd_cli
    **kwargs):
  File "/usr/local/lib/python3.5/dist-packages/salt/client/__init__.py", line 1589, in get_cli_event_returns
    **kwargs
  File "/usr/local/lib/python3.5/dist-packages/salt/client/__init__.py", line 1174, in get_iter_returns
    for raw in ret_iter:
  File "/usr/local/lib/python3.5/dist-packages/salt/client/__init__.py", line 1099, in get_returns_no_block
    no_block=True, auto_reconnect=self.auto_reconnect)
  File "/usr/local/lib/python3.5/dist-packages/salt/utils/event.py", line 632, in get_event
    ret = self._get_event(wait, tag, match_func, no_block)
  File "/usr/local/lib/python3.5/dist-packages/salt/utils/event.py", line 540, in _get_event
    raw = self.subscriber.read_sync(timeout=wait)
  File "/usr/local/lib/python3.5/dist-packages/salt/transport/ipc.py", line 705, in read_sync
    return ret_future.result()
  File "/usr/local/lib/python3.5/dist-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/local/lib/python3.5/dist-packages/tornado/gen.py", line 1069, in run
    yielded = self.gen.send(value)
  File "/usr/local/lib/python3.5/dist-packages/salt/transport/ipc.py", line 681, in _read_sync
    raise exc_to_raise  # pylint: disable=E0702
  File "/usr/local/lib/python3.5/dist-packages/salt/transport/ipc.py", line 653, in _read_sync
    for framed_msg in self.unpacker:
  File "msgpack/_unpacker.pyx", line 574, in msgpack._unpacker.Unpacker.__next__
  File "msgpack/_unpacker.pyx", line 519, in msgpack._unpacker.Unpacker._unpack
msgpack.exceptions.UnpackValueError: 'utf-8' codec can't decode byte 0x82 in position 22: invalid start byte
Traceback (most recent call last):
  File "msgpack/_unpacker.pyx", line 499, in msgpack._unpacker.Unpacker._unpack
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 22: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/salt", line 11, in <module>
    sys.exit(salt_main())
  File "/usr/local/lib/python3.5/dist-packages/salt/scripts.py", line 476, in salt_main
    client.run()
  File "/usr/local/lib/python3.5/dist-packages/salt/cli/salt.py", line 173, in run
    for full_ret in cmd_func(**kwargs):
  File "/usr/local/lib/python3.5/dist-packages/salt/client/__init__.py", line 806, in cmd_cli
    **kwargs):
  File "/usr/local/lib/python3.5/dist-packages/salt/client/__init__.py", line 1589, in get_cli_event_returns
    **kwargs
  File "/usr/local/lib/python3.5/dist-packages/salt/client/__init__.py", line 1174, in get_iter_returns
    for raw in ret_iter:
  File "/usr/local/lib/python3.5/dist-packages/salt/client/__init__.py", line 1099, in get_returns_no_block
    no_block=True, auto_reconnect=self.auto_reconnect)
  File "/usr/local/lib/python3.5/dist-packages/salt/utils/event.py", line 632, in get_event
    ret = self._get_event(wait, tag, match_func, no_block)
  File "/usr/local/lib/python3.5/dist-packages/salt/utils/event.py", line 540, in _get_event
    raw = self.subscriber.read_sync(timeout=wait)
  File "/usr/local/lib/python3.5/dist-packages/salt/transport/ipc.py", line 705, in read_sync
    return ret_future.result()
  File "/usr/local/lib/python3.5/dist-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/local/lib/python3.5/dist-packages/tornado/gen.py", line 1069, in run
    yielded = self.gen.send(value)
  File "/usr/local/lib/python3.5/dist-packages/salt/transport/ipc.py", line 681, in _read_sync
    raise exc_to_raise  # pylint: disable=E0702
  File "/usr/local/lib/python3.5/dist-packages/salt/transport/ipc.py", line 653, in _read_sync
    for framed_msg in self.unpacker:
  File "msgpack/_unpacker.pyx", line 574, in msgpack._unpacker.Unpacker.__next__
  File "msgpack/_unpacker.pyx", line 519, in msgpack._unpacker.Unpacker._unpack
msgpack.exceptions.UnpackValueError: 'utf-8' codec can't decode byte 0x82 in position 22: invalid start byte

salt-run commands are OK (for instance salt-run manage.present outputs the connected minions)

Versions Report

# salt --versions-report
Salt Version:
           Salt: 2017.7.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.2
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.5.3 (default, Jan 19 2017, 14:11:04)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.4
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.1.6

System Versions:
           dist: debian 9.3
         locale: UTF-8
        machine: x86_64
        release: 4.9.0-5-amd64
         system: Linux
        version: debian 9.3

[wsw70]

After some search I discovered that a new installation of salt appeared in /usr/local (including /usr/local/bin/salt). Would that happen after installing the python module? (the initial installation was through the repository

[gtmanfred]

On debian/ubuntu pip installs put stuff in /usr/local/lib, which could cause problems if you have python2 version installed to /usr/lib. I would not recommend having separate installs like this on the same system, as it will definitely cause problems with python trying to import from different places.

Also, your salt-master is probably still running the python2 version. since the service unit probably has it hard coded to /usr/bin/salt-master.

[wsw70]

@gtmanfred Thanks - this is the reason. Is there a way to install the Python3 version then? I tried via apt (installs the Python 2 version) and via the bootstrap script (which also installs the python 2 version). Should I then remove them and leave the pip3-installed one?

[gtmanfred]

We are going to have python3 packages for oxygen release which is our next major release. And it should be much more stable as a python3 release.

Personally i would not run python3 until the oxygen release. But if you have to run it, i would not use bootstrap or apt, use a clean machine and pip3 install it. Then copy the /etc/systemd/system scripts, and then the config files to /etc/salt/.

[wsw70]

@gtmanfred Thank you - all is clear now. Since all our scripting is Python 3 I will go the pip3 way for now, awaiting the oxygen release (and, should you be interested - feedback on possible issues)

[blake-st]

@gtmanfred Hi Daniel, can you provide a bit more details how to solve the problem. I installed the salt cluster previously it is working fine. Then I run "pip3 install salt" trying to install the salt python client. After that I ran into the same error

[timwsuqld]

@gtmanfred Did we actually get python3 packages with Oxygen? We're trying to use pip3 install salt packages for some scripting and getting issues. (#44283)

[gtmanfred]

I no longer work for saltstack.

You can pip3 install salt, 2018.3 and 2019.2 should both work.

But there are py3 repos listed on repo.saltstack.com for both of those releases.

[timwsuqld]

@gtmanfred Thanks for that. For anyone else stumbling across this issue, pip3 install salt doesn't play nice if you're trying to use the python2 salt minon/master. There are python3 versions of the salt packages, which you should upgrade to. If using salt-bootstrap you can pass -x python3 to it when installing salt and you'll get the python3 packages. Otherwise, head to http://repo.saltstack.com and ensure you choose the py3 versions. I've had good luck just replacing the apt repos with the py3 versions and apt detects that it's an upgrade from the previous py2 versions.