Salt: [BUG] 3000.3 msgpack version error

[srinath4791]

Recently we upgraded the salt version to 3000.3 and we get the below error, when we try to execute a command

• salt -G environment:x state.sls *.sls

Exception occurred while handling stream: 3279509 exceeds max_str_len(1048576).

Above command generates oddly 4 lines.

Gone through couple of reports related to this issue and it was related to msgpack version used > 0.5.6. But salt 3000.3 version demands msgpack 0.6.2.

When installed msgpack 0.5.6, while starting the salt master it throws below error.

pkg_resources.DistributionNotFound: The 'msgpack!=0.5.5,<1.0.0,>=0.5' distribution was not found and is required by salt .

We use RHEL 6.10 with python 2.7 and took rpm from below.

https://repo.saltstack.com/yum/redhat/6.10/x86_64/archive/3000.3/.

Downgrade is the only option?

[sagetherage]

@srinath4791 are you able to upgrade to Python 3?

[alexey-zhukovin]

@srinath4791 Thank you for your comment.
Could you provide more info:
salt --versions-report

[srinath4791]

@sagetherage no we don't want to upgrade to python 3 as other few things were running on python 2.7 and don't want to disturb them. with the same python 2.7 , salt version 2019.2.5 works like charm.

[srinath4791]

@alexey-zhukovin

/usr/lib/python2.7/site-packages/salt/scripts.py:212: DeprecationWarning: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. Salt will drop support for Python 2.7 in the Sodium release or later.
Salt Version:
Salt: 3000.3

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.8.1
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.6.2
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.13 (default, May 8 2020, 22:36:22)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 14.5.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.0.5

System Versions:
dist: redhat 6.10 Santiago
locale: UTF-8
machine: x86_64
release: 2.6.32-696.20.1.el6.x86_64
system: Linux
version: Red Hat Enterprise Linux Server 6.10 Santiago

[srinath4791]

@alexey-zhukovin - We have now downgraded to 2019.5.2 version and we don't face any issues.

with 3003.3, we faced two issues.

1. as highlighted the error due to msgpack and we were not able to downgrade msgpack to 0.5.6 or lower as the salt version demands 0.6.2 for installation.

after installing msgpack 0.5.6 with --nodeps, service salt-master start was throwing an error below

pkg_resources.DistributionNotFound: The 'msgpack!=0.5.5,<1.0.0,>=0.5' distribution was not found and is required by salt .

msgpack which we installed was msgpack-0.5.6-cp36-cp36m-manylinux1_x86_64.whl.
we also tried with python2-msgpack-0.5.6-5.el7.x86_64.rpm . both was not recognized when we tried to start the salt master.

1. At times, master was not able to connect minion, we were getting below error

salt -G environment:x cmd.run 'yum clean all'
hostname*
Minion did not return. [No response]
ERROR: Minions returned with non-zero exit code

we could not see any errors in the log trace both on master and minion. some times it worked after restart of salt master.

both the above issues we did not faced when we downgraded to 2019.5.2 with python 2.7 and msgpack here is 0.4.6

[alexey-zhukovin]

@srinath4791
Sorry, We can't work on this issue, python 2.7 is not supported for 3000+ versions

[srinath4791]

okay in that case better to put a discretionary message in the 3000.3 Salt repo link, that it is not supported with python2.7.

Below link offers 3000 version with python 2.7?

https://repo.saltstack.com/yum/redhat/6.10/x86_64/archive/3000.3/

[sagetherage]

We can't put a message at that repo level @srinath4791 but we have one here on the landing page: https://repo.saltstack.com/ that we no longer support from Salt v3001 on -- not 3000, however, we did fix some msgpack issues recently.

Attempt to dispel the confusion here, we did support python 2.7 when we originally released Salt v3000 and RHEL 6 , but no longer do as RHEL 6 was EOL and Py 2 and for bugfix releases or point releases they are supported very a short amount of time; however; I we have fixed some msgpack errors in the bugfix release in 3001.1 releasing Monday - 2020-07-20, let me check on that and link back here.

[cmcmarrow]

@srinath4791 Here was the PR that upgraded msgpack to 1.0.0 #57571 it was just merged into the master branch of salt. Some contributors of salt told me that msgpack had a leak of some kind in older versions of msgpack. But I could not find a report any newer than 2018 https://github.com/msgpack/msgpack-python/issues/283. We decide it to upgrade msgpack because some OS’s came with msgpack 1.0.0 built in

I don’t have your .sls so I can’t confirm that my PR does fix the problem. But Exception occurred while handling stream: 3279509 exceeds max_str_len(1048576). smells like a memory leak to me.

But the down side is msgpack 1.0.0 dropped a lot of python2 support and only runs pure python code which will slow things down for you. Also msgpack broke backwards compatibility and it looks like it broke even harder on Pyhton2 https://pypi.org/project/msgpack/

Here are the options you have.
• Upgrade to Python 3 Painful but worth it long term
• Find another state that does what you need
• Fork Salt and Patch msgpack for python 2
• Downgrade to an older version of Salt.

[srinath4791]

@cmcmarrow Thanks mate for the options. we downgraded the salt to 2019 version and looks fine. Will check if we can upgrade to Python 3.x and move to Salt 3001.

[sagetherage]

closing this ticket then and let us know if you continue to see problems