Salt: bytes error in mysql module

[angeloudy]

Description of Issue/Question

I recently switch from python2.7 to python 3.6 and a lot of modules are stopped working.

The first issue is pip: #44980
Second issue is mysql

Function: mysql_grants.present
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/local/lib/python3.6/site-packages/salt/state.py", line 1843, in call
                  **cdata['kwargs'])
                File "/usr/local/lib/python3.6/site-packages/salt/loader.py", line 1795, in wrapper
                  return f(*args, **kwargs)
                File "/usr/local/lib/python3.6/site-packages/salt/states/mysql_grants.py", line 150, in present
                  grant, database, user, host, grant_option, escape, **connection_args
                File "/usr/local/lib/python3.6/site-packages/salt/modules/mysql.py", line 1778, in grant_exists
                  grants = user_grants(user, host, **connection_args)
                File "/usr/local/lib/python3.6/site-packages/salt/modules/mysql.py", line 1745, in user_grants
                  tmp = grant[0].split(' IDENTIFIED BY')[0]
              TypeError: a bytes-like object is required, not 'str'
     Started: 10:50:28.599004
    Duration: 11.843 ms
     Changes:

Third issue is with http

 Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/local/lib/python3.6/site-packages/salt/state.py", line 1843, in call
                  **cdata['kwargs'])
                File "/usr/local/lib/python3.6/site-packages/salt/loader.py", line 1795, in wrapper
                  return f(*args, **kwargs)
                File "/usr/local/lib/python3.6/site-packages/salt/states/http.py", line 153, in wait_for_successful_query
                  raise caught_exception  # pylint: disable=E0702
                File "/usr/local/lib/python3.6/site-packages/salt/states/http.py", line 144, in wait_for_successful_query
                  ret = query(name, wait_for=wait_for, **kwargs)
                File "/usr/local/lib/python3.6/site-packages/salt/states/http.py", line 96, in query
                  if match in data.get('text', ''):
              TypeError: a bytes-like object is required, not 'str'
     Started: 10:27:10.800930
    Duration: 301579.96 ms

All the issues I am having have similar errors:

TypeError: a bytes-like object is required, not 'str'

Can someone look at this?

Setup

FreeBSD 11.1

Steps to Reproduce Issue

1. install package py36-salt
2. to reproduce pip issue, create a state file with pip.installed and added it to top.sls and run salt state.highstate
3. to reproduce mysql issue, create a state file with mysql state and run it from highstate.

Versions Report

Salt Version:
Salt: 2017.7.2

Dependency Versions:
cffi: 1.11.2
cherrypy: Not Installed
dateutil: 2.6.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.10
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.8
mysql-python: 1.2.5
pycparser: 2.18
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.3 (default, Dec 11 2017, 00:25:34)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 16.0.3
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.2
ZMQ: 4.2.2

System Versions:
dist:
locale: UTF-8
machine: amd64
release: 11.1-RELEASE-p4
system: FreeBSD
version: Not Installed

[Ch3LL]

@angeloudy can you please share a sanitized version of your state files to help replicate this issue?

[drel4o]

Hello there,

I am experiencing the same problem, so here is an example of a failing state:

test_db grants:
mysql_grants.present:
- host: localhost
- grant: all
- database: 'test_db.*'
- user: {{ pillar['db']['user'] }}

The database and the user exist in mysql.

Salt Version:
Salt: 2017.7.4

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.6.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.10
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.7
msgpack-pure: Not Installed
msgpack-python: 0.5.4
mysql-python: 1.3.10
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.4+ (default, Feb 12 2018, 08:25:03)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 16.0.2
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.2.3
System Versions:
dist: Ubuntu 18.04 bionic
locale: UTF-8
machine: x86_64
release: 4.13.0-32-generic
system: Linux
version: Ubuntu 18.04 bionic

Thank you in advance. :)
D

[angeloudy]

This was not an issue of salt, but the issue of python mysql library.
I fixed it with the following patch

--- MySQLdb/cursors.py.orig
+++ MySQLdb/cursors.py
@@ -21,7 +21,17 @@
 else:
     text_type = str

-
+def convert_to_str(var):
+    if isinstance(var,tuple):
+        return tuple(convert_to_str(item) for item in var)
+    if isinstance(var,list):
+        return ([convert_to_str(item) for item in var])
+    elif isinstance(var,dict):
+        return {convert_to_str(key):convert_to_str(value) for key,value in var.items()}
+    elif isinstance(var,bytes):
+        return var.decode('utf-8')
+    else:
+        return var
 #: Regular expression for :meth:`Cursor.executemany`.
 #: executemany only supports simple bulk insert.
 #: You can use it to load large dataset.
@@ -443,6 +453,9 @@
         else:
             result = self._rows
         self.rownumber = len(self._rows)
+        if not PY2:
+            db = self._get_db()
+            result = tuple(convert_to_str(result))
         return result

     def scroll(self, value, mode='relative'):

[angeloudy]

I am still not sure if this needs to be fixed in mysqlclient library or in salt.

[methane]

What is connection_args?
mysqlclient returns Unicode normally.

[methane]

You need to pass charset="utf8mb4" or "utf8" to connection args.

[methane]

I found, salt disables unicode for PY2, but it is disabled for PY3 too.
https://github.com/saltstack/salt/blob/7d41e035f55a21c949a1d75a4e8da22db604f34a/salt/modules/mysql.py#L316-L323

[Ch3LL]

is this still an issue on 2018.3? we added a lot of improvements with unicode in this version

[methane]

I think so, because same configs are in 2018.3.2 tag.
https://github.com/saltstack/salt/blob/2018.3.2/salt/modules/mysql.py#L318-L325

[Ch3LL]

And if you set that to True the failure goes away?

[methane]

I think so, but I'm not salt user.
I'm a maintainer of mysqlclient-python.

[Ch3LL]

ping @regilero looks like you set this to False. Is there a specific reason you see to not change it to True?

[methane]

I suppose he did it for PY2.

use_unicode=True is OK for PY2 if the code is written to use unicode string.
But if not, it should be use_unicode=(not PY2).

[Ch3LL]

k thanks for clarifying. looks like this will need to get fixed up. Will label as a bug.

[andreasnuesslein]

i believe i have the same issue, for me it helped to str() the relevant return-values from MySQLdb's cur.fetchall(), like so:

diff --git a/salt/modules/mysql.py b/salt/modules/mysql.py
index 889de1c..1e32556 100644
--- a/mysql.py
+++ b/mysql.py
@@ -1766,8 +1766,8 @@ def user_grants(user,
     ret = []
     results = cur.fetchall()
     for grant in results:
-        tmp = grant[0].split(' IDENTIFIED BY')[0]
-        if 'WITH GRANT OPTION' in grant[0] and 'WITH GRANT OPTION' not in tmp:
+        tmp = str(grant[0]).split(' IDENTIFIED BY')[0]
+        if 'WITH GRANT OPTION' in str(grant[0]) and 'WITH GRANT OPTION' not in tmp:
             tmp = '{0} WITH GRANT OPTION'.format(tmp)
         ret.append(tmp)
     log.debug(ret)

[zerthimon]

The diff provided by @andreasnuesslein solves the issue!
Thanks a lot!

[Ch3LL]

does it work when using salt.utils.stringutils.to_str() as that will handle cases for both python2 and python3.

[Ch3LL]

actually salt.utils.stringutils.to_unicode() should be used when you are attempting to simply convert a value from a non-string type to string type

[Ch3LL]

but @terminalmage can verify my thinking here

[terminalmage]

Fixed by https://github.com/saltstack/salt/pull/50823.

[morph027]

Any chance we can get this backported to 2017.7 too? ;)

[terminalmage]

Unlikely, since A) there are no more 2017.7 releases planned, and B) the helper function used to make this fix doesn't exist in that release branch.

[morph027]

Okay...as the fix from @andreasnuesslein works, i'm going to check how i can incorporate it into salt.

[ernestas-s]

I use onliner to fix this on daily basis with v2018.3.3
sed -i '1756,1758{s/grant\[0\]/str\(grant\[0\]\)/}' /usr/lib/python3/dist-packages/salt/modules/mysql.py

[terminalmage]

@saltstack/team-core This should be able to be closed since the fix was merged.

[waynew]

:+1:

[pkramme]

I opened a bug report on Launchpad: https://bugs.launchpad.net/ubuntu/+source/salt/+bug/1823364