Salt: External file_tree pillar is broken with salt-ssh

Created on 27 Jun 2019  路  11Comments  路  Source: saltstack/salt

Description of Issue

Having configured a file tree external pillar (without templating enabled), salt-ssh call results in the following trace:

[ERROR   ] TypeError encountered executing state.apply: Object of type bytes is not JSON serializable
Traceback (most recent call last):
  File "/Users/user1/Library/Python/3.7/lib/python/site-packages/salt/client/ssh/__init__.py", line 1159, in run_wfunc
    result = self.wfuncs[self.fun](*self.args, **self.kwargs)
  File "/Users/user1/Library/Python/3.7/lib/python/site-packages/salt/client/ssh/wrapper/state.py", line 513, in apply_
    return highstate(**kwargs)
  File "/Users/user1/Library/Python/3.7/lib/python/site-packages/salt/client/ssh/wrapper/state.py", line 705, in highstate
    roster_grains)
  File "/Users/user1/Library/Python/3.7/lib/python/site-packages/salt/client/ssh/state.py", line 193, in prep_trans_tar
    salt.utils.json.dump(pillar, fp_)
  File "/Users/user1/Library/Python/3.7/lib/python/site-packages/salt/utils/json.py", line 121, in dump
    return json_module.dump(obj, fp, **kwargs)  # future lint: blacklisted-function
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 179, in dump
    for chunk in iterable:
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable

This does not occur when using Python2.7.

Setup

Setup pillar.file_tree

ext_pillar:
  - file_tree:
        root_dir: file-tree
        follow_dir_links: False
        keep_newline: True

Place binary file in file_tree.

Steps to Reproduce Issue

Attempt to apply state using salt-ssh.

Versions Report

Salt Version:
           Salt: 2019.2.0

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.7.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10.1
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.12
   msgpack-pure: Not Installed
 msgpack-python: 0.6.1
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.7.3 (default, Mar 27 2019, 09:23:15)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 18.0.1
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1

System Versions:
           dist:
         locale: UTF-8
        machine: x86_64
        release: 18.6.0
         system: Darwin
        version: 10.14.5 x86_64
Bug Salt-SSH help-wanted
Source
picture kevinhore

Most helpful comment

Please fix this, we can't use salt now for weeks due to this bug :(

picture h0jeZvgoxFepBQ2C on 21 Jul 2020
👍2

All 11 comments

Hello,

same issue here.
We are using the pillar_ldap external pillar, which explicitly return bytes as per the documentation of the python-ldap library it's using: https://www.python-ldap.org/en/latest/bytes_mode.html#what-s-text-and-what-s-bytes

By default, Salt is using the Python json module.
However, there's code available to use another module instead.

I have a quick and dirty hack that use the ujson module instead in the Salt-SSH client which is working fine with bytestrings.

ret = salt.utils.json.dumps({'local': {'return': result}}, _json_module=ujson)

Maybe a nice way to handle this would be to be able to set a default json module for Salt instead of the standard one (in the configuration file, for example).

Or simply handle more properly bytestring in the JSON Utils module: https://github.com/saltstack/salt/blob/master/salt/utils/json.py

If there's a preferred method, I'm willing to try for a PR :)

Versions Report

Salt Version:
           Salt: 2019.2.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.8 (default, Oct  7 2019, 12:59:55)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-48-generic
         system: Linux
        version: Ubuntu 18.04 bionic
ghost picture ghost on 5 Nov 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

stale[bot] picture stale[bot] on 7 Jan 2020

Still a problem.

kevinhore picture kevinhore on 7 Jan 2020

Thank you for updating this issue. It is no longer marked as stale.

stale[bot] picture stale[bot] on 7 Jan 2020

This same issue happens when using !!binary YAML syntax on a pillar and then try to use salt-ssh against it.
According to the docs this is not "allowed" on salt-ssh, though - wonder if this issue is one of its causes.

zenitraM picture zenitraM on 16 Jan 2020

This also affects me, raising the error while parsing this:

set-timezone-vienna:
  timezone.system:
    - name: "Europe/Berlin"
    - utc: True
h0jeZvgoxFepBQ2C picture h0jeZvgoxFepBQ2C on 18 May 2020

Using an ext_pillar of type file_tree appears to be fundamentally broken with salt-ssh running under Python 3, and not just with binary files as implied by the issue summary. The file_tree implementation appears to be loading each file in the file tree as a chunk of bytes, and these objects cannot subsequently be serialized.

Having looked at the file_tree implementation, I believe I have a workaround (for text files, at least). By declaring that templates are allowed in the file tree鈥檚 files, and also declaring a default renderer (e.g. jinja), a working code path in the file_tree implementation is used instead of the broken one. Thus, using an ext_pillar definition along the following lines appears to work:

ext_pillar:
  - file_tree:
        root_dir: some-path-here
        follow_dir_links: False
        keep_newline: True
        render_default: jinja
        template: True

Naturally, this workaround requires that the files in the file tree not contain anything that may be inadvertently processed as a Jinja template.

@kevinhore If the above seems plausible to you, perhaps you could update this issue summary accordingly, as it presently may be underplaying the magnitude of the problem? TIA.

djneades picture djneades on 26 Jun 2020
👍1

Hello,

same issue here.
We are using the pillar_ldap external pillar, which explicitly return bytes as per the documentation of the python-ldap library it's using: https://www.python-ldap.org/en/latest/bytes_mode.html#what-s-text-and-what-s-bytes

By default, Salt is using the Python json module.
However, there's code available to use another module instead.

I have a quick and dirty hack that use the ujson module instead in the Salt-SSH client which is working fine with bytestrings.

ret = salt.utils.json.dumps({'local': {'return': result}}, _json_module=ujson)

Maybe a nice way to handle this would be to be able to set a default json module for Salt instead of the standard one (in the configuration file, for example).

Or simply handle more properly bytestring in the JSON Utils module: https://github.com/saltstack/salt/blob/master/salt/utils/json.py

If there's a preferred method, I'm willing to try for a PR :)

Versions Report

Salt Version:
           Salt: 2019.2.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.8 (default, Oct  7 2019, 12:59:55)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-48-generic
         system: Linux
        version: Ubuntu 18.04 bionic

I was also having this issue with salt-ssh with pillar_ldap so I have tried your hack and it seems to work for me. I did:

  1. install python3-ujson (on the salt-ssh master)
  2. modified /usr/lib/python3.6/site-packages/salt/client/ssh/__init__.py as follows:
@@ -21,6 +21,7 @@
 import binascii
 import sys
 import datetime
+import ujson

 # Import salt libs
 import salt.output
@@ -1174,7 +1175,8 @@
         if isinstance(result, dict) and 'local' in result:
             ret = salt.utils.json.dumps({'local': result['local']})
         else:
-            ret = salt.utils.json.dumps({'local': {'return': result}})
+            #ret = salt.utils.json.dumps({'local': {'return': result}})
+            ret = salt.utils.json.dumps({'local': {'return': result}}, _json_module=ujson)
         return ret, retcode

     def _cmd_str(self):

Was there any other locations I should be doing this?

ericjmcalvin picture ericjmcalvin on 9 Jul 2020

https://github.com/saltstack/salt/issues/53620#issuecomment-650153705 shows that the current implementation of the file_tree external pillar is attempting to load each file as bytes which can't then be serialized.

Updated title to emphasize that file_tree is fundamentally broken.

kevinhore picture kevinhore on 9 Jul 2020

Please fix this, we can't use salt now for weeks due to this bug :(

h0jeZvgoxFepBQ2C picture h0jeZvgoxFepBQ2C on 21 Jul 2020
👍2

Here is my stacktrace:

...
[DEBUG   ] Rendered data from file: /etc/salt/tmp/cache/master/files/base/openssh/init.sls:


openssh:

  pkg.latest:
    - name: openssh-server

  service.running:
    - enable: True
    - name: ssh

    - require:
      - pkg: openssh-server

  require:
    - user: deployuser

[DEBUG   ] Results of YAML rendering:
OrderedDict([('openssh', OrderedDict([('pkg.latest', [OrderedDict([('name', 'openssh-server')])]), ('service.running', [OrderedDict([('enable', True)]), OrderedDict([('name', 'ssh')]), OrderedDict([('require', [OrderedDict([('pkg', 'openssh-server')])])])]), ('require', [OrderedDict([('user', 'deployuser')])])]))])
[PROFILE ] Time (in seconds) to render '/etc/salt/tmp/cache/master/files/base/openssh/init.sls' using 'yaml' renderer: 0.00039505958557128906
[ERROR   ] TypeError encountered executing state.apply: Object of type bytes is not JSON serializable
Traceback (most recent call last):
  File "/Users/user/.pyenv/versions/3.7.7/lib/python3.7/site-packages/salt-3001_154_g45efc4c142-py3.7.egg/salt/client/ssh/__init__.py", line 1240, in run_wfunc
    result = self.wfuncs[self.fun](*self.args, **self.kwargs)
  File "/Users/user/.pyenv/versions/3.7.7/lib/python3.7/site-packages/salt-3001_154_g45efc4c142-py3.7.egg/salt/client/ssh/wrapper/state.py", line 504, in apply_
    return sls(mods, **kwargs)
  File "/Users/user/.pyenv/versions/3.7.7/lib/python3.7/site-packages/salt-3001_154_g45efc4c142-py3.7.egg/salt/client/ssh/wrapper/state.py", line 226, in sls
    roster_grains,
  File "/Users/user/.pyenv/versions/3.7.7/lib/python3.7/site-packages/salt-3001_154_g45efc4c142-py3.7.egg/salt/client/ssh/state.py", line 203, in prep_trans_tar
    salt.utils.json.dump(pillar, fp_)
  File "/Users/user/.pyenv/versions/3.7.7/lib/python3.7/site-packages/salt-3001_154_g45efc4c142-py3.7.egg/salt/utils/json.py", line 126, in dump
    return json_module.dump(obj, fp, **kwargs)  # future lint: blacklisted-function
  File "/Users/user/.pyenv/versions/3.7.7/lib/python3.7/json/__init__.py", line 179, in dump
    for chunk in iterable:
  File "/Users/user/.pyenv/versions/3.7.7/lib/python3.7/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/Users/user/.pyenv/versions/3.7.7/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/Users/user/.pyenv/versions/3.7.7/lib/python3.7/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/Users/user/.pyenv/versions/3.7.7/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable
[DEBUG   ] LazyLoaded nested.output
myserver:
    TypeError encountered executing state.apply: Object of type bytes is not JSON serializable
h0jeZvgoxFepBQ2C picture h0jeZvgoxFepBQ2C on 21 Jul 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

saurabhnemade picture saurabhnemade  路  3Comments
seanacais picture seanacais  路  3Comments
golmaal picture golmaal  路  3Comments
Inveracity picture Inveracity  路  3Comments
commutecat picture commutecat  路  3Comments