Salt: Salt runners broken with master job cache and postgresql returner

[breshead]

Description of Issue/Question

When the master job cache is setup using postgres, calling salt-run will throw a 'TypeError' in the postgres_local_cache.py module.

My use case is salt-run manage.[not_]present but I tried

    salt-run manage.list_state
    salt-run manage.alived
    salt-run manage.allowed
    salt-run manage.joined
    salt-run doc.execution
    salt-run saltutil.sync_all
    salt-run salt.cmd test.ping

To see if it was specific or not. It appears to be all runners.

When I disable the master job cache (remark out in the config file), the salt runner completes successfully.

Setup

/etc/salt/master.d/master_job_cache.conf

    master_job_cache: postgres_local_cache
    event_return: postgres_local_cache
    master_job_cache.postgres.host: '192.168.8.14'
    master_job_cache.postgres.user: 'myuser'
    master_job_cache.postgres.passwd: 'mypass'
    master_job_cache.postgres.db: 'ejcache'
    master_job_cache.postgres.port: 5432

Steps to Reproduce Issue

With master job cache setup..
On the command salt-master CLI run..

~# salt-run manage.list_state
---- RESULT ----

[ERROR   ] An un-handled exception was caught by salt's global exception handler:
TypeError: decoding Unicode is not supported
Traceback (most recent call last):
  File "/usr/bin/salt-run", line 10, in <module>
    salt_run()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 412, in salt_run
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/run.py", line 40, in run
    ret = runner.run()
  File "/usr/lib/python2.7/dist-packages/salt/runner.py", line 271, in run
    daemonize=False)
  File "/usr/lib/python2.7/dist-packages/salt/client/mixins.py", line 485, in _proc_function
    return self.low(fun, low, full_return=False)
  File "/usr/lib/python2.7/dist-packages/salt/client/mixins.py", line 261, in low
    return self._low(fun, low, print_event=print_event, full_return=full_return)
  File "/usr/lib/python2.7/dist-packages/salt/client/mixins.py", line 418, in _low
    mminion=self.mminion,
  File "/usr/lib/python2.7/dist-packages/salt/utils/job.py", line 110, in store_job
    mminion.returners[fstr](load)
  File "/usr/lib/python2.7/dist-packages/salt/returners/postgres_local_cache.py", line 235, in returner
    job_ret = {'return': six.text_type(six.text_type(load['return']), 'utf-8', 'replace')}
TypeError: decoding Unicode is not supported
Traceback (most recent call last):
  File "/usr/bin/salt-run", line 10, in <module>
    salt_run()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 412, in salt_run
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/run.py", line 40, in run
    ret = runner.run()
  File "/usr/lib/python2.7/dist-packages/salt/runner.py", line 271, in run
    daemonize=False)
  File "/usr/lib/python2.7/dist-packages/salt/client/mixins.py", line 485, in _proc_function
    return self.low(fun, low, full_return=False)
  File "/usr/lib/python2.7/dist-packages/salt/client/mixins.py", line 261, in low
    return self._low(fun, low, print_event=print_event, full_return=full_return)
  File "/usr/lib/python2.7/dist-packages/salt/client/mixins.py", line 418, in _low
    mminion=self.mminion,
  File "/usr/lib/python2.7/dist-packages/salt/utils/job.py", line 110, in store_job
    mminion.returners[fstr](load)
  File "/usr/lib/python2.7/dist-packages/salt/returners/postgres_local_cache.py", line 235, in returner
    job_ret = {'return': six.text_type(six.text_type(load['return']), 'utf-8', 'replace')}
TypeError: decoding Unicode is not supported

----- RESULTS master job cache disabled ----
````

• HEAIRCONTROL
• HECAM01
• HEDEXPIVOTS
• HEDT1
• HEDT12
• HEDT13
• HEDT14
• HEDT15
• HEDT16
• HEDT17
• HEDT2
• HEDT23
• HEDT29
• HEDT3
• HEDT4
• HEDT5
• HEDT6
• HEDT7
• HEDT8
• HEIRRIGVIEW
• HEKMS
• HENAS01
• HEPG01
• HEPUMPS
• HESTILL1
• HEUNIFI
• apt-cache
• domaintest
• eset-era
• fuelview
• gitlab
• heirrig-nvr
• heshare01
• hessh
• hestillctrl
• hewiki
• irrigctl
• storagevmhost01
• taskdb
• vmhost01
• vmhost02
  ````
  --------- END RESULTS -----

Versions Report

Salt Version:
           Salt: 2018.3.0

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
      docker-py: Not Installed
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.6 (default, Nov 23 2017, 15:49:48)
   python-gnupg: 0.3.6
         PyYAML: 3.10
          PyZMQ: 14.4.0
           RAET: Not Installed
          smmap: 0.8.2
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: Ubuntu 14.04 trusty
         locale: UTF-8
        machine: x86_64
        release: 3.13.0-74-generic
         system: Linux
        version: Ubuntu 14.04 trusty

[Ch3LL]

ping @terminalmage any ideas here looks like you made this change here: c65d4714e52

[breshead]

So I dug in to the suspect area and turned the one line of stacked commands into several to figure out exactly what is going on..

It became obvious that we are "Double encoding" a string value but I have no idea why we should do such a thing.

Removing the second six.text_type() encode (And renaming tmpRET as needed) seemed to fix the issue.

Can anybody shed some more light on this as to what direction to go. Is it really that simple (don't need the double encode) or is there something deeper that I am not seeing?

           # This is the original line, expanded below..
235     #job_ret = {'return': six.text_type(six.text_type(load['return']), 'utf-8', 'replace')}
236 
237     # Encode to unicode from ascii type
238     tmpRET = six.text_type(load['return'])
239 
240     # Encode already encoded tmpRet from utf-8, replacing errors with 'U+FFFD'
241     tmpRET2 = six.text_type(tmpRET, 'utf-8', 'replace')
242     # So six.text_type gets sent to 'unicode()' in python 2 which only
243     # supports 8-bit strings as input, tmpRET is a 16 or 32 bit string
244     # which is why it is puking.
245     # Why are we "Double Encoding" this load['return'] value?
246 
247 
248     # Return the encoded/decoded string..
249     job_ret = {'return': tmpRET2}

New Error

File "/usr/lib/python2.7/dist-packages/salt/returners/postgres_local_cache.py", line 241, in returner
    tmpRET2 = six.text_type(tmpRET, 'utf-8', 'replace') 
TypeError: decoding Unicode is not supported

This puts the error on the second level of encoding, which makes sense. Again removing that second level makes the error go away and the salt-run.* begins working again.

[Ch3LL]

ping @saltstack/team-core any thoughts here? Should we be using salt.utils.stringutils.to_str() in this instance? or just removing the double encoding?

[Ch3LL]

ping @breshead would you mind pushing up as PR and we will get more eyes on this proposed fix.

[breshead]

I'll get on it as soon as I can.

[breshead]

Pull request created.

[breshead]

Updated pull request for issues raised by terminalmage. Not sure if this is a new request or updated.

[breshead]

Removed the duplicate pull request and updated the current one with the changes.