Salt: 2018.3.4 TypeError: unicode argument without an encoding

[gregor-tb]

Description of Issue/Question

Seems similar to other issues (#51298) involving 16 byte long strings, in our case the "master" entry on the minion has exactly 16 bytes and causes an exception.

We didn't address this problem with prior 2018.3 branch.

Steps to Reproduce Issue

Just installed fresh Debian 9.8. minimal, installed salt-minion, run salt-call grains.items (or any other command).

[ERROR   ] An un-handled exception was caught by salt's global exception handler:
TypeError: unicode argument without an encoding
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 410, in salt_call
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/call.py", line 47, in run
    caller = salt.cli.caller.Caller.factory(self.config)
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 79, in factory
    return ZeroMQCaller(opts, **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 291, in __init__
    super(ZeroMQCaller, self).__init__(opts)
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 102, in __init__
    self.minion = salt.minion.SMinion(opts)
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 806, in __init__
    lambda: self.eval_master(self.opts, failed=True)
  File "/usr/lib/python2.7/dist-packages/tornado/ioloop.py", line 457, in run_sync
    return future_cell[0].result()
  File "/usr/lib/python2.7/dist-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 285, in wrapper
    yielded = next(result)
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 694, in eval_master
    opts.update(prep_ip_port(opts))
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 265, in prep_ip_port
    host, port = parse_host_port(opts['master'])
  File "/usr/lib/python2.7/dist-packages/salt/utils/network.py", line 1956, in parse_host_port
    host_ip = ipaddress.ip_address(host).compressed
  File "/usr/lib/python2.7/dist-packages/salt/ext/ipaddress.py", line 115, in ip_address
    return IPv6Address(address)
  File "/usr/lib/python2.7/dist-packages/salt/_compat.py", line 175, in __init__
    elif self._is_packed_binary(address):
  File "/usr/lib/python2.7/dist-packages/salt/_compat.py", line 194, in _is_packed_binary
    packed = bool(int(str(bytearray(data)).encode('hex'), 16))
TypeError: unicode argument without an encoding
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 410, in salt_call
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/call.py", line 47, in run
    caller = salt.cli.caller.Caller.factory(self.config)
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 79, in factory
    return ZeroMQCaller(opts, **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 291, in __init__
    super(ZeroMQCaller, self).__init__(opts)
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 102, in __init__
    self.minion = salt.minion.SMinion(opts)
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 806, in __init__
    lambda: self.eval_master(self.opts, failed=True)
  File "/usr/lib/python2.7/dist-packages/tornado/ioloop.py", line 457, in run_sync
    return future_cell[0].result()
  File "/usr/lib/python2.7/dist-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 285, in wrapper
    yielded = next(result)
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 694, in eval_master
    opts.update(prep_ip_port(opts))
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 265, in prep_ip_port
    host, port = parse_host_port(opts['master'])
  File "/usr/lib/python2.7/dist-packages/salt/utils/network.py", line 1956, in parse_host_port
    host_ip = ipaddress.ip_address(host).compressed
  File "/usr/lib/python2.7/dist-packages/salt/ext/ipaddress.py", line 115, in ip_address
    return IPv6Address(address)
  File "/usr/lib/python2.7/dist-packages/salt/_compat.py", line 175, in __init__
    elif self._is_packed_binary(address):
  File "/usr/lib/python2.7/dist-packages/salt/_compat.py", line 194, in _is_packed_binary
    packed = bool(int(str(bytearray(data)).encode('hex'), 16))
TypeError: unicode argument without an encoding

Versions Report

2018.3.4+ds-1

[Ch3LL]

can you verify that the PR fixes it on 2018.3.4 as well?

[gregor-tb]

Sorry, I don't know what information you need from me.
I installed the latest deb from http://repo.saltstack.com/apt/debian/9/amd64/2018.3
The fix is not there.

[dawidmalina]

I have exactly the same issue with pure saltstack 2018.3.4 installation on ubuntu 18.04 with python3 (previously it was working perfectly).

@Ch3LL I can confirmed that https://github.com/saltstack/salt/pull/51298 is working and fixing this issue (I manually applied this patch).

[dawidmalina]

Fix definitely is not applied to 2018.3 branch: https://github.com/saltstack/salt/blob/2018.3/salt/_compat.py#L192

[9numbernine9]

I can also confirm that I'm seeing the exact same issue (Salt 2018.3.4, CentOS 7.5 with Python 2.7) and manually applying the patch from #51298 resolves the issue.

[Ch3LL]

thanks for confirming. i will work on getting it backported right now.

[Ch3LL]

I went ahead and backported both #51298 and #51342 since we had to revert some of the changes. Any chance anyone can re-confirm that this PR https://github.com/saltstack/salt/pull/51931 indeed fixes it since we had to roll back changes from the initial PR i asked you to test.

[9numbernine9]

@Ch3LL I applied the changes in PR #51931 to a couple of my 2018.3.4 minions and everything seemed to work just fine! :+1:

[Ch3LL]

thanks for trying that @9numbernine9 I really appreciate that :)

[patrick7]

Will this bug also be fixed in upcoming debian buster?

[tlamr]

This broke our debian stretch upgrade to debian buster as well. Minion just prints these errors into log until it fills whole disk.
This patch fixes it.

Thanks!

[Ch3LL]

thanks for all the verifications. i'll go ahead and close since its verified.

@patrick7 we do not have debian buster packages up on repo.saltstack.com but you should be able to install via pip when this fix is included in the 2018.3.4 release. Or you can patch the file.

[tlamr]

We actually also rely on debian and thanks to Benjamin Drung this was already backported to sid and is available in 2018.3.4+dfsg1-5 and newer

see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=928337

Thanks!

[jacksontj]

@Ch3LL It doesn't seem that this fix is available in a 2018.3.x release yet? Presumably it will be in https://docs.saltstack.com/en/latest/topics/releases/2018.3.5.html which is yet unreleased; any EA for release?

[Ch3LL]

ping @KChandrashekhar do we have a timeline on a 2018.3.5 release?

[floridop]

Same problem here on Ubuntu 18.04 and release 2018.3.4+ds-1 from your repositories.
For 6 years I kept 2015.x because every update sucked more and more. Now that I decided to move to a "previous" the minion fills the disks with warnings (and KILLS A SERVER).

Seriously guys, you have no quality control. I can understand a bug but not an exception handling that allows code to flood disks. I understand why there is no concept of "stable" version.

I am really surprised you made a commercial product out of this. I hope your stable commercial stuff don't do the same with your paying customers....

So can you give me a suggestion for some release that just works or I have to move to ansible? I'd stick to my old 2015 but it has been out of support for quite long time now.

I think this should be reopened.

[Tanoti]

Please can we get an update on when 2018.3.5 will be released?

[floridop]

FYI for other people struggling with this, I had to switch all the infrastructure to 2017.7 latest, which seems stable enough now, although different version of 2017.7 wreaked havoc within 2015.x and 2017.x minions (different versions), some started filling logs saying that they could not recognize the master. It took few minions update to latest figure out the issue was in the master which I had to update to the latest 2017 as well. It's kind of a mess but looks stable now.

Btw better like this since with 2018.x I had problems with some of the conditions in my states, lots of stuff was not picked up, I guess changes in Jinja parsing. Maybe there is a bug or release note about that as well. Maybe in 5 years from now when 2018 will be ok I'll take a look and report in more detail...

My machines are Ubuntu 12.04, 14.04, 16.04, 18.04 . master runs on a 18.04 now. Other minions are on Centos6/7, versions 2015.x and 2017.x and it all looks stable right now. Which is nice with such variation. You guys are good when you don't behave like python.

[KChandrashekhar]

Please can we get an update on when 2018.3.5 will be released?

Will get the approximate 2018.3.5 release dates by Tuesday (4th June)

[KChandrashekhar]

@Tanoti @jacksontj Currently we are working on 2019.2.1 which is estimated for July. Neon is estimated to come out in Aug-sept. 2018.3.5 is estimated to be released in Oct-Nov of 2019.

[tlamr]

❤️ Debian Buster fixed https://tracker.debian.org/news/1040463/salt-201834dfsg1-6-migrated-to-testing/

[Tanoti]

In case anyone comes here looking for answers, the fix is not in 2018.3.5.