Salt: [2018.3.4] jinja filter ipv6 doesn't work with Python 2+

[b-a-t]

Description of Issue/Question

It looks like in the combination of Python 2.7+ and Salt 2018.3.4 the Jinja filter ipv6 doesn't recognize any IPv6 networks/ranges as valid IPv6 ones. Not even ::/0.

I think the problem is deeper and affects not only Jinja filters, but overall processing and handling of IPv6 network and interface addresses.

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)
This is a simple test case, that exposes the problem:

{% if '2001:67c:2e8::/48' | ipv6 -%}
This is an IPv6 range!
{%- else -%}
Not an IPv6 range!
{%- endif -%}

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)
Putting the snippet above into the file test.sls and running:

# salt-call --output=yaml --output-file=/dev/null cp.get_template /home/user/test.sls /dev/stdout

outputs:

Not an IPv6 range!

On the similar boxes with 2018.3.3 minions the same test gives expected This is an IPv6 range!.

Versions Report

(Provided by running salt-call --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
Salt: 2018.3.4

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 1.5
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.5 (default, Apr 9 2019, 14:30:50)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.3.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: centos 7.6.1810 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-957.5.1.el7.x86_64
system: Linux
version: CentOS Linux 7.6.1810 Core

This is on a minion, master is on 2018.3.3.

[b-a-t]

The problem is exposed by this simple Python code:

#!/usr/bin/python2.7
#
from salt._compat import ipaddress

addr = '2001:67c:2e8::/48'

print("Address: %s" % type(addr))
print("str is bytes: %s" % issubclass(str, bytes))

ip6 = ipaddress.IPv6Interface(addr)

print(ip6)

which fails with the given error:

Address: <type 'str'>
str is bytes: True
Traceback (most recent call last):
  File "./test.py", line 10, in <module>
    ip6 = ipaddress.IPv6Interface(addr)
  File "/usr/lib/python2.7/site-packages/salt/_compat.py", line 220, in __init__
    IPv6AddressScoped.__init__(self, address)
  File "/usr/lib/python2.7/site-packages/salt/_compat.py", line 163, in __init__
    ipaddress._BaseAddress.__init__(self, address)
  File "/usr/lib/python2.7/site-packages/salt/ext/ipaddress.py", line 645, in __init__
    raise AddressValueError("Unexpected '/' in %r" % address)
salt.ext.ipaddress.AddressValueError: Unexpected '/' in '2001:67c:2e8::/48'

This is how Jinja filter fails to recognize IPv6 address at the end.

The problem seems to be in this code snippet in _compat.py:

class IPv6InterfaceScoped(ipaddress.IPv6Interface, IPv6AddressScoped):
    '''
    Update
    '''
    def __init__(self, address):
=>      if isinstance(address, (bytes, int)):
            IPv6AddressScoped.__init__(self, address)
            self.network = ipaddress.IPv6Network(self._ip)
            self._prefixlen = self._max_prefixlen
            return

The line if isinstance(address, (bytes, int)): tries to perform special processing for the byte packed address, but due the fact that str is a subclass of bytes it gets false positive when a string with IPv6 address/network is passed.

[b-a-t]

It looks like cefb16b5c2676e6c940031e1ce093061cfac705d addresses exactly this issue.

[b-a-t]

Well, hoping that the commit above fixed it in develop branch at least I've tested on a fresh checkout today:

$ git status
On branch develop
Your branch is up to date with 'origin/develop'.

$ python2.7 test.py
Address: <type 'str'>
str is bytes: True
Traceback (most recent call last):
  File "test.py", line 10, in <module>
    ip6 = ipaddress.IPv6Interface(addr)
  File "/Users/b-a-t/git/salt-stack/saltstack/salt/_compat.py", line 219, in __init__
    IPv6AddressScoped.__init__(self, address)
  File "/Users/b-a-t/git/salt-stack/saltstack/salt/_compat.py", line 162, in __init__
    ipaddress._BaseAddress.__init__(self, address)
  File "/Users/b-a-t/git/salt-stack/saltstack/salt/ext/ipaddress.py", line 645, in __init__
    raise AddressValueError("Unexpected '/' in %r" % address)
salt.ext.ipaddress.AddressValueError: Unexpected '/' in '2001:67c:2e8::/48'

$ python3.7 test.py
Address: <class 'str'>
str is bytes: False
2001:67c:2e8::/48

So the problem is still there.

[Ch3LL]

looks like i'm able to replicate this on 2018.3.4 and the head of 2018.3. It was working on 2018.3.3. We will need to get this fixed thanks.

[garethgreenaway]

I think I found the cause and a potential fix for this. Need to do bit more testing tomorrow to confirm the change doesn't break anything.

[b-a-t]

Well, 2019.2.1 is out and 2019.2.2 is on the way. Still the said problem not fixed, despite being blocker and a fix being provided. We still can't switch to the newer Salt, as we are heavily dependent on IPv6 in our environment.

$ git status
On branch master
Your branch is up to date with 'origin/master'.
$ python2.7 test.py
Address: <type 'str'>
str is bytes: True
Traceback (most recent call last):
  File "test.py", line 10, in <module>
    ip6 = ipaddress.IPv6Interface(addr)
  File "/Users/b-a-t/git/salt-stack/saltstack/salt/_compat.py", line 219, in __init__
    IPv6AddressScoped.__init__(self, address)
  File "/Users/b-a-t/git/salt-stack/saltstack/salt/_compat.py", line 162, in __init__
    ipaddress._BaseAddress.__init__(self, address)
  File "/Users/b-a-t/git/salt-stack/saltstack/salt/ext/ipaddress.py", line 645, in __init__
    raise AddressValueError("Unexpected '/' in %r" % address)
salt.ext.ipaddress.AddressValueError: Unexpected '/' in '2001:67c:2e8::/48'