Salt: Python 3.8 Support

[s0undt3ch]

Description of Issue

Support Python 3.8

Arch Linux updated to python 3.8 and one of the tests upgrades the system, which means, while the test suite runs, we upgrade from 3.7 to 3.8, thus failing at the very end.

https://jenkinsci.saltstack.com/job/pr-kitchen-archlts-py3/job/PR-55276/4/

Current stopgap: https://github.com/saltstack/salt/pull/55309

[s0undt3ch]

Problematic code:

• [ ] [from platform import _supported_dists](https://github.com/saltstack/salt/blob/master/salt/grains/core.py#L40)

[Ch3LL]

• [ ] Need to have a running list of deprecation's which will affect Salt, for example: abc from collections in Py 3.8. Plan to have public list in GitHub ? where contributors / Salt can add to the list and generate Issues such that Salt can be updated.

[Ikke]

For Alpine, the lack of python3.8 support prevents salt-minion from running, so this is not just a test failure, but actual run-time errors:

```
$ salt-minion
Traceback (most recent call last):
[..]
File "/usr/lib/python3.8/site-packages/salt/grains/core.py", line 40, in
from platform import _supported_dists
ImportError: cannot import name '_supported_dists' from 'platform' (/usr/lib/python3.8/platform.py)

[itamarjp]

Python 3.8 will land fedora soon. (already there for Fedora 32)
https://koji.fedoraproject.org/koji/packageinfo?packageID=9781

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[Vaelatern]

This crops up after working around problems in issue #56551 specifically from platform import _supported_dists

[chalbersma]

I think this will block support for Ubuntu 20.04 as it uses python3.8 by default.

[Ch3LL]

can you try this PR https://github.com/saltstack/salt/pull/56031?

[brianthelion]

@Ch3LL The platform._supported_dists issue had not be fixed as of yesterday. See #56752

[terminalmage]

It's not just platform._supported_dists that is gone in Python 3.8, it is also platform.linux_distribution(). This code was not being maintained and now no longer exists.

The core grains already support using distro, which is the recommended replacement for platform.linux_distribution().

We may want to consider adding distro as a new hard dependency and remove from the code any usage of platform.linux_distribution(). The only problem I can see with this approach is that we may lose some fidelity when it comes to the osrelease grain, due to the differences between the two modules:

>>> import distro
>>> import platform
>>> distro.linux_distribution()
('CentOS Linux', '8', 'Core')
>>> platform.linux_distribution()
('CentOS Linux', '8.1.1911', 'Core')

[richardnpaul]

It's not just platform._supported_dists that is gone in Python 3.8, it is also platform.linux_distribution(). This code was not being maintained and now no longer exists.

The core grains already support using distro, which is the recommended replacement for platform.linux_distribution().

We may want to consider adding distro as a new hard dependency and remove from the code any usage of platform.linux_distribution(). The only problem I can see with this approach is that we may lose some fidelity when it comes to the osrelease grain, due to the differences between the two modules:

>>> import distro
>>> import platform
>>> distro.linux_distribution()
('CentOS Linux', '8', 'Core')
>>> platform.linux_distribution()
('CentOS Linux', '8.1.1911', 'Core')

You should be able to dig out the full version number. In the docs the example given is:

$ python
>>> import distro
>>> distro.linux_distribution(full_distribution_name=False)
('centos', '7.1.1503', 'Core')

The default for full_distribution_name is True which might explain your output difference to the example.

The full version number seems to be available from distro.version(best=True), the default for this is unfortunately False.

[terminalmage]

Good catch. I'm going to add some logic to the core grains to get the "best" version.

[terminalmage]

Well, first I'm going to open an upstream bug report.

[richardnpaul]

What is a pain is that changing this leads to the Distribution Name not being returned fully. I can't check, because I'm on Ubuntu, but maybe changing the version default to best=True in the distro project itself, or https://github.com/nir0s/distro/blob/d93643a066f72b1fea3864d30a407c2ee7ec7b44/distro.py#L681 adding best=True here so that linux_distribution returns the full version number by default (maybe and parameter could be added to turn this off again, like the full_distribution_name one does. They may be amenable to the change as it's maintaining the same output as platform which it's replacing.

[terminalmage]

I've opened an issue upstream and we'll see what comes of it.

What is a pain is that changing this leads to the Distribution Name not being returned fully.

How so? It looks fine to me:

>>> import platform
>>> import distro
>>> platform.linux_distribution()
('Ubuntu', '20.04', 'focal')
>>> distro.linux_distribution()
('Ubuntu', '20.04', 'focal')

[richardnpaul]

I've opened an issue upstream and we'll see what comes of it.

What is a pain is that changing this leads to the Distribution Name not being returned fully.

How so? It looks fine to me:

>>> import platform
>>> import distro
>>> platform.linux_distribution()
('Ubuntu', '20.04', 'focal')
>>> distro.linux_distribution()
('Ubuntu', '20.04', 'focal')

Sorry, I meant if you do this:
distro.linux_distribution(full_distribution_name=False)

[terminalmage]

OK, Salt doesn't do this though. It uses the full name.

[nergdron]

just ran into this problem, after upgrading some servers to the latest Ubuntu LTS (20.04). it's really surprising to me that this ticket's been open since last November and there still isn't Python 3.8 support, given that's what every major distro is moving to even in their LTS branches. is there a timeline for getting this fixed, now that it's likely to break a lot of production systems as folks upgrade?

[h0jeZvgoxFepBQ2C]

Ran into this as well.. Would be great to have this fixed soon.

[asvany]

this bug still alive on mandjaro linux.
"ImportError: cannot import name '_supported_dists' from 'platform' (/usr/lib/python3.8/platform.py)
"

[terminalmage]

@asvany _supported_dists is no longer referenced in Salt code as of the upcoming 3001 (Sodium) release, aside from in a comment.

% git show v3000:salt/grains/core.py | fgrep _supported_dists
from platform import _supported_dists
_supported_dists += ('arch', 'mageia', 'meego', 'vmware', 'bluewhite64',
# Please see os_data() and _supported_dists.
             linux_distribution(supported_dists=_supported_dists)]


% git show v3001rc1:salt/grains/core.py | fgrep _supported_dists
# Please see os_data() and _supported_dists.

[s0undt3ch]

@asvany As @terminalmage mentioned, this issue should be fixed.
What version of salt are you running?
Please try v3001rc1.

[asvany]

┌──[as@as17]─[/tmp]─[○] [p jún 12 - 12:23]
└──● virtualenv py_env_salt
created virtual environment CPython3.8.3.final.0-64 in 391ms
creator CPython3Posix(dest=/tmp/py_env_salt, clear=False, global=False)
seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/as/.local/share/virtualenv/seed-app-data/v1.0.1)
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
┌──[as@as17]─[/tmp]─[○] [p jún 12 - 12:23]
└──● . ./py_env_salt/bin/activate
(py_env_salt) ┌──[as@as17]─[/tmp]─[○] [p jún 12 - 12:24]
└──● pip install salt-ssh
Collecting salt-ssh
Downloading salt-ssh-3000.3.tar.gz (15.2 MB)
|████████████████████████████████| 15.2 MB 5.5 MB/s
ERROR: Command errored out with exit status 1:
command: /tmp/py_env_salt/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-6vboudje/salt-ssh/setup.py'"'"'; __file__='"'"'/tmp/pip-install-6vboudje/salt-ssh/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-yublgzzw
cwd: /tmp/pip-install-6vboudje/salt-ssh/
Complete output (14 lines):
Traceback (most recent call last):
File "/tmp/pip-install-6vboudje/salt-ssh/salt/version.py", line 15, in
from platform import linux_distribution as _deprecated_linux_distribution
ImportError: cannot import name 'linux_distribution' from 'platform' (/usr/lib/python3.8/platform.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-install-6vboudje/salt-ssh/setup.py", line 126, in <module>
    exec(compile(open(SALT_VERSION).read(), SALT_VERSION, 'exec'))
  File "/tmp/pip-install-6vboudje/salt-ssh/salt/version.py", line 22, in <module>
    from distro import linux_distribution
ModuleNotFoundError: No module named 'distro'
----------------------------------------

ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
(py_env_salt) ┌─[✗]──[as@as17]─[/tmp]─[○] [p jún 12 - 12:24]
└──●

[louwers]

salt-ssh doesn't seem to be working when only Python 3.8 is installed on the host.

$ sudo -E salt-ssh tgt test.ping
tgt:
    ----------
    retcode:
        10
    stderr:
    stdout:
        ERROR: Python version error. Recommendation(s) follow:
        Install Python 2.6 / Python 3 Salt dependencies on the Salt SSH master 
        to interact with Python 2.6 / Python 3 targets
$ sudo -E salt-ssh -r tgt "python --version"
tgt:
    ----------
    retcode:
        0
    stderr:
    stdout:
        Python 3.8.3

Edit: my bad, I'm running an old version of salt-ssh

$ salt-ssh --version
salt-ssh 2019.2.4 (Fluorine)

Edit2: with salt-ssh 3001.1

tgt:
    ----------
    retcode:
        1
    stderr:
        Traceback (most recent call last):
          File "/var/tmp/.root_97d892_salt/salt-call", line 27, in <module>
            salt_call()
          File "/var/tmp/.root_97d892_salt/pyall/salt/scripts.py", line 426, in salt_call
            import salt.cli.call
          File "/var/tmp/.root_97d892_salt/pyall/salt/cli/call.py", line 5, in <module>
            import salt.utils.parsers
          File "/var/tmp/.root_97d892_salt/pyall/salt/utils/parsers.py", line 27, in <module>
            import salt.config as config
          File "/var/tmp/.root_97d892_salt/pyall/salt/config/__init__.py", line 99, in <module>
            _DFLT_IPC_WBUFFER = _gather_buffer_space() * .5
          File "/var/tmp/.root_97d892_salt/pyall/salt/config/__init__.py", line 88, in _gather_buffer_space
            import salt.grains.core
          File "/var/tmp/.root_97d892_salt/pyall/salt/grains/core.py", line 40, in <module>
            from platform import _supported_dists
        ImportError: cannot import name '_supported_dists' from 'platform' (/usr/lib/python3.8/platform.py)
    stdout:

[Ch3LL]

@louwers i'm not seeing from platform import _supported_dists anywhere in the code for 3001.1. i'm guessing that is still the 2019.2.4 code on the target host. It should have updated it to 3001.1 but can you wipe that out -w and make sure you are still not seeing the issue?

[cobordism]

For the record, I am seeing the same error in qubesOS using a debian-11 template:

Error on updating debian-testing: Command '['sudo', 'qubesctl', '--skip-dom0', '--targets=debian-testing', '--show-output', 'state.sls', 'update.qubes-vm']' returned non-zero exit status 20
debian-testing:
      ----------
      _error:
          Failed to return clean data
      retcode:
          1
      stderr:
          Traceback (most recent call last):
            File "/var/tmp/.root_62a99a_salt/salt-call", line 27, in <module>
              salt_call()
            File "/var/tmp/.root_62a99a_salt/pyall/salt/scripts.py", line 440, in salt_call
              import salt.cli.call
            File "/var/tmp/.root_62a99a_salt/pyall/salt/cli/call.py", line 5, in <module>
              import salt.utils.parsers
            File "/var/tmp/.root_62a99a_salt/pyall/salt/utils/parsers.py", line 28, in <module>
              import salt.config as config
            File "/var/tmp/.root_62a99a_salt/pyall/salt/config/__init__.py", line 100, in <module>
              _DFLT_IPC_WBUFFER = _gather_buffer_space() * .5
            File "/var/tmp/.root_62a99a_salt/pyall/salt/config/__init__.py", line 89, in _gather_buffer_space
              import salt.grains.core
            File "/var/tmp/.root_62a99a_salt/pyall/salt/grains/core.py", line 40, in <module>
              from platform import _supported_dists
          ImportError: cannot import name '_supported_dists' from 'platform' (/usr/lib/python3.8/platform.py)

[s0undt3ch]

@cobordism which version of salt is that?

[s0undt3ch]

see @Ch3LL's comment above

[cobordism]

@cobordism which version of salt is that?

I'm afraid I'm not sure. It's whatever is in Qubes 4.0.3.

dom0:
salt-minion --version gives me 2017.7.1 (Nitrogen), as does salt-call --version and salt-proxy --version
Those are all the salt-* commands I can see in the dom0 system.

AppVM:
However in the fedora appvm I can see salt-ssh --version as 3000.2.

Perhaps this is an issue I have to raise with Qubes?
https://github.com/QubesOS?q=salt

EDIT:
Looks like it was fixed a few days ago:
https://github.com/QubesOS/qubes-issues/issues/6188

[s0undt3ch]

Indeed, 2017.x is no longer supported.