Salt: 2019.2.1/2019.2.0 pip failures even when not using pip

[Reiner030]

While testing my setup with Debian Buster and "stable" Saltstack version (2019.2.0) on a vagrant instance I run into weird problems...
Especially i didn't use pip and most of the other modules notified in the failure messages I was search for over half hour for the cause:

````

salt-call state.apply

[ERROR ] Failed to import states pip_state, this is due most likely to a syntax error:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
mod = imp.load_module(mod_namespace, fn_, fpath, desc)
File "/usr/lib/python2.7/dist-packages/salt/states/pip_state.py", line 58, in
del pip
NameError: name 'pip' is not defined
[ERROR ] Failed to import states pkg, this is due most likely to a syntax error:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
mod = imp.load_module(mod_namespace, fn_, fpath, desc)
File "/usr/lib/python2.7/dist-packages/salt/states/pkg.py", line 79, in
import logging
File "/usr/lib/python2.7/logging/__init__.py", line 26, in
import sys, os, time, cStringIO, traceback, warnings, weakref, collections
File "/usr/lib/python2.7/os.py", line 119, in
sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR ] Failed to import states pkgbuild, this is due most likely to a syntax error:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
mod = imp.load_module(mod_namespace, fn_, fpath, desc)
File "/usr/lib/python2.7/dist-packages/salt/states/pkgbuild.py", line 48, in
import logging
File "/usr/lib/python2.7/logging/__init__.py", line 26, in
import sys, os, time, cStringIO, traceback, warnings, weakref, collections
File "/usr/lib/python2.7/os.py", line 119, in
sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR ] Failed to import states pkgrepo, this is due most likely to a syntax error:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
mod = imp.load_module(mod_namespace, fn_, fpath, desc)
File "/usr/lib/python2.7/dist-packages/salt/states/pkgrepo.py", line 93, in
from salt.exceptions import CommandExecutionError, SaltInvocationError
File "/usr/lib/python2.7/dist-packages/salt/exceptions.py", line 9, in
import logging
File "/usr/lib/python2.7/logging/__init__.py", line 26, in
import sys, os, time, cStringIO, traceback, warnings, weakref, collections
File "/usr/lib/python2.7/os.py", line 119, in
sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR ] Exception raised when processing __virtual__ function for salt.loaded.int.states.portage_config. Module will not be loaded: 'NoneType' object is not callable
[WARNING ] salt.loaded.int.states.portage_config.__virtual__() is wrongly returning None. It should either return True, False or a new name. If you're the developer of the module 'portage_config', please fix this.
[ERROR ] Failed to import states ports, this is due most likely to a syntax error:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
mod = imp.load_module(mod_namespace, fn_, fpath, desc)
File "/usr/lib/python2.7/dist-packages/salt/states/ports.py", line 21, in
import logging
File "/usr/lib/python2.7/logging/__init__.py", line 26, in
import sys, os, time, cStringIO, traceback, warnings, weakref, collections
File "/usr/lib/python2.7/os.py", line 119, in
sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR ] Exception raised when processing __virtual__ function for salt.loaded.int.states.postgres_cluster. Module will not be loaded: 'NoneType' object is not callable
[WARNING ] salt.loaded.int.states.postgres_cluster.__virtual__() is wrongly returning None. It should either return True, False or a new name. If you're the developer of the module 'postgres_cluster', please fix this.
[ERROR ] Exception raised when processing __virtual__ function for salt.loaded.int.states.postgres_database. Module will not be loaded: 'NoneType' object is not callable
[WARNING ] salt.loaded.int.states.postgres_database.__virtual__() is wrongly returning None. It should either return True, False or a new name. If you're the developer of the module 'postgres_database', please fix this.
[ERROR ] Failed to import states postgres_extension, this is due most likely to a syntax error:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
mod = imp.load_module(mod_namespace, fn_, fpath, desc)
File "/usr/lib/python2.7/dist-packages/salt/states/postgres_extension.py", line 19, in
import logging
File "/usr/lib/python2.7/logging/__init__.py", line 26, in
import sys, os, time, cStringIO, traceback, warnings, weakref, collections
File "/usr/lib/python2.7/os.py", line 119, in
sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR ] Failed to import states postgres_group, this is due most likely to a syntax error:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
mod = imp.load_module(mod_namespace, fn_, fpath, desc)
File "/usr/lib/python2.7/dist-packages/salt/states/postgres_group.py", line 18, in
import logging
File "/usr/lib/python2.7/logging/__init__.py", line 26, in
import sys, os, time, cStringIO, traceback, warnings, weakref, collections
File "/usr/lib/python2.7/os.py", line 119, in
sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR ] Exception raised when processing __virtual__ function for salt.loaded.int.states.postgres_initdb. Module will not be loaded: 'NoneType' object is not callable
[WARNING ] salt.loaded.int.states.postgres_initdb.__virtual__() is wrongly returning None. It should either return True, False or a new name. If you're the developer of the module 'postgres_initdb', please fix this.
[ERROR ] Exception raised when processing __virtual__ function for salt.loaded.int.states.postgres_language. Module will not be loaded: 'NoneType' object is not callable
[WARNING ] salt.loaded.int.states.postgres_language.__virtual__() is wrongly returning None. It should either return True, False or a new name. If you're the developer of the module 'postgres_language', please fix this.
[ERROR ] Exception raised when processing __virtual__ function for salt.loaded.int.states.postgres_privileges. Module will not be loaded: 'NoneType' object is not callable
[WARNING ] salt.loaded.int.states.postgres_privileges.__virtual__() is wrongly returning None. It should either return True, False or a new name. If you're the developer of the module 'postgres_privileges', please fix this.
[ERROR ] Failed to import states postgres_schema, this is due most likely to a syntax error:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
mod = imp.load_module(mod_namespace, fn_, fpath, desc)
File "/usr/lib/python2.7/dist-packages/salt/states/postgres_schema.py", line 16, in
import logging
File "/usr/lib/python2.7/logging/__init__.py", line 26, in
import sys, os, time, cStringIO, traceback, warnings, weakref, collections
File "/usr/lib/python2.7/os.py", line 119, in
sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR ] Failed to import states postgres_tablespace, this is due most likely to a syntax error:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
mod = imp.load_module(mod_namespace, fn_, fpath, desc)
File "/usr/lib/python2.7/dist-packages/salt/states/postgres_tablespace.py", line 23, in
import salt.utils.dictupdate as dictupdate
File "/usr/lib/python2.7/dist-packages/salt/utils/__init__.py", line 18, in
from salt.ext import six
File "/usr/lib/python2.7/dist-packages/salt/ext/six.py", line 728, in
print_ = getattr(moves.builtins, "print", None)
File "/usr/lib/python2.7/dist-packages/salt/ext/six.py", line 99, in __get__
result = self._resolve()
File "/usr/lib/python2.7/dist-packages/salt/ext/six.py", line 122, in _resolve
return _import_module(self.mod)
File "/usr/lib/python2.7/dist-packages/salt/ext/six.py", line 90, in _import_module
return sys.modules[name]
AttributeError: 'module' object has no attribute 'modules'
[ERROR ] Failed to import states postgres_user, this is due most likely to a syntax error:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
mod = imp.load_module(mod_namespace, fn_, fpath, desc)
File "/usr/lib/python2.7/dist-packages/salt/states/postgres_user.py", line 17, in
import logging
File "/usr/lib/python2.7/logging/__init__.py", line 26, in
import sys, os, time, cStringIO, traceback, warnings, weakref, collections
File "/usr/lib/python2.7/os.py", line 119, in
sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'

Passed invalid arguments: 'NoneType' object is not callable.

Usage:

.. versionadded:: 2015.5.0

This function will call :mod:`state.highstate
<salt.modules.state.highstate>` or :mod:`state.sls
<salt.modules.state.sls>` based on the arguments passed to this function.
It exists as a more intuitive way of applying states.

.. rubric:: APPLYING ALL STATES CONFIGURED IN TOP.SLS (A.K.A. :ref:`HIGHSTATE <running-highstate>`)

To apply all configured states, simply run ``state.apply``:

.. code-block:: bash

    salt '*' state.apply

The following additional arguments are also accepted when applying all
states configured in top.sls:

test
    Run states in test-only (dry-run) mode

mock
    The mock option allows for the state run to execute without actually
    calling any states. This then returns a mocked return which will show
    the requisite ordering as well as fully validate the state run.

    .. versionadded:: 2015.8.4

pillar
    Custom Pillar values, passed as a dictionary of key-value pairs

    .. code-block:: bash

        salt '*' state.apply stuff pillar='{"foo": "bar"}'

    .. note::
        Values passed this way will override Pillar values set via
        ``pillar_roots`` or an external Pillar source.

exclude
    Exclude specific states from execution. Accepts a list of sls names, a
    comma-separated string of sls names, or a list of dictionaries
    containing ``sls`` or ``id`` keys. Glob-patterns may be used to match
    multiple states.

    .. code-block:: bash

        salt '*' state.apply exclude=bar,baz
        salt '*' state.apply exclude=foo*
        salt '*' state.apply exclude="[{'id': 'id_to_exclude'}, {'sls': 'sls_to_exclude'}]"

queue : False
    Instead of failing immediately when another state run is in progress,
    queue the new state run to begin running once the other has finished.

    This option starts a new thread for each queued state run, so use this
    option sparingly.

localconfig
    Optionally, instead of using the minion config, load minion opts from
    the file specified by this argument, and then merge them with the
    options from the minion config. This functionality allows for specific
    states to be run with their own custom minion configuration, including
    different pillars, file_roots, etc.

    .. code-block:: bash

        salt '*' state.apply localconfig=/path/to/minion.yml


.. rubric:: APPLYING INDIVIDUAL SLS FILES (A.K.A. :py:func:`STATE.SLS <salt.modules.state.sls>`)

To apply individual SLS files, pass them as a comma-separated list:

.. code-block:: bash

    # Run the states configured in salt://stuff.sls (or salt://stuff/init.sls)
    salt '*' state.apply stuff
    # Run the states configured in salt://stuff.sls (or salt://stuff/init.sls)
    # and salt://pkgs.sls (or salt://pkgs/init.sls).
    salt '*' state.apply stuff,pkgs

The following additional arguments are also accepted when applying
individual SLS files:

test
    Run states in test-only (dry-run) mode

mock
    The mock option allows for the state run to execute without actually
    calling any states. This then returns a mocked return which will show
    the requisite ordering as well as fully validate the state run.

    .. versionadded:: 2015.8.4

pillar
    Custom Pillar values, passed as a dictionary of key-value pairs

    .. code-block:: bash

        salt '*' state.apply stuff pillar='{"foo": "bar"}'

    .. note::
        Values passed this way will override Pillar values set via
        ``pillar_roots`` or an external Pillar source.

queue : False
    Instead of failing immediately when another state run is in progress,
    queue the new state run to begin running once the other has finished.

    This option starts a new thread for each queued state run, so use this
    option sparingly.

concurrent : False
    Execute state runs concurrently instead of serially

    .. warning::

        This flag is potentially dangerous. It is designed for use when
        multiple state runs can safely be run at the same time. Do *not*
        use this flag for performance optimization.

saltenv
    Specify a salt fileserver environment to be used when applying states

    .. versionchanged:: 0.17.0
        Argument name changed from ``env`` to ``saltenv``

    .. versionchanged:: 2014.7.0
        If no saltenv is specified, the minion config will be checked for an
        ``environment`` parameter and if found, it will be used. If none is
        found, ``base`` will be used. In prior releases, the minion config
        was not checked and ``base`` would always be assumed when the
        saltenv was not explicitly set.

pillarenv
    Specify a Pillar environment to be used when applying states. This
    can also be set in the minion config file using the
    :conf_minion:`pillarenv` option. When neither the
    :conf_minion:`pillarenv` minion config option nor this CLI argument is
    used, all Pillar environments will be merged together.

localconfig
    Optionally, instead of using the minion config, load minion opts from
    the file specified by this argument, and then merge them with the
    options from the minion config. This functionality allows for specific
    states to be run with their own custom minion configuration, including
    different pillars, file_roots, etc.

    .. code-block:: bash

        salt '*' state.apply stuff localconfig=/path/to/minion.yml

sync_mods
    If specified, the desired custom module types will be synced prior to
    running the SLS files:

    .. code-block:: bash

        salt '*' state.apply stuff sync_mods=states,modules
        salt '*' state.apply stuff sync_mods=all

    .. note::
        This option is ignored when no SLS files are specified, as a
        :ref:`highstate <running-highstate>` automatically syncs all custom
        module types.

    .. versionadded:: 2017.7.8,2018.3.3,2019.2.0

````

After wasting my nightly time I found luckily that the system has installed a new version which is officially not yet relased: 2019.2.1

And thx to the autoupdate function now also my other instances where influenced and I have to manually downgrade dist to

deb https://repo.saltstack.com/apt/debian/9/amd64/archive/2019.2.0 stretch main

Interesting also that my common vagrant setup has setup the line with stretch and not buster, but this seems because you reuse stretch packages for buster ?

AFTER 10 MINUTES of REPAIRING EACH INSTANCE caused by addition broken package dependencies I can run my states again as usual:

````

salt-call state.apply

[WARNING ] /usr/lib/python2.7/dist-packages/salt/modules/mysql.py:546: Warning: (1681L, "'PASSWORD' is deprecated and will be removed in a future release.")
return cur.execute(qry, args)

local:

Summary for local

Succeeded: 140

Failed: 0

Total states run: 140
Total run time: 9.274 s
````

ah... now documentation shows the buggy version as released...

[garethgreenaway]

@Reiner030 thanks for reporting this. Would you be able to provide the state file in question that you were trying to run when this error happened? Thanks!

[tomlaredo]

I'm using SaltStack for some years now and I must admit that almost EVERY release comes with important and obvious regressions showing a bunch of errors.

Yes code quality control isn't a simple thing to deal with, yes SaltStack has a lot of modules witten by many people but what could be worse than wating for months (sometimes years) that a good PR is merged, packaged and deployed on one hand and then having bad troubles every time we update SaltStack on the other?

I know that this comment is a bit rude but please understand that this is very frustrating.

https://github.com/saltstack/salt/issues/34096

[r-lindner]

this bug was likely introduced with 238fd0f071fbbc2e1efbe6107d68df4c33133ea0
The statefile is not relevant - it crashes if pip is not installed making it a grave bug:
It happens when the apache formula is used but not if the nginx formula is used, we are looking for a better test case.

salt/states/pip_state.py

try:
....import pip
....HAS_PIP = True
except ImportError:
....HAS_PIP = False
....# [some more lines on this level]
....del pip # pip is not imported, cannot be deleted!

[robmp]

I'm hitting this issue with states using unless: if that helps to narrow it down.

[maennlse]

we also traced it down to that states using unless: are failing...

the following state is working:

vim:
  pkg.installed:
    - name: vim-enhanced

but it fails with as soon as unless: is added:

vim:
  pkg.installed:
    - name: vim-enhanced
    - unless: /bin/bash -c false

[OrlandoArcapix]

A similar issue also affects anyone using pip between versions 10.0 and 18.1, due to this:

    if salt.utils.versions.compare(ver1=pip.__version__,
                                   oper='>=',
                                   ver2='18.1'):
        from pip._internal.exceptions import InstallationError

The exceptions module was moved from pip.exceptions to pip._internal.exceptions in pip version 10.0 - so the 18.1 above should probably read 10.0.

See: https://github.com/saltstack/salt/pull/54772

[stratusjerry]

Also seeing the issue when targeting state using "Unless", as well as with "OnlyIf" logic. However the root cause seems to be related to changes to the pip_state in 2019.2.1 See issue:

53570

[maennlse]

I can confirm that installing python2-pip (on centos 7) fixes the issue with unless: or onlyif: used...

[smitelli]

Similar to @maennlse, this is fixed on Debian Buster by installing the python-pip package before running salt-call, etc. (https://packages.debian.org/buster/python-pip)

[mamorunl]

I can also confirm that it works on Ubuntu 16.04 when installing python-pip first.

[max-arnold]

Related to #53570

[tj90241]

+1 to this happening when using states with onlyif and unless requisites.

Furthermore, even AFTER installing python-pip, something feels wrong... a state as simple as below takes ~10sec (!!!) to execute whereas it takes ~150ms to execute without the requisites.

try-this-without-requisites:
  module.run:
    - name: test.ping
    - onlyif:
      - /bin/true

[the-glu]

Hello,

So this is patched in #54826 and merged, do you have any idea when the fix will be available in packages ? Do you plan to release a new release soon ?

The bug is blocking on our side (we need it the latest version for debian 10 ^^), installing pip is not really a good solution and we would like to know how long we should expect to wait :)

Thanks!

[arizvisa]

Maintainers. Just some advice...

If you'd start vendoring the Python modules that salt-stack's modules depend on instead of sharing modules with Python's site-packages and having to suffer the mercy of version skew, the majority of these types of problems will go away until a dever gets a chance to actually look at it. Something like python-vendorize would be worth researching in order to improve saltstack's stability.

You can't develop a complex application such as this against a variety of moving targets. Perl's CPAN has taught this to us Perl developers the hard way. This is not an uncommon pattern either, as golang and rust have solutions for dealing with these problems as well.

[tj90241]

@arizvisa the issue here is that the code was being tested in an environment with a package installed that was not marked as a dep for installation. It just so happened that there was a bug if and when that module was NOT installed.

There was another issue pertaining to proxy minions that was missed because their testing harness was testing against the checked out repo rather that the sdist that's bundled for pypi.

I don't think either of these problems would have been solved by vendorized modules. OTOH, the team would benefit from creating the sdist and testing against that in a clean-room environment.

[arizvisa]

Actually, the first issue _would_ actually be solved by vendorizing said module. Actually it would've removed the need for the breaking-commit even. Bundling the deps that you're deving against would also mean that the second case would never happen. Without bundling module versions, your testers essentially need to test how things work with all different versions and without said core module in order to test comprehensively. This can be unreasonable for testers despite potentially being rockstars.

Another benefit is that it'll fix that need to try-except-import at the beginning of all modules in order to test whether salt's able to import a core module or not. Actually thinking about it, that's probably one aspect of the performance issue that salt seems to have, and salt is just riddled with these types of handlers.

Writing a quick testcase that is similar to the try-except that each core module wraps their import dependent-module header with looks like the following. f1() would normally import something like seco.range for example.

import sys, time
def f1():
    try:
        # Imagine an import pip being here
        raise Exception
    except:
        pass
    return

def f2():
    try:
    return

f = f1 if sys.argv[1] == '1' else f2
t = time.time()
for x in range(4096*1024):
    f()
print(time.time() - t)

This code when calling f2() which simply does nothing has pretty good performance because well...it's hitting the hot-path. Even if it creates an exception handler, as long as it hits the hot path performance will be minimal.

[2962] user@phatty ~/audit/acrobat-fuzz/lol/deploy$ python ~/f.py
0.522000074387
[2963] user@phatty ~/audit/acrobat-fuzz/lol/deploy$ python3 ~/f.py
0.357666015625

Now when calling f1() which is when an exception is raised and caught which will happen at the beginning of any-and-all lazyloaded modules.

[2964] user@phatty ~/audit/acrobat-fuzz/lol/deploy$ python ~/f.py
4.42999982834
[2965] user@phatty ~/audit/acrobat-fuzz/lol/deploy$ python3 ~/f.py
1.3732171058654785

Yea, that's 4.4 seconds vs 0.5 seconds for python2, and 1.3 vs 0.3 seconds for python3.. This is an exaggerated number of cases of course, but literally every single core module in salt checks for a likely non-existent module, _especially_ during matching.

So if there's a dependent-module that's not found, that try-except case is hit _all-the-time_ which has super poor performance because of the cost of dispatching an exception (course this is in a hundred other places in salt). If you were using reload() (despite it being janky) this'd be fine because things like HAS_PIP would still be cached and there'd be no need to check whether the import raised an exception.

But since you use imp.load_module, each exception in the module has to be dispatched if the dependent-module does not exist. Not just that but specifically to everytime a module is imported, that imp.load_module has to re-compile the entire module which can potentially take even longer.

On another note, I sympathize for you guys because everytime you hit a release you can tell that you're in crunchtime because there's lapses in the responses to bug reports. If this has been like this since 2016 as noted by tomlaredo, you should consider putting things on pause while you profile and refactor as long as your "paying" customers allow for it.

[tj90241]

Vendorizing it fixes it by making it a component of the release... which comes with other disadvantages (drift from upstream out side of the release, which is a double-edged sword... added memory consumption for those who formerly did not have or use pip come to mind). Unsure how this approach would handle modules like pyzmq which have C extensions too...

[marek-obuchowicz]

Vendorizing might (or might not) be a long term strategy to manage dependencies. Right now, @rallytime please help us with releasing this fix as soon as possible, this makes latest saltstack release (again!) completely broken for some cases.

[arizvisa]

At the very least, things won't break until a developer finally gets a chance to look at it and test it against the latest version (instead of having to check against all possible versions for _every_ release). You also get the benefit in that you can ask your community to synchronize the versions if you're busy since it's typically an easy PR.

Don't forget that you guys aren't the only one that works on this. Because of saltstack's unfortunate lack of robustness, each of us are essentially required to be Python programmers.

[cskowronnek]

I use salt-cloud to create VMs on Azure, then salt-minion is bootstrapped on the machine by salt-cloud and in the end state.apply rolls out config.
How am I supposed to install python-pip before applying a state when I want a fully automated rollout

[max-arnold]

@cskowronnek You can use the script_args option: https://docs.saltstack.com/en/latest/topics/cloud/deploy.html#deploy-script-arguments

The -p argument allows you to install any extra packages during the bootstrap process: https://github.com/saltstack/salt-bootstrap/blob/76de44695323b8d70e6bd3b98968de262241a5eb/bootstrap-salt.sh#L353-L354

Another option is to pin Salt to a previous stable version (e.g., stable 2019.2.0): https://github.com/saltstack/salt-bootstrap/blob/76de44695323b8d70e6bd3b98968de262241a5eb/bootstrap-salt.sh#L303

[cskowronnek]

@max-arnold Thank your the fast help.

script_args: -p python-pip did the trick!

[dynek]

so you guys decided to remove existing package from the repository instead of releasing a version including the fix? :-)

https://repo.saltstack.com/py3/debian/10/amd64/

[tomlaredo]

Is there any news on this one?
Installing pip on each server isn't an acceptable solution.

[CaptainSanders]

If you've installed salt from a repo I recommend downgrading to 2019.2.0. That worked for me.

[tomlaredo]

Yes, I edited my previous post as we're not looking for a workaround but a real solution.
Thanks anyway.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[TheJJ]

bad bot!

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[OrlandoArcapix]

I think that this was fixed in 2019.2.2 - in https://github.com/saltstack/salt/pull/54826 and https://github.com/saltstack/salt/pull/54807

[arizvisa]

Oh, lol. Looky here, another issue that could've been fixed by vendorizing module deps: #54392

[jdrowne]

We found a similar issue by using 'latest'. A lot of packages were not being installed at all so we had to downgrade to 2019.2 per @CaptainSanders https://github.com/saltstack/salt/issues/54755#issuecomment-544857544

Here's the error for us:

Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1607, in _load_module
mod = imp.load_module(mod_namespace, fn_, fpath, desc)
File "/usr/lib/python2.7/dist-packages/salt/states/pip_state.py", line 110, in
del pip
NameError: name 'pip' is not defined
2020-02-14 12:51:29,334 [salt.loader :1624][ERROR ][7182] Failed to import states pkg, this is due most likely to a syntax error:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1607, in _load_module
mod = imp.load_module(mod_namespace, fn_, fpath, desc)
File "/usr/lib/python2.7/dist-packages/salt/states/pkg.py", line 77, in
import logging
File "/usr/lib/python2.7/logging/__init__.py", line 26, in
import sys, os, time, cStringIO, traceback, warnings, weakref, collections
File "/usr/lib/python2.7/os.py", line 119, in
sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
2020-02-14 12:51:29,343 [salt.loader :1624][ERROR ][7182] Failed to import states pkgbuild, this is due most likely to a syntax error:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1607, in _load_module
mod = imp.load_module(mod_namespace, fn_, fpath, desc)
File "/usr/lib/python2.7/dist-packages/salt/states/pkgbuild.py", line 48, in
import logging
File "/usr/lib/python2.7/logging/__init__.py", line 26, in
import sys, os, time, cStringIO, traceback, warnings, weakref, collections
File "/usr/lib/python2.7/os.py", line 119, in
sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'

[sagetherage]

Not sure why the stale bot is attacking this issue, but attempting to rid the ticket of the enemy evil doer with Confirmed and moving this out of Blocked into Approved will review again later.

[CaptainSanders]

Since this issue re-emerged in 3000.1 can we keep it open?

[PeterS242]

Since this issue re-emerged in 3000.1 can we keep it open?

Agreed. We're having to fix this on our RHEL6 and RHEL7 minions currently. salt-3000-1.el6 and salt-3000-1.el7 .