Salt: Salt v3000 crashes with unless clause if pip is missing

Created on 11 Feb 2020 · 13Comments · Source: saltstack/salt

Description of Issue

archive.extracted (and possibly others?) fails if setting an unless clause and pip is not installed. Works on 2019.2.3.

Setup

# archive-test
vault:
    archive.extracted:
        - name: /usr/local/bin/
        - source: https://releases.hashicorp.com/vault/1.3.2/vault_1.3.2_linux_amd64.zip
        - source_hash: sha256=b49de8fd508eb9c2c222fa0f38e23546fb28991af2e8bfdb9bbe381a380f9baa
        - archive_format: zip
        - enforce_toplevel: False
        - unless:
            - echo hello

Steps to Reproduce Issue

salt-call state.sls archive-test without pip installed. Removing the unless clause or installing python-pip resolves the issue.

Versions Report

Masterless.

Salt Version:
           Salt: 3000

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.13 (default, Sep 26 2018, 18:42:22)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.1

System Versions:
           dist: debian 9.4
         locale: UTF-8
        machine: x86_64
        release: 4.9.0-6-amd64
         system: Linux
        version: debian 9.4

There's a very long stacktrace when it crashes, this is the beginning and end:

[ERROR   ] Failed to import states pip_state, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1607, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/pip_state.py", line 110, in <module>
    del pip
NameError: name 'pip' is not defined
[ERROR   ] Failed to import states pkg, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1607, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/pkg.py", line 77, in <module>
    import logging
  File "/usr/lib/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref, collections
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR   ] Failed to import states pkgbuild, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1607, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/pkgbuild.py", line 48, in <module>
    import logging
  File "/usr/lib/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref, collections
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
...

[ERROR   ] Failed to import states zpool, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1607, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/zpool.py", line 75, in <module>
    import os
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
KeyError: 'salt.loaded.int.module.state'
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    load_entry_point('salt==3000', 'console_scripts', 'salt-call')()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 445, in salt_call
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/call.py", line 57, in run
    caller.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 119, in run
    ret = self.call()
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 232, in call
    func.__module__].__context__.get('retcode', 0)
KeyError: 'salt.loaded.int.module.state'
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    load_entry_point('salt==3000', 'console_scripts', 'salt-call')()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 445, in salt_call
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/call.py", line 57, in run
    caller.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 119, in run
    ret = self.call()
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 232, in call
    func.__module__].__context__.get('retcode', 0)
KeyError: 'salt.loaded.int.module.state'

Bug Confirmed Core P3 Regression State Compiler severity-medium v3000.1

Source

thusoy

👍3

Most helpful comment

It looks like the old problem manifested itself in a different way: https://github.com/saltstack/salt/issues/53570

Below is a quick fix:

patches/pip.sls:

{% if grains.saltversion == "3000" %}
patch:
  pkg.installed

pip_state_patch:
  file.patch:
    - name: '{{ grains.saltpath }}'
    - source: salt://patches/files/3000-pip.diff
    - strip: 2
    - require:
        - pkg: patch

restart_salt_minion:
  cmd.run:
    - name: 'salt-call service.restart salt-minion'
    - bg: true
    - onchanges:
      - file: pip_state_patch
{%- endif %}

patches/files/3000-pip.diff:

diff --git a/salt/states/pip_state.py b/salt/states/pip_state.py
index 0f762752d0..97a9feeb48 100644
--- a/salt/states/pip_state.py
+++ b/salt/states/pip_state.py
@@ -100,14 +100,13 @@ except ImportError:
     import sys
     pip_related_entries = [
         (k, v) for (k, v) in sys.modules.items()
-        or getattr(v, '__module__', '').startswith('pip.')
+        if getattr(v, '__module__', '').startswith('pip.')
         or (isinstance(v, types.ModuleType) and v.__name__.startswith('pip.'))
     ]
     for name, entry in pip_related_entries:
         sys.modules.pop(name)
         del entry

-    del pip
     sys_modules_pip = sys.modules.pop('pip', None)
     if sys_modules_pip is not None:
         del sys_modules_pip

Run sudo MINION state.apply patches.pip to apply the fix.

Another option is to override the pip_state.py module. However, the process is more complex.

max-arnold on 12 Feb 2020

👍3

All 13 comments

FWIW I'm seeing the same issue on Debian Buster (10).

thusoy on 11 Feb 2020

I feel lucky to not use Saltstack (or the combo Salt and Debian) in a production environment. Pretty much each time there's a release since I started using it there are such problems.
So there is my +1 for Debian 10.3 and Salt 3000+ds-1 (also happens on an old Debian 8.11 that I'm migrating)

dynek on 12 Feb 2020

It looks like the old problem manifested itself in a different way: https://github.com/saltstack/salt/issues/53570

Below is a quick fix:

patches/pip.sls:

{% if grains.saltversion == "3000" %}
patch:
  pkg.installed

pip_state_patch:
  file.patch:
    - name: '{{ grains.saltpath }}'
    - source: salt://patches/files/3000-pip.diff
    - strip: 2
    - require:
        - pkg: patch

restart_salt_minion:
  cmd.run:
    - name: 'salt-call service.restart salt-minion'
    - bg: true
    - onchanges:
      - file: pip_state_patch
{%- endif %}

patches/files/3000-pip.diff:

diff --git a/salt/states/pip_state.py b/salt/states/pip_state.py
index 0f762752d0..97a9feeb48 100644
--- a/salt/states/pip_state.py
+++ b/salt/states/pip_state.py
@@ -100,14 +100,13 @@ except ImportError:
     import sys
     pip_related_entries = [
         (k, v) for (k, v) in sys.modules.items()
-        or getattr(v, '__module__', '').startswith('pip.')
+        if getattr(v, '__module__', '').startswith('pip.')
         or (isinstance(v, types.ModuleType) and v.__name__.startswith('pip.'))
     ]
     for name, entry in pip_related_entries:
         sys.modules.pop(name)
         del entry

-    del pip
     sys_modules_pip = sys.modules.pop('pip', None)
     if sys_modules_pip is not None:
         del sys_modules_pip

Run sudo MINION state.apply patches.pip to apply the fix.

Another option is to override the pip_state.py module. However, the process is more complex.

max-arnold on 12 Feb 2020

👍3

Thanks @max-arnold

ymasson on 12 Feb 2020

Thanks for opening this. Was wondering why all my Travis jobs started turning red for content I han't changed.

ferricoxide on 12 Feb 2020

Sounds like https://github.com/saltstack/salt/issues/54755#issuecomment-535646926

stratusjerry on 14 Feb 2020

Sounds like #54755 (comment)

It is exactly like this. So an old bug has re-emerged yet again.

CaptainSanders on 26 Feb 2020

fixed in #56215

Ch3LL on 13 Mar 2020

fixed in #56215

How sure of a fix is this? Without pip installed salt 3000.1 (both server and minion) still ends in python error

CaptainSanders on 30 Mar 2020

On my end (Debian) everything is fixed with 3000.1

dynek on 1 Apr 2020

Is it possible that this is python 2 related?

CaptainSanders on 2 Apr 2020

The oldest Debian I have here is 8 and I'm pretty sure it's still using the python 2 flavor:
deb http://repo.saltstack.com/apt/debian/8/amd64/latest jessie main

dynek on 2 Apr 2020

I know how this is going to sound, but it started working out of nowhere. Let me assure you I checked and double-checked versions and the function, all updates had been installed. I'm at a loss why it started working but happy it did.

CaptainSanders on 2 Apr 2020

Was this page helpful?

0 / 5 - 0 ratings