Salt: KeyError: 'runas' after updating to latest salt in yum repo.

Created on 17 Oct 2017  路  15Comments  路  Source: saltstack/salt

Description of Issue/Question

After upgrading to Salt 2017.7.2-1.el6 from the salt repo, we started getting a keyerror 'runas' followed by a python traceback. We've tried increasing the verbosity to identify the specific failure in our SLS files but the best we can figure is we use cmd.script to call a script as a specific user. I'm not sure where else we would be calling anything that could cause a reference to 'runas'.

Setup

This is the only section that I think might be affected:
cmd.script:
- name: /tmp/CLIS/patch_wrapper.sh
- args: {{ install_path }}/install_update
- user: hci
- shell: /bin/bash
- cwd: {{ install_path }}

Steps to Reproduce Issue

[DEBUG ] File /var/cache/salt/minion/accumulator/139664175529552 does not exist, no need to cleanup.
[ERROR ] An un-handled exception was caught by salt's global exception handler:
KeyError: 'runas'
Traceback (most recent call last):
File "/usr/bin/salt-call", line 11, in
salt_call()
File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 391, in salt_call
client.run()
File "/usr/lib/python2.7/site-packages/salt/cli/call.py", line 58, in run
caller.run()
File "/usr/lib/python2.7/site-packages/salt/cli/caller.py", line 134, in run
ret = self.call()
File "/usr/lib/python2.7/site-packages/salt/cli/caller.py", line 204, in call
ret['return'] = func(args, *kwargs)
File "/usr/lib/python2.7/site-packages/salt/modules/state.py", line 191, in orchestrate
pillarenv=pillarenv)
File "/usr/lib/python2.7/site-packages/salt/runners/state.py", line 82, in orchestrate
orchestration_jid=orchestration_jid)
File "/usr/lib/python2.7/site-packages/salt/modules/state.py", line 1109, in sls
ret = st_.state.call_high(high_, orchestration_jid)
File "/usr/lib/python2.7/site-packages/salt/state.py", line 2531, in call_high
ret = self.call_chunks(chunks)
File "/usr/lib/python2.7/site-packages/salt/state.py", line 2017, in call_chunks
running = self.call_chunk(low, running, chunks)
File "/usr/lib/python2.7/site-packages/salt/state.py", line 2364, in call_chunk
running[tag] = self.call(low, chunks, running)
File "/usr/lib/python2.7/site-packages/salt/state.py", line 1873, in call
self.state_con.pop('runas')
KeyError: 'runas'
Traceback (most recent call last):
File "/usr/bin/salt-call", line 11, in
salt_call()
File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 391, in salt_call
client.run()
File "/usr/lib/python2.7/site-packages/salt/cli/call.py", line 58, in run
caller.run()
File "/usr/lib/python2.7/site-packages/salt/cli/caller.py", line 134, in run
ret = self.call()
File "/usr/lib/python2.7/site-packages/salt/cli/caller.py", line 204, in call
ret['return'] = func(args, *kwargs)
File "/usr/lib/python2.7/site-packages/salt/modules/state.py", line 191, in orchestrate
pillarenv=pillarenv)
File "/usr/lib/python2.7/site-packages/salt/runners/state.py", line 82, in orchestrate
orchestration_jid=orchestration_jid)
File "/usr/lib/python2.7/site-packages/salt/modules/state.py", line 1109, in sls
ret = st_.state.call_high(high_, orchestration_jid)
File "/usr/lib/python2.7/site-packages/salt/state.py", line 2531, in call_high
ret = self.call_chunks(chunks)
File "/usr/lib/python2.7/site-packages/salt/state.py", line 2017, in call_chunks
running = self.call_chunk(low, running, chunks)
File "/usr/lib/python2.7/site-packages/salt/state.py", line 2364, in call_chunk
running[tag] = self.call(low, chunks, running)
File "/usr/lib/python2.7/site-packages/salt/state.py", line 1873, in call
self.state_con.pop('runas')
KeyError: 'runas'

Versions Report

running salt in a masterless configuration:
[root@ip-10-0-0-33 opt]# salt-call --versions-report
Salt Version:
Salt: 2017.7.2

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.8.1
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.13 (default, Jul 12 2017, 17:32:34)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 14.5.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.0.5

System Versions:
dist: centos 6.9 Final
locale: UTF-8
machine: x86_64
release: 2.6.32-696.10.2.el6.x86_64
system: Linux
version: CentOS 6.9 Final

ZD cannot-reproduce
Source
picture dupsatou

Most helpful comment

Ahh, that kind of makes sense, if you are applying a state in a module it could potentially overwrite the whole context.

We should just switch the pop to 

self.state_con.pop('runas', None)
self.state_con.pop('runas_password', None)

and be done with it.

Thanks,
Daniel

picture gtmanfred on 16 Nov 2017
👍2

All 15 comments

@twangboy Looks like this might be related to a change you made recently, https://github.com/saltstack/salt/commit/6c71ab6f803320931b38ac07d642bc0a4adea02e

garethgreenaway picture garethgreenaway on 17 Oct 2017

After further investigation I haven't been able to reproduce this issue. Can you confirm that the minion is being restarted after the upgrade?

garethgreenaway picture garethgreenaway on 17 Oct 2017

We're running masterless so we aren't using a minion. I noticed I did post mine with some jinja in it but that shouldn't matter. let me see if I can narrow down the failure. Maybe I pulled the wrong part of my SLS. Let me troubleshoot further and narrow it down better. Sorry!

dupsatou picture dupsatou on 17 Oct 2017

Interestingly, I may have made an incorrect assumption on where it is failing. We run salt to bootstrap a server like so: salt-call -l debug --local state.orchestrate orch.software

When I call the states below individually they run just fine without errors. If I call the orchestration file instead I get the error above. The debug output doesn't tell me exactly where its failing but it seems to get through the tools_setup and fail on the tools_install. This failure does not occur on 2017.7.1. We are able to downgrade and it runs perfectly. I suspected it was the cmd.run and the 'runas' and saw the same commit you referenced thinking that may have broke it. But maybe not?

The orch file looks like this:

software_setup:
  salt.state:
    - tgt: '*'
    - sls: tools_setup
software_install:
  salt.state:
    - tgt: '*'
    - sls: tools_install
    - require:
      - salt: software_setup

tools_setup like this:

update yum conf:
  file.append:
    - name: /etc/yum.conf
    - text:
      - "multilib_policy=all"


Development Tools:
  pkg.group_installed

Additional Development:
  pkg.group_installed

Compatibility Libraries:
  pkg.group_installed:
    - skip:
      - compat-libgcc-296
      - compat-libstdc++-295
      - compat-libstdc++-296
      - compat-openmpi
      - compat-openmpi-psm

Additional Packages:
    pkg.installed:
      - pkgs:
        - unzip
        - nano
        - tcsh
        - vim-enhanced
        - vsftpd
        - xrdp
        - tigervnc-server
        - fluxbox
        - xterm
        - pdfgrep

{% for file in salt['cmd.shell']('ls -1 /tmp/CLIS/*.tar.gz').splitlines() %}
{% set folder = file.split('.tar.') %}
extract_{{ file }}:
  archive.extracted:
    - name: {{ folder[0] }}
    - source: {{ file }}
    - user: root
    - group: root
    - enforce_toplevel: false

{% endfor %}

/tmp/CLIS/patch_wrapper.sh:
  file.managed:
    - contents:  |
        #!/bin/bash

        echo "Y 1" | $1
    - mode: 775
    - replace: false

and tools_install:

{% for version in salt['cmd.shell']('find /tmp/CLIS/ -name ReleaseNotes.pdf -exec pdfgrep Release\\: {} \\;').splitlines()|sort %}
{% set release_path = salt['cmd.shell']('find /tmp/CLIS/ -name ReleaseNotes.pdf -exec pdfgrep -H \"' ~ version ~ '\" {} \; | grep \"' ~ version ~ '\"\$ ') %}
{% set dirs = release_path.split('/') %}
{% set install_path = "/tmp/CLIS/" ~ dirs[3] %}
install {{ dirs[3] }}:
{% if loop.index == 1 %}
  cmd.run:
    - name: {{ install_path }}/CLInstall.sh -i silent
{% else %}
  cmd.script:
    - name: /tmp/CLIS/patch_wrapper.sh
    - args: {{ install_path }}/install_update
    - user: hci
{% endif %}
    - shell: /bin/bash
    - cwd: {{ install_path }}
{% endfor %}

copy_license:
  file.copy:
    - name: salt['environ.get']('LICENSE_DIR')
    - source: /tmp/CLIS/license.dat
    - user: hci
    - group: hci

/tmp/CLIS/patch_wrapper.sh:
  file.absent
dupsatou picture dupsatou on 17 Oct 2017

We could check that it exists before popping it... Maybe state_con persists across state runs in orchestration...

twangboy picture twangboy on 17 Oct 2017

I started getting a similar/same error after updating a minion to 2017.7.2.
Get the below error when applying states from master. Does not happen when minion is running 2017.7.1
The states do seem to apply without problems, exception pops up in the return as far as I can tell. No impact on cmd.run or some other functions I tested.

The minion function caused an exception: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1470, in _thread_return
    return_data = executor.execute()
  File "/usr/lib/python2.7/site-packages/salt/executors/direct_call.py", line 28, in execute
    return self.func(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/salt/modules/state.py", line 592, in apply_
    return sls(mods, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/modules/state.py", line 1109, in sls
    ret = st_.state.call_high(high_, orchestration_jid)
  File "/usr/lib/python2.7/site-packages/salt/state.py", line 2531, in call_high
    ret = self.call_chunks(chunks)
  File "/usr/lib/python2.7/site-packages/salt/state.py", line 2017, in call_chunks
    running = self.call_chunk(low, running, chunks)
  File "/usr/lib/python2.7/site-packages/salt/state.py", line 2364, in call_chunk
    running[tag] = self.call(low, chunks, running)
  File "/usr/lib/python2.7/site-packages/salt/state.py", line 1873, in call
    self.state_con.pop('runas')
  File "/usr/lib64/python2.7/_abcoll.py", line 497, in pop
    value = self[key]
  File "/usr/lib/python2.7/site-packages/salt/utils/context.py", line 227, in __getitem__
    return self._dict()[key]
KeyError: 'runas'

salt-call --versions-report
Salt Version:
Salt: 2017.7.2

Dependency Versions:
cffi: 1.10.0
cherrypy: Not Installed
dateutil: 2.6.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.8
mysql-python: Not Installed
pycparser: 2.18
pycrypto: 2.6.1
pycryptodome: 3.4.3
pygit2: Not Installed
Python: 2.7.5 (default, Aug 4 2017, 00:39:18)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 15.3.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: centos 7.4.1708 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-693.2.2.el7.x86_64
system: Linux
version: CentOS Linux 7.4.1708 Core

nulfox picture nulfox on 18 Oct 2017

I am also facing this
```
[ERROR ] Unhandled exception running state.apply
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/salt/utils/schedule.py", line 859, in handle_func
ret['return'] = self.functionsfunc
File "/usr/local/lib/python2.7/dist-packages/salt/modules/state.py", line 595, in apply_
return sls(mods, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/salt/modules/state.py", line 1102, in sls
ret = st_.state.call_high(high_, orchestration_jid)
File "/usr/local/lib/python2.7/dist-packages/salt/state.py", line 2591, in call_high
ret = self.call_chunks(chunks)
File "/usr/local/lib/python2.7/dist-packages/salt/state.py", line 2075, in call_chunks
running = self.call_chunk(low, running, chunks)
File "/usr/local/lib/python2.7/dist-packages/salt/state.py", line 2424, in call_chunk
running[tag] = self.call(low, chunks, running)
File "/usr/local/lib/python2.7/dist-packages/salt/state.py", line 1927, in call
self.state_con.pop('runas')
File "/usr/lib/python2.7/_abcoll.py", line 497, in pop
value = self[key]
File "/usr/local/lib/python2.7/dist-packages/salt/utils/context.py", line 228, in __getitem__
return self._dict()[key]
KeyError: 'runas'

vnitinv picture vnitinv on 29 Oct 2017

This can be reproduced, at some level, with the two following states.

a.sls

repro:
  module.run:
    - name: state.apply
    - mods: b

b.sls

/tmp/b:
  file.managed:
    - contents: c

Running state.apply a results in the following output.

[root@ip-10-24-44-24 base]# salt-call state.apply a
[WARNING ] The function "module.run" is using its deprecated version and will expire in version "Sodium".
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
KeyError: 'runas'
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 391, in salt_call
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/call.py", line 58, in run
    caller.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 134, in run
    ret = self.call()
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 204, in call
    ret['return'] = func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/modules/state.py", line 592, in apply_
    return sls(mods, **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/modules/state.py", line 1109, in sls
    ret = st_.state.call_high(high_, orchestration_jid)
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2531, in call_high
    ret = self.call_chunks(chunks)
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2017, in call_chunks
    running = self.call_chunk(low, running, chunks)
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2364, in call_chunk
    running[tag] = self.call(low, chunks, running)
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1873, in call
    self.state_con.pop('runas')
KeyError: 'runas'
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 391, in salt_call
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/call.py", line 58, in run
    caller.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 134, in run
    ret = self.call()
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 204, in call
    ret['return'] = func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/modules/state.py", line 592, in apply_
    return sls(mods, **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/modules/state.py", line 1109, in sls
    ret = st_.state.call_high(high_, orchestration_jid)
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2531, in call_high
    ret = self.call_chunks(chunks)
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2017, in call_chunks
    running = self.call_chunk(low, running, chunks)
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2364, in call_chunk
    running[tag] = self.call(low, chunks, running)
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1873, in call
    self.state_con.pop('runas')
KeyError: 'runas'

Correct me if I'm wrong, but wouldn't it be possible to circumvent this issue by simply adding a default None the two offending calls to state_con.pop?

self.state_con.pop('runas', None)
self.state_con.pop('runas_password', None)
vschumaker picture vschumaker on 13 Nov 2017

I have the same issue, is there any ETA for a fix?
I am also using module.run in salt state file

maneamarius picture maneamarius on 16 Nov 2017

@vschumaker That seems to work for me (though, I am using try/except). Good quick fix. Ideally we would remove those lines or figure out why they no longer work and adjust. But it seems to goal is to remove those fields (if present), potentially cleaning up return?

nulfox picture nulfox on 16 Nov 2017

Ahh, that kind of makes sense, if you are applying a state in a module it could potentially overwrite the whole context.

We should just switch the pop to 

self.state_con.pop('runas', None)
self.state_con.pop('runas_password', None)

and be done with it.

Thanks,
Daniel

gtmanfred picture gtmanfred on 16 Nov 2017
👍2

ZD-1993

rickh563 picture rickh563 on 20 Nov 2017

I noticed that this is tagged with "cannot reproduce". Is that still true?

wwalker picture wwalker on 27 Nov 2017

I have still been unable to reproduce it, but I don't see a reason we can't just switch it over to popping None if the key is not found, we don't want to error there anyway.

Opening a PR now.

Daniel

gtmanfred picture gtmanfred on 27 Nov 2017

This will be fixed in 2017.7.3 here https://github.com/saltstack/salt/pull/44695

gtmanfred picture gtmanfred on 27 Nov 2017
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

mooperd picture mooperd  路  3Comments
Arguros picture Arguros  路  3Comments
twangboy picture twangboy  路  3Comments
sfozz picture sfozz  路  3Comments
nixjdm picture nixjdm  路  3Comments