salt-ssh pkg.install debconf= broken (module and state)

[leonardinius]

This is what I see running smth like https://github.com/rentalita/ubuntu-setup/blob/master/salt/gitolite/init.sls#L4-L7 For the sake of readability - will copy it here as well

gitolite:
  pkg:
    - latest
    - debconf: salt://gitolite/gitolite.answers

Out

...
  pkg_|-gitolite_|-gitolite_|-latest:
        ----------
        __run_num__:
            18
        changes:
            ----------
        comment:
            An exception occurred in this state: Traceback (most recent call last):
              File "/tmp/.salt/salt/state.py", line 1368, in call
                **cdata['kwargs'])
              File "/tmp/.salt/salt/states/pkg.py", line 701, in latest
                **kwargs)
              File "/tmp/.salt/salt/modules/apt.py", line 378, in install
                __salt__['debconf.set_file'](debconf)
            KeyError: 'debconf.set_file'

        name:
            gitolite
        result:
            False
...

Initially I thought this option is not supported, since this option seemed not be state module, but instead salt module (I might misuse terminology, using the terms used in salt documentation pages)

Salt state module has no mention of it, however seems in might be supported (especially if found online examples it's usage, see above)

Salt apt module however lists debconf as supported option.

Initially I asked questions about this on salt-user iirc channel and got initial response I mix salt states and salt-modules. However since I have both found the online example (presumably working) and failed to run salt-module (see below) as documented - I decided to submit my question in the form of github issue. imo, either documentation or code is broken.

---

Questions

• is debconf supported as part of state module option?
• is debconf supported as part of sast system module? e.g.

salt-ssh '*'  pkg.install gitolite 'debconf=salt://gitolite/gitolite.answers'
minion1:
    Traceback (most recent call last):
      File "/tmp/.salt/salt-call", line 4, in <module>
        salt_call()
      File "/tmp/.salt/salt/scripts.py", line 82, in salt_call
        client.run()
      File "/tmp/.salt/salt/cli/__init__.py", line 313, in run
        caller.run()
      File "/tmp/.salt/salt/cli/caller.py", line 137, in run
        ret = self.call()
      File "/tmp/.salt/salt/cli/caller.py", line 78, in call
        ret['return'] = func(*args, **kwargs)
      File "/tmp/.salt/salt/modules/apt.py", line 378, in install
        __salt__['debconf.set_file'](debconf)
    KeyError: 'debconf.set_file'

• is debconf just broken with salt-ssh or broken in last/development? _Note_: The way I've installed salt is as documented at saltstack/salt-bootstrap e.g. curl -L http://bootstrap.saltstack.org | sudo sh -s -- git develop

version info

root@salt:/media/sf_provisioning# salt --versions-report
           Salt: 2014.1.0-438-g23eb3a7
         Python: 2.7.3 (default, Sep 26 2013, 20:03:06)
         Jinja2: 2.6
       M2Crypto: 0.21.1
 msgpack-python: 0.1.10
   msgpack-pure: Not Installed
       pycrypto: 2.4.1
         PyYAML: 3.10
          PyZMQ: 13.0.0
            ZMQ: 3.2.2
root@salt:/media/sf_provisioning#

[leonardinius]

It seems debconf is either missing or not being deployed in bootstrapped version

e.g.

salt-ssh '*' debconf.set_file 'file:/tmp/gitolite.answers'
minion1:
    Function debconf.set_file is not available

E.g. if I run salt-ssh '*' sys.doc pkg.install I get a solid piece of documentation, howeer if I run salt-ssh '*' sys.doc 'debconf' I got none.

[leonardinius]

According to irc (see http://irclog.perlgeek.de/salt/2014-01-20#i_8144186) it works like a charm in salt v 0.17.4

Thus I consider this to be regression error introduced in latest develop branch

Update: see comments below - it works either locally or via maste/minion mode, not via salt-ssh

[leonardinius]

Checked how it behaves on v0.17.4

It seems it works in locally and via salt/master/salt-minion (according to irc logs) on v0.17.4, does not work via salt-ssh

salt-call --local

root@salt:/media/sf_provisioning# salt-call --local pkg.install gitolite 'debconf=salt://gitolite/gitolite.answers'
[INFO    ] Executing command 'debconf-set-selections /srv/salt/gitolite/gitolite.answers' in directory '/home/leonids'
[INFO    ] Executing command "dpkg-query --showformat='${Status} ${Package} ${Version} ${Architecture}\n' -W" in directory '/home/leonids'
[INFO    ] Executing command 'grep-available -F Provides -s Package,Provides -e "^.+$"' in directory '/home/leonids'
[INFO    ] Executing command ['apt-get', '-q', '-y', '-o', 'DPkg::Options::=--force-confold', '-o', 'DPkg::Options::=--force-confdef', 'install', 'gitolite'] in directory '/home/leonids'
^C
Exiting gracefully on Ctrl-c

salt-ssh '*'

root@salt:/media/sf_provisioning# salt-ssh '*' pkg.install gitolite 'debconf=salt://gitolite/gitolite.answers'
minion1:
    Traceback (most recent call last):
      File "/tmp/.salt/salt-call", line 4, in <module>
        salt_call()
      File "/tmp/.salt/salt/scripts.py", line 77, in salt_call
        client.run()
      File "/tmp/.salt/salt/cli/__init__.py", line 303, in run
        caller.run()
      File "/tmp/.salt/salt/cli/caller.py", line 135, in run
        ret = self.call()
      File "/tmp/.salt/salt/cli/caller.py", line 78, in call
        ret['return'] = func(*args, **kwargs)
      File "/tmp/.salt/salt/modules/apt.py", line 371, in install
        __salt__['debconf.set_file'](debconf)
    KeyError: 'debconf.set_file'

``

[leonardinius]

See previous comment - latest develop branch actually behaves the same

[basepi]

This may actually be related to the PATH setting on the target system. The debconf module is only loaded if the grain 'os_family' is 'Debian' and if it can find the command debconf-get-selections on the system. So one of these is failing, and I'm betting it's the debconf-get-selection. I would recommend using grains.item to make sure that the 'os_family' is set correctly to verify my hypothesis, and then we need to start debugging to figure out why salt.utils.which() is not finding that debconf command.

[leonardinius]

Both local (salt-ssh host) and remote (minion salt-ssh client) are the same OS, both are Ubuntu.

It might be however the local is 12.10, but remote 13.10 (I remember it trying on the same OS installed from the same iso, but I might be wrong)

I will check grains output and will be back shortly to confirm

[leonardinius]

See remote node (salt-ssh 'minion') grains below:

...
  kernel: Linux
  kernelrelease: 3.11.0-12-generic
  lsb_distrib_codename: saucy
  lsb_distrib_description: Ubuntu 13.10
  lsb_distrib_id: Ubuntu
  lsb_distrib_release: 13.10
  manufacturer: Bochs
  os_family: Debian
  osarch: i386
  oscodename: saucy
  osfinger: Ubuntu-13.10
  osfullname: Ubuntu
  osrelease: 13.10
  path: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
  productname: Bochs
  pythonpath:
      /tmp/.salt
      /usr/lib/python2.7
      /usr/lib/python2.7/plat-i386-linux-gnu
      /usr/lib/python2.7/lib-tk
      /usr/lib/python2.7/lib-old
      /usr/lib/python2.7/lib-dynload
      /usr/local/lib/python2.7/dist-packages
      /usr/lib/python2.7/dist-packages
  pythonversion: 2.7.5.final.0
  saltpath: /tmp/.salt/salt
  saltversion: 2014.1.0-562-g70f637d
  saltversioninfo:
      2014
      1
      0
  serialnumber: Not Specified
  server_id: 1210048638
  shell: /bin/bash
  virtual: kvm
...

[basepi]

So now we just have to figure out what's different about the PATH that it can't find that executable....

[leonardinius]

It's rather lengthy comment, so I've splitted it into parts: a) salt-ssh host grain, b) path envs

---

a) salt-ssh host grains

I've executed salt-call --local grains.items --out grains > grainst.txt on salt-ssh host, here it is

...
  kernel: Linux
  kernelrelease: 3.8.0-35-generic
  localhost: salt
  lsb_distrib_codename: precise
  lsb_distrib_description: Ubuntu 12.04.4 LTS
  lsb_distrib_id: Ubuntu
  lsb_distrib_release: 12.04
  manufacturer: innotek GmbH
  master: salt
  mem_total: 112
  nodename: salt
  num_cpus: 1
  num_gpus: 1
  os: Ubuntu
  os_family: Debian
  osarch: amd64
  oscodename: precise
  osfinger: Ubuntu-12.04
  osfullname: Ubuntu
  osrelease: 12.04
  path: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  productname: VirtualBox
  ps: ps -efHww
  pythonpath:
      /usr/bin
      /usr/lib/python2.7
      /usr/lib/python2.7/plat-linux2
      /usr/lib/python2.7/lib-tk
      /usr/lib/python2.7/lib-old
      /usr/lib/python2.7/lib-dynload
      /usr/local/lib/python2.7/dist-packages
      /usr/lib/python2.7/dist-packages
      /usr/lib/pymodules/python2.7
  pythonversion: 2.7.3.final.0
  saltpath: /usr/lib/python2.7/dist-packages/salt
  saltversion: 2014.1.0-562-g70f637d
  saltversioninfo: (2014, 1, 0)
  serialnumber: 0
  server_id: 1540459516
  shell: /bin/bash
  virtual: VirtualBox
...

---

b) path envs

• salt-ssh host : path: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

$ which debconf-set-selections && which dpkg-query
/usr/bin/debconf-set-selections
/usr/bin/dpkg-query

• salt-ssh target: path: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

$ salt-ssh '*' -r 'which debconf-set-selections && which dpkg-query'
minion1:
    /usr/bin/debconf-set-selections
    /usr/bin/dpkg-query

---

summary

as far i can tell there is no other differences rather than OS version number (however as mentioned above I believe I have had the same result on the same version) and extra PATH directories.

[leonardinius]

found another difference - python version

• salt-ssh target (minion): pythonversion: 2.7.5.final.0
• salt-ssh host (master): pythonversion: 2.7.3.final.0

[leonardinius]

Mistery solved, my bad actually

I somehow misread your comment and checked against debconf-set-selections presence on minion (because of it's presence available in the log) instead of debconf-get-selections you actually mentioned in your comment #9849#issuecomment-33076862

Confusion
Nevertheless I believe this behavior is somehow confusing: e.g. pkg.install xxx 'debconf=yyy' only uses debconf-set-selections underneath (via debconf.set_file), not debconf-get-selections. debconf-get-selections is used only for debconf.get_selections and debconf.show.

I find it confusing that in this particular case the subcomponent the pkg.install functionallity actually depends on is available, but is disabled due to absence of another subcomponent.

Or even worse, if for some reason debconf-get-selections is present, the debconf module will be loaded, but pkg.install (via debconf.set_file) still might fail if debconf-set-selections is absent.

Q1
To make things more consistent I would recommend to reconsider this approach and either find another way to check for debconf sub-components presence, OR check for both debconf-get-selections and debconf-set-selections presence; if only one is present - the warning might be issued.

That's you take on it? If the later sounds ok to you, I might try to come up with small 2-5 lines big pull request (with your help and technical guidance of-course)

[basepi]

The latter idea does sound reasonable. We can't put these checks inside of the pkg state, because it's designed to be platform agnostic. But putting them within the apt module could work.

We do need to be careful, however. We never put logs in __virtual__ functions, for example, because that would _rapidly_ take over our logs, as anyone who _wasn't_ running apt would get those warnings. So we need to be very careful what debug information we put in.

[leonardinius]

@basepi Are you sure it's the apt /-> pkg things should be altered / addressed in this regard?

To summarize: it's the debconf state / module which behaves the way described as follows: https://github.com/saltstack/salt/issues/9849#issuecomment-33172107
The apt state/module just assumes the debconf should be always available (but is not as per comment above). See https://github.com/saltstack/salt/blob/develop/salt/modules/aptpkg.py#L386-L387, this is the apt part which makes this assumption. Should we check for __salt__['debconf.set_file'] presence here and issue warning if it is not detected as expected?

[basepi]

Ah, I understand. Yes, assuming that apt is found on the system (so not everyone gets the warnings) but it can't find __salt__['debconf.set_file'], we should emit a warning.

[abcfy2]

Same issue here.

# salt-ssh --versions
Salt Version:
           Salt: 2016.11.0

Dependency Versions:
           cffi: 1.9.1
       cherrypy: Not Installed
       dateutil: 1.5
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.3
      pycparser: 2.17
       pycrypto: Not Installed
         pygit2: Not Installed
         Python: 2.7.6 (default, Oct 26 2016, 20:30:19)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: Not Installed
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: Not Installed

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 4.4.0-53-generic
         system: Linux
        version: Ubuntu 14.04 trusty

/srv/salt/java/java8.sls:

{% if grains['osrelease'] < '16.04' %}
java8_ppa_proxy:
  pkgrepo.managed:
    - name: "deb http://launchpad.proxy.ustclug.org/webupd8team/java/ubuntu {{grains['oscodename']}} main"
    - file: /etc/apt/sources.list.d/java8.list
    - keyid: EEA14886
    - keyserver: keyserver.ubuntu.com
    - clean_file: True
    - require:
      - pkg: common_pkgs

java8-debconf:
  debconf.set:
    - name: oracle-java8-installer
    - data:
        'shared/accepted-oracle-license-v1-1': {'type': 'select', 'value': 'true'}

oracle-java8-set-default:
  pkg.installed:
    - require:
      - pkgrepo: java8_ppa_proxy
{% else %}
openjdk-8-jdk:
  pkg.installed
{% endif %}

salt-ssh '*' state.apply java.java8

----------
          ID: java8-debconf
    Function: debconf.set
        Name: oracle-java8-installer
      Result: False
     Comment: State 'debconf.set' was not found in SLS 'java.java8'
              Reason: 'debconf.set' is not available.
     Changes:   
----------
          ID: oracle-java8-set-default
    Function: pkg.installed
      Result: False
     Comment: Problem encountered installing package(s). Additional info follows:

              errors:
                  - 
                    oracle-license-v1-1 license could not be presented
                    try 'dpkg-reconfigure debconf' to select a frontend other than noninteractive

                    dpkg: error processing archive /var/cache/apt/archives/oracle-java8-installer_8u111+8u111arm-1~webupd8~0_all.deb (--unpack):
                     subprocess new pre-installation script returned error exit status 2
                    Errors were encountered while processing:
                     /var/cache/apt/archives/oracle-java8-installer_8u111+8u111arm-1~webupd8~0_all.deb
                    E: Sub-process /usr/bin/dpkg returned an error code (1)
     Started: 15:27:05.576413
    Duration: 7054.683 ms
     Changes:   

But salt-minion works very well.

I have to use cmd.run instead of debconf.set to solve this issue.

[chrisxaustin]

I had the same issue because debconf-get-selections wasn't installed on the minion, which is a bento/ubuntu-16.04 vagrant box.

Added this to the state, which fixed it

debconf-utils:
  pkg.installed

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.