Salt: pkgrepo.managed broken in 2017.7.2

[DenisBY]

Description of Issue/Question

2017.7.1:

[INFO    ] Running state [deb http://repo.saltstack.com/apt/ubuntu/14.04/amd64/archive/2017.7.2 trusty main] at time 14:55:20.779647
[INFO    ] Executing state pkgrepo.managed for [deb http://repo.saltstack.com/apt/ubuntu/14.04/amd64/archive/2017.7.2 trusty main]
[DEBUG   ] LazyLoaded cp.cache_file
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'i-0d94f55c48cf0a929', 'tcp://10.165.9.222:4506', 'aes')
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'i-0d94f55c48cf0a929', 'tcp://10.165.9.222:4506')
[DEBUG   ] Requesting URL https://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest/SALTSTACK-GPG-KEY.pub using GET method
[DEBUG   ] LazyLoaded cmd.run_stdout
[INFO    ] Executing command ['apt-key', 'add', '/var/cache/salt/minion/extrn_files/base/repo.saltstack.com/apt/ubuntu/14.04/amd64/latest/SALTSTACK-GPG-KEY.pub'] in directory '/root'
[DEBUG   ] stdout: OK
[INFO    ] Executing command ['apt-get', '-q', 'update'] in directory '/root'
^C

2017.7.2:

[INFO    ] Running state [deb http://repo.saltstack.com/apt/ubuntu/14.04/amd64/archive/2017.7.2 trusty main] at time 14:49:36.006740
[INFO    ] Executing state pkgrepo.managed for [deb http://repo.saltstack.com/apt/ubuntu/14.04/amd64/archive/2017.7.2 trusty main]
[DEBUG   ] LazyLoaded cp.cache_file
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'i-0d94f55c48cf0a929', 'tcp://10.165.9.222:4506', 'aes')
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'i-0d94f55c48cf0a929', 'tcp://10.165.9.222:4506')
[DEBUG   ] Requesting URL https://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest/SALTSTACK-GPG-KEY.pub using GET method
[DEBUG   ] LazyLoaded cmd.run_stdout
[INFO    ] Executing command ['apt-key', 'add', '/var/cache/salt/minion/extrn_files/base/repo.saltstack.com/apt/ubuntu/14.04/amd64/latest/SALTSTACK-GPG-KEY.pub'] in directory '/root'
[ERROR   ] Command '['apt-key', 'add', '/var/cache/salt/minion/extrn_files/base/repo.saltstack.com/apt/ubuntu/14.04/amd64/latest/SALTSTACK-GPG-KEY.pub']' failed with return code: 2
[ERROR   ] stderr: gpg: no valid OpenPGP data found.
[ERROR   ] retcode: 2
[ERROR   ] Failed to configure repo 'deb http://repo.saltstack.com/apt/ubuntu/14.04/amd64/archive/2017.7.2 trusty main': Error: failed to add key from https://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest/SALTSTACK-GPG-KEY.pub
[INFO    ] Completed state [deb http://repo.saltstack.com/apt/ubuntu/14.04/amd64/archive/2017.7.2 trusty main] at time 14:49:36.625604 duration_in_ms=618.863

# ls -l /var/cache/salt/minion/extrn_files/base/repo.saltstack.com/apt/ubuntu/14.04/amd64/latest/SALTSTACK-GPG-KEY.pub
-rw-r--r-- 1 root root 0 Oct 10 14:44 /var/cache/salt/minion/extrn_files/base/repo.saltstack.com/apt/ubuntu/14.04/amd64/latest/SALTSTACK-GPG-KEY.pub

File is empty

Setup

{% set salt_version = "2017.7.2" %}

salt-repo:
  pkgrepo.managed:
    - clean_file: True
    - name: deb http://repo.saltstack.com/apt/ubuntu/{{ salt['grains.get']('lsb_distrib_release') }}/amd64/archive/{{ salt_version }} {{ salt['grains.get']('lsb_distrib_codename') }} main
    - file: /etc/apt/sources.list.d/saltstack.list
    - key_url: https://repo.saltstack.com/apt/ubuntu/{{ salt['grains.get']('lsb_distrib_release') }}/amd64/latest/SALTSTACK-GPG-KEY.pub
    - require:
- file: /etc/apt/sources.list.d/saltstack-salt2015-8-trusty.list

Versions Report

# salt --versions-report
Salt Version:                                                                                                                                                                                
           Salt: 2017.7.2                                                                                                                                                                    

Dependency Versions:
           cffi: 1.9.1
       cherrypy: 3.2.2
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 0.6.3
      gitpython: 0.3.4
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.3.0
   mysql-python: 1.2.3
      pycparser: 2.17
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.6 (default, Oct 26 2016, 20:30:19)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 14.4.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.4

System Versions:
           dist: Ubuntu 14.04 trusty
         locale: UTF-8
        machine: x86_64
        release: 3.13.0-115-generic
         system: Linux
        version: Ubuntu 14.04 trusty

[gtmanfred]

I am able to replicate this, it looks like cp.cache_file is caching an empty file.

It also looks like this was introduced in https://github.com/saltstack/salt/commit/b8384608160a88b91cecc8d8bf90e0a09719875f

@terminalmage is going to take a look.

Thanks for reporting,
Daniel

[terminalmage]

Fixed in https://github.com/saltstack/salt/pull/44016.

[DenisBY]

Will be there a hot fix or I should wait for .7.3? I'm installing from deb packages on Ubuntu and cannot use version from git.

[terminalmage]

This will be in 2017.7.3.

[boltronics]

Unfortunately this broke some of our environments. I note 2017.7.2 was a security release as well, so that leaves us between a rock and a hard place.

@terminalmage Can we get an ETA on when 2017.7.3 will be out? If it's not expected to be out in the next few days, I'll need to repackage myself for now.

[eliasp]

@boltronics Although that's a bit dirty and bypasses your distributions package manager, you could distribute salt/fileclient.py from #44016 temporarily using a file.managed state directly to /usr/lib/python2.7/dist-packages/salt/fileclient.py (might differ depending on distribution).

I usually deploy those kind of "hotfixes" using Salt's Dynamic Module Distribution but this doesn't work in this case as it's a Salt core module.

[boltronics]

Thanks for the suggestion @eliasp. In this case, I feel it's cleaner to just package Salt myself. I already have a custom repo for any package changes we need and this prevents making changes to all our state sls repos for a temporary fix (and is also probably the cleanest solution), but was crossing my fingers for an imminent release so I didn't need to do anything aside from maybe adjust version pinning.

[thehunmonkgroup]

Goodness, this issue sucks. In my testing, the state still fails even if I manually install the GPG key first.

Here's my reasonably clean hack to work around things until the next release (Debian based only):

• Figure out the keyid of the key to be installed
• Comment out key_url in the pkgrepo.managed state
• Add a prior state like this, subbing [key_url] and [key_id] as appropriate:

add-gpg-key:
  cmd.run:
    - name: 'curl [key_url] | apt-key add -'
    - onlyif: 'test -z "`apt-key list | grep [key_id]`"'

It depends on the curl binary, but this seems much better than hacking core.

[jaul]

+1 for an ETA on this release.

[kiemlicz]

I _kind of_ understand your release cycle,
But issue like this should be considered for immediate backport to existing releases.
The whole pkgrepo state is practically unusable.
Please let me know what do you think

[Rumbles]

Am I right thinking this is fixed in 2017.3 ? I just hit this issue and amanged to update to 2017.3 and don;t see the issue any more...

[kiemlicz]

2017.7.3 is no longer affected

[gtmanfred]

This is fixed in 2017.7.3