Salt no longer imports keys when using pkgrepo.managed
with a ppa
repository.
Ubuntu 14.04 LTS
salt 2015.8.8.2 (Beryllium)
Master - minion setup
Noticed when I was upgrading all my servers PHP packages to 7.
----------
ID: php-pkgs
Function: pkg.installed
Result: False
Comment: The following packages failed to install/update: php-pear, php7.0-gd, php7.0-intl, php7.0-cli, php7.0, php7.0-mcrypt, php7.0-fpm, php7.0-mysql, php7.0-curl, php7.0-json, php7.0-sqlite3
Started: 01:07:47.464101
Duration: 10235.009 ms
Changes:
State that causes the issue:
php-repo:
pkgrepo.managed:
- humanname: PHP Repo
- ppa: ondrej/php
List of keys after running the salt state:
/etc/apt/trusted.gpg
--------------------
pub 1024D/437D05B5 2004-09-12
uid Ubuntu Archive Automatic Signing Key <[email protected]>
sub 2048g/79164387 2004-09-12
pub 1024D/FBB75451 2004-12-30
uid Ubuntu CD Image Automatic Signing Key <[email protected]>
pub 4096R/C0B21F32 2012-05-11
uid Ubuntu Archive Automatic Signing Key (2012) <[email protected]>
pub 4096R/EFE21092 2012-05-11
uid Ubuntu CD Image Automatic Signing Key (2012) <[email protected]>
pub 4096R/68576280 2014-06-13
uid NodeSource <[email protected]>
sub 4096R/AA01DA2C 2014-06-13
pub 1024D/548C16BF 2010-06-29
uid New Relic <[email protected]>
sub 2048g/77E8EB9A 2010-06-29
pub 2048R/DE57BFBE 2014-06-24
uid SaltStack Packaging Team <[email protected]>
sub 2048R/17928113 2014-06-24
/etc/apt/trusted.gpg.d/nginx-stable.gpg
---------------------------------------
pub 1024R/C300EE8C 2010-07-21
uid Launchpad Stable
As you can see, it is missing the key for the specific PPA.
Salt Version:
Salt: 2015.8.8.2
Dependency Versions:
Jinja2: 2.7.2
M2Crypto: Not Installed
Mako: 0.9.1
PyYAML: 3.10
PyZMQ: 14.0.1
Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
RAET: Not Installed
Tornado: 4.2.1
ZMQ: 4.0.4
cffi: Not Installed
cherrypy: Not Installed
dateutil: 1.5
gitdb: 0.5.4
gitpython: 0.3.2 RC1
ioflo: Not Installed
libgit2: Not Installed
libnacl: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.3.0
mysql-python: 1.2.3
pycparser: Not Installed
pycrypto: 2.6.1
pygit2: Not Installed
python-gnupg: Not Installed
smmap: 0.8.2
timelib: Not Installed
System Versions:
dist: Ubuntu 14.04 trusty
machine: x86_64
release: 3.13.0-83-generic
system: Ubuntu 14.04 trusty
@ZacharyDuBois, thanks for reporting. Does importing the key separately, like in another state, resolve the install problem? Do you know the last version of salt you were using when this was working?
@jfindlay Sorry, I don't know the last version when it was working since I haven't had to do a whole lot with PPAs in awhile :(
I have not tried importing the keys separately. I just ran salt -I 'roles:php' cmd.run 'add-apt-repository ppa:ondrej/php'
@ZacharyDuBois, no problem and thanks for the extra info. I am glad you have found a workaround.
@jfindlay No problem :) Anything to help out
Hi,
I'd like to follow up with this issue with versions:
salt 2016.3.0 (Boron)
salt-minion 2016.3.1 (Boron)
FILE: => rsyslog/init.sls:
# install rsyslog PPA repo:
pkgrepo.managed:
- humanname: rsyslog ppa
- ppa: adiscon/v8-stable
- refresh_db: True
pkg.installed:
- name: rsyslog
- refresh: True
FAILS:
----------
ID: install rsyslog PPA repo
Function: pkgrepo.managed
Result: False
Comment: Failed to configure repo 'install rsyslog PPA repo': Unable to add PPA 'adiscon/v8-stable'. '['apt-add-repository', '-y', 'ppa:adiscon/v8-stable']' exited with status 1: ''
Started: 22:01:37.518964
Duration: 1369.465 ms
Changes:
If run from command line:
# apt-add-repository -y ppa:adiscon/v8-stable
gpg: keyring `/tmp/tmpswf56esy/secring.gpg' created
gpg: keyring `/tmp/tmpswf56esy/pubring.gpg' created
gpg: requesting key 5234BF2B from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpswf56esy/trustdb.gpg: trustdb created
gpg: key 5234BF2B: public key "Launchpad PPA for Adiscon" imported
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)
OK
@ndobbs, could you clarify what you're asking?
The bug still exists in 2016.3.3.
@jfindlay I can confirm this was working in 2015.8. Can't give info about previous versions in the 2016 series.
Same happening for me:
salt 2015.8.5 (Beryllium)
salt-call 2016.3.1 (Boron)
Am going to try this as a workaround:
php_ppa:
pkgrepo.managed:
- humanname: PHP Repo
- ppa: ondrej/php
- keyid: E5267A6C
- keyserver: keyserver.ubuntu.com
Hmm. Manually specifying it ^^^ does not work.
I note that when you run the command line sudo add-apt-repository ppa:ondrej/php
it works.
I also note that the command line hangs and asks you to press the enter key.
I wonder if that wait/enter cycle is something new that salt is not accounting for?
Me, too.
I am running salt-call --local on Ubuntu 14.04 with Salt 2013.3.3.
toolchain_gcc:
pkgrepo.managed:
- humanname: Ubuntu Toolchain
- ppa: ubuntu-toolchain-r/test
#- keyurl: salt://ubuntu_toolchain_public_key
- refresh_db: true
- require_in:
- pkg: gcc_install_{{ gcc_version }}
This works, if I have manually configured the public key,
sudo apt-key add lib/build_tools/salt/ubuntu_toolchain_public_key
but not using the keyurl option.
I tried all sorts of combinations of keyurl, or keyserver with various key id formats, etc, etc. All fail.
Manually configuring the public key is not a very good work-around, since I plan on using this on build servers created on the fly by salt-cloud.
Good News!
The command works correctly using salt-call 2016.11.0rc1
Name: toolchain_gcc - Function: pkgrepo.managed - Result: Clean Started: - 01:55:53.375174 Duration: 1677.934 ms
Name: gcc_install_5 - Function: pkg.installed - Result: Clean Started: - 01:55:55.772813 Duration: 5.307 ms
What worked for me was to add the keys manually:
# Manually install GPG-Key
/etc/apt/trusted.gpg.d/ondrej_ubuntu_php.gpg:
file:
- managed
- source: salt://php5-fpm/ondrej_ubuntu_php.gpg
- user: root
- group: root
- mode: 644
Waaaaw ! I can't understand how a bug like this one is still present since few one year without being fixed !
The bug is no longer present. I have been using the following code for several months with no problem on Carbon (2016.11.1 and later):
toolchain_gcc:
pkgrepo.managed:
- humanname: UbuntuToolchain
- ppa: ubuntu-toolchain-r/test
- keyserver: keyserver.ubuntu.com
- keyid: BA9EF27F
- refresh_db: true
gcc_install_{{ gcc_compiler_version }}:
pkg.installed:
- pkgs:
- gcc-{{ gcc_compiler_version }}
- g++-{{ gcc_compiler_version }}
- require:
- pkgrepo: toolchain_gcc
@jfindlay add a regression test and close this issue!
This still not working !
----------
ID: install php5.6
Function: pkgrepo.managed
Result: True
Comment: Configured package repo 'install php5.6'
Started: 19:39:54.512418
Duration: 4105.239 ms
Changes:
----------
repo:
ppa:ondrej/php
----------
ID: install php5.6
Function: pkg.installed
Result: False
Comment: Problem encountered installing package(s). Additional info follows:
errors:
- Running scope as unit run-rcedb82f82f384b0cb750bd04062e140f.scope.
E: There were unauthenticated packages and -y was used without --allow-unauthenticated
Started: 19:39:58.617790
Duration: 2279.163 ms
Changes:
Saltstack report :
Salt Version:
Salt: 2016.11.2
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.4.2
gitdb: 0.6.4
gitpython: 1.0.1
ioflo: Not Installed
Jinja2: 2.8
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.3
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pygit2: Not Installed
Python: 2.7.12 (default, Nov 19 2016, 06:48:10)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.2.0
RAET: Not Installed
smmap: 0.9.0
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4
System Versions:
dist: Ubuntu 16.04 xenial
machine: x86_64
release: 4.4.0-62-generic
system: Linux
version: Ubuntu 16.04 xenial
My state file :
install php5.6:
pkgrepo.managed:
- humanname: ondrej/php
- ppa: ondrej/php
- keyserver: keyserver.ubuntu.com
- keyid: E5267A6C
- refresh_db: true
pkg.installed:
- pkgs:
- libapache2-mod-php5.6
- php-cache-lite
- php-http
- php-memcache
- php-memcached
- php-pear
- php-redis
- php5.6-cli
- php5.6-common
- php5.6-curl
- php5.6-dev
- php5.6-gd
- php5.6-imagick
- php5.6-imap
- php5.6-intl
- php5.6-json
- php5.6-mcrypt
- php5.6-mysql
- php5.6-propro
- php5.6-raphf
- php5.6-readline
- php5.6-xml
- refresh: True
still present in 2016.11.3
bad workaround would be to add - skip_verify: True
On ubuntu 16.04 with saltstack packages version "2016.11.5+ds-3". Trying to add ppa:ondrej/php results in this:
root@saltmaster:/srv/salt/fs# salt minion1 state.apply webserver
minion1:
----------
ID: ondrej-php-ppa
Function: pkgrepo.managed
Result: False
Comment: Failed to configure repo 'ondrej-php-ppa': Unable to add PPA 'ondrej/php'. '['apt-add-repository', '-y', 'ppa:ondrej/php']' exited with status 1: 'gpg: keyring `/tmp/tmpdlfjtc8z/secring.gpg' created
gpg: keyring `/tmp/tmpdlfjtc8z/pubring.gpg' created
gpg: requesting key E5267A6C from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpdlfjtc8z/trustdb.gpg: trustdb created
gpg: key E5267A6C: public key "Launchpad PPA for Ond\xc5\x99ej Sur媒" imported
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)'
Started: 11:55:17.987772
Duration: 1746.782 ms
Changes:
Summary for minion1
------------
Succeeded: 0
Failed: 1
------------
Total states run: 1
Total run time: 1.747 s
-------------------------------------------
Summary
-------------------------------------------
# of minions targeted: 1
# of minions returned: 1
# of minions that did not return: 0
# of minions with errors: 1
-------------------------------------------
ERROR: Minions returned with non-zero exit code
root@saltmaster:/srv/salt/fs#
So apparently it imports the key to a file "/tmp/tmpdlfjtc8z/trustdb.gpg" and forgets about it?
my saltstate file:
# vim: expandtab ts=2 sw=2 softtabstop=2
ondrej-php-ppa:
pkgrepo.managed:
- ppa: ondrej/php
- refresh_db: true
Resulting "apt-key list" output:
root@minion1:~# apt-key list
/etc/apt/trusted.gpg
--------------------
pub 1024D/437D05B5 2004-09-12
uid Ubuntu Archive Automatic Signing Key <[email protected]>
sub 2048g/79164387 2004-09-12
pub 4096R/C0B21F32 2012-05-11
uid Ubuntu Archive Automatic Signing Key (2012) <[email protected]>
pub 4096R/EFE21092 2012-05-11
uid Ubuntu CD Image Automatic Signing Key (2012) <[email protected]>
pub 1024D/FBB75451 2004-12-30
uid Ubuntu CD Image Automatic Signing Key <[email protected]>
pub 2048R/DE57BFBE 2014-06-24
uid SaltStack Packaging Team <[email protected]>
sub 2048R/17928113 2014-06-24
root@minion1:~#
Saltstack versions report:
root@minion1:~# salt-minion --versions-report
Salt Version:
Salt: 2016.11.4
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.4.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.8
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.3
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.12 (default, Nov 19 2016, 06:48:10)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.2.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4
System Versions:
dist: Ubuntu 16.04 xenial
machine: x86_64
release: 4.4.0-78-generic
system: Linux
version: Ubuntu 16.04 xenial
root@minion1:~#
And even worse; running it again reveals:
root@saltmaster:/srv/salt/fs# salt minion1 state.apply webserver
minion1:
----------
ID: ondrej-php-ppa
Function: pkgrepo.managed
Result: True
Comment: Configured package repo 'ondrej-php-ppa'
Started: 12:09:35.918318
Duration: 1390.908 ms
Changes:
Summary for minion1
------------
Succeeded: 1
Failed: 0
------------
Total states run: 1
Total run time: 1.391 s
-------------------------------------------
Summary
-------------------------------------------
# of minions targeted: 1
# of minions returned: 1
# of minions that did not return: 0
# of minions with errors: 0
-------------------------------------------
root@saltmaster:/srv/salt/fs#
But the key is actually still missing, so it is actually causing errors when I try to use it.
Trying to add the repository via cmd.run reveals this. Maybe related?
root@saltmaster:/srv/salt/fs# salt minion1 state.apply webserver
minion1:
----------
ID: ondrej-php-ppa
Function: cmd.run
Name: add-apt-repository -y ppa:ondrej/php
Result: False
Comment: Command "add-apt-repository -y ppa:ondrej/php" run
Started: 15:04:49.785624
Duration: 721.442 ms
Changes:
----------
pid:
6732
retcode:
1
stderr:
gpg: keyring `/tmp/tmp_1kydwwk/secring.gpg' created
gpg: keyring `/tmp/tmp_1kydwwk/pubring.gpg' created
gpg: requesting key E5267A6C from hkp server keyserver.ubuntu.com
gpg: /tmp/tmp_1kydwwk/trustdb.gpg: trustdb created
gpg: key E5267A6C: public key "Launchpad PPA for Ond\xc5\x99ej Sur媒" imported
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)
stdout:
'ascii' codec can't decode byte 0xc5 in position 92: ordinal not in range(128)
Summary for minion1
------------
Succeeded: 0 (changed=1)
Failed: 1
------------
Total states run: 1
Total run time: 721.442 ms
-------------------------------------------
Summary
-------------------------------------------
# of minions targeted: 1
# of minions returned: 1
# of minions that did not return: 0
# of minions with errors: 1
-------------------------------------------
ERROR: Minions returned with non-zero exit code
root@saltmaster:/srv/salt/fs#
After lots of debugging, I have found that the problem seems to be with apt-add-repository provided by Ubuntu, so this isn't really a problem with salt. apt-add-repository apparently breaks when it encounters non-unicode characters when executing without an UTF-8 locale:
https://github.com/oerdnj/deb.sury.org/issues/56
So, the workaround I used was to ignore the PPA bit of pkgrepo, and adding the repository like any other repository:
ondrej-php-ppa:
pkgrepo.managed:
- name: deb http://ppa.launchpad.net/ondrej/php/ubuntu xenial main
- dist: xenial
- file: /etc/apt/sources.list.d/ondrej-ubuntu-php-xenial.list
- keyserver: keyserver.ubuntu.com
- keyid: E5267A6C
- refresh_db: true
The result of this is almost exactly the same as using apt-add-repository, except that the imported key ends up in /etc/apt/trusted.gpg instead of a separate file. Using apt-add-repository would import that key into /etc/apt/trusted.gpg.d/ondrej_ubuntu_php.gpg instead, but I would say this is of minor importance.
Bigtime having this issue again. Should be fixed as it shouldn't be a hard fix.
Still an issue.
Yeah and this issue is really getting annoying to have to run cmd.run 'apt-key add' etc every time a new repo needs to be added.
This workaround has been working for us for a while:
php_ppa:
pkgrepo.managed:
- humanname: PHP Repo
- ppa: ondrej/php
- keyid: E5267A6C
- keyserver: keyserver.ubuntu.com
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4F4EA0AAE5267A6C:
cmd.run:
- unless: 'apt-key list | grep Launchpad'
- order: first
php7.1:
pkg:
- installed
- require:
- pkgrepo: php_ppa
- pkgs:
- php7.1-cli
- php7.1-mysql
- php7.1-curl
- php7.1-mcrypt
- php7.1-xmlrpc
- php7.1-gd
- php7.1-imagick
- php7.1-memcached
- php7.1-intl
- php7.1-dev
- php7.1-mbstring
- php7.1-xml
- php7.1-zip
It's because of letters with accents in Ondrej's name... salt seems to not like non ascii 7 bits letters in names associated with ppa.
Can confirm for salt-minion 2017.7.4.
[salt.loaded.int.module.cmdmod:1715][ERROR ][19226] Command '['apt-add-repository', '-y', 'ppa:ondrej/php']' failed with return code: 1
[salt.loaded.int.module.cmdmod:1717][ERROR ][19226] stdout: 'ascii' codec can't decode byte 0xc5 in position 92: ordinal not in range(128)
Manually executing apt-add-repository works fine.
Same issue, still, with Salt 2018.3.0
Many thanks to @jv2222 for the workaround provided Sept 2, 2017.
Seems it was fixed in Ubuntu 18.04, probably still an issue in Ubuntu 16.04. According to the comments above it's an Ubuntu issue.
With Ubuntu 18.04, SaltStack 2018.3.2:
php-repo:
pkgrepo.managed:
- humanname: PHP Repo
- ppa: ondrej/php
adds the key to the list as it should have:
~# apt-key list
/etc/apt/trusted.gpg
--------------------
pub rsa2048 2014-06-24 [SC]
754A 1A7A E731 F165 D5E6 D4BD 0E08 A149 DE57 BFBE
uid [ unknown] SaltStack Packaging Team <[email protected]>
sub rsa2048 2014-06-24 [E]
/etc/apt/trusted.gpg.d/ondrej_ubuntu_php.gpg
--------------------------------------------
pub rsa1024 2009-01-26 [SC]
14AA 40EC 0831 7567 56D7 F66C 4F4E A0AA E526 7A6C
uid [ unknown] Launchpad PPA for Ond艡ej Sur媒
/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-archive.gpg
------------------------------------------------------
pub rsa4096 2012-05-11 [SC]
790B C727 7767 219C 42C8 6F93 3B4F E6AC C0B2 1F32
uid [ unknown] Ubuntu Archive Automatic Signing Key (2012) <[email protected]>
/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg
------------------------------------------------------
pub rsa4096 2012-05-11 [SC]
8439 38DF 228D 22F7 B374 2BC0 D94A A3F0 EFE2 1092
uid [ unknown] Ubuntu CD Image Automatic Signing Key (2012) <[email protected]>
And PHP installs with no issues.
I wanted to report that I'm seeing this issue as well.
(root) ~ # salt --versions
Salt Version:
Salt: 2019.2.2
Dependency Versions:
cffi: 1.5.2
cherrypy: 3.5.0
dateutil: 2.4.2
docker-py: Not Installed
gitdb: 0.6.4
gitpython: 1.0.1
ioflo: Not Installed
Jinja2: 2.8
libgit2: 0.24.0
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.3
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: 2.14
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: 0.24.0
Python: 2.7.12 (default, Oct 8 2019, 14:14:10)
python-gnupg: 0.3.8
PyYAML: 3.11
PyZMQ: 15.2.0
RAET: Not Installed
smmap: 0.9.0
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4
System Versions:
dist: Ubuntu 16.04 xenial
locale: UTF-8
machine: x86_64
release: 4.15.0-1050-aws
system: Linux
version: Ubuntu 16.04 xenial
State:
# manage our salt repository
/etc/apt/sources.list.d/salt-stack.list:
pkgrepo.managed:
- name: deb http://repo.saltstack.com/apt/debian/9/amd64/latest stretch main
- file: /etc/apt/sources.list.d/salt-stack.list
- keyurl: salt://salt-minion/files/SALTSTACK-GPG-KEY.pub
After running this state, package still show as unauthenticated. The key does not show up when running apt-key list
on the target minion. apt update
shows the repository as unauthenticated.
After running wget -O - https://repo.saltstack.com/apt/debian/9/amd64/latest/SALTSTACK-GPG-KEY.pub | apt-key add -
using a cmd.run
call, apt update
returns successfully, and I'm able to manage packages in this repository.
Minion versions:
Salt Version:
Salt: 2016.11.2
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.5.3
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.8
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.8
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pygit2: Not Installed
Python: 2.7.13 (default, Sep 26 2018, 18:42:22)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 16.0.2
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.4.3
ZMQ: 4.2.1
System Versions:
dist: debian 9.4
machine: x86_64
release: 4.19.29
system: Linux
version: debian 9.4
After going through the issue comments, it appears that the issue only presents itself when running salt on 16.04 correct?
With so many issues that need attention, 16.04 reaching EOL in April 2021, and users reporting a working workaround for 16.04, I'm inclined to say we should not waste time fixing this and just let 16.04 reach it's EOL and closing the issue.
Please feel free to disagree with me and let me know why we should still focus on fixing this.
Most helpful comment
After lots of debugging, I have found that the problem seems to be with apt-add-repository provided by Ubuntu, so this isn't really a problem with salt. apt-add-repository apparently breaks when it encounters non-unicode characters when executing without an UTF-8 locale:
https://github.com/oerdnj/deb.sury.org/issues/56
So, the workaround I used was to ignore the PPA bit of pkgrepo, and adding the repository like any other repository:
The result of this is almost exactly the same as using apt-add-repository, except that the imported key ends up in /etc/apt/trusted.gpg instead of a separate file. Using apt-add-repository would import that key into /etc/apt/trusted.gpg.d/ondrej_ubuntu_php.gpg instead, but I would say this is of minor importance.