Salt: Can't Install Salt while provisioning in AWS for Ubuntu Xenial 16.04

[leonkatz]

Description of Issue/Question

Starting yesterday AWS Instance won't install salt after provisioning if the image is a Ubuntu Xenial 16.04.

Setup

W: GPG error: https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 0E08A149DE57BFBE
W: The repository 'https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial InRelease' is not signed.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.

Steps to Reproduce Issue

salt-cloud -m -P mymap.map

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in
Master
Salt Version:
Salt: 2016.11.6

Dependency Versions:
cffi: Not Installed
cherrypy: 3.5.0
dateutil: 2.4.2
docker-py: Not Installed
gitdb: 0.6.4
gitpython: 1.0.1
ioflo: Not Installed
Jinja2: 2.8
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.3
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.12 (default, Nov 19 2016, 06:48:10)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.2.0
RAET: Not Installed
smmap: 0.9.0
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: Ubuntu 16.04 xenial
machine: x86_64
release: 4.4.0-1020-aws
system: Linux
version: Ubuntu 16.04 xenial

[leonkatz]

Found workaround. Added key to the system and recreated the ami in AWS.

wget http://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest/SALTSTACK-GPG-KEY.pub
sudo apt-key add SALTSTACK-GPG-KEY.pub

[Ch3LL]

How are you installing salt? bootstrap? If so are you passing in any script_args in your cloud config?

[leonkatz]

Yes, bootstrap via salt-cloud
no string arg just minion params

  minion:
    master: mymaster
    master_type: str
    environment: dev1
    startup_states: highstate

[leonkatz]

It is happening again intermittently even with the image where I have accepted the key already but now the error is.

 [DEBUG   ] E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
                  E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?

Maybe a race condition?
If I manually run

ssh -t -t -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oControlPath=none -oPasswordAuthentication=no -oChallengeResponseAuthentication=no -oPubkeyAuthentication=yes -oIdentitiesOnly=yes -oKbdInteractiveAuthentication=no -i /home/ubuntu/.ssh/common.pem -p 22 [email protected] 'sudo /tmp/.saltcloud-907f5bba-a355-4fee-8fa9-fbc7e66ef95a/deploy.sh -c '"'"'/tmp/.saltcloud-907f5bba-a355-4fee-8fa9-fbc7e66ef95a'"'"' git v2017.7.0'

It works. But still fails via salt-cloud provisioning the next time. But not every time.

[leonkatz]

Is there any status on this. It should no longer be blocked.

[SJern]

I'm hitting the lock issue as well, interestingly, only with Ubuntu 16.04.3 (from the AWS Deep Learning AMI). The non-deep-learning Ubuntu 16 (16.04.2) AMI works perfectly fine.
I'm still investigating. Would this be a Ubuntu version issue (.2 vs .3)? or an issue arises when the salt-cloud apt-get operations racing with some Deep-Learning related apt-get that still last for a while after the booting of the new ec2 instances?

Update:
I have ruled out any highstate as the cause because the lock issue happens far before any 'SALT/CLOUD//CREATED' event is fired. It seems to have failed at the very first apt-get by salt-cloud.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.