salt-ssh + salt-cloud: cloud roster not working and/or `update_cachedir` is broken

Created on 8 Nov 2017 · 16Comments · Source: saltstack/salt

Description of Issue/Question

The goal of the exercise is to provision a machine using salt-cloud and then put it into a desired state using salt-ssh. The --roster=cloud option was added to salt-ssh to assist with this. Using the script (below), we see the following behavior:

Without update_cachedir: true:
- [ERROR ] No matching targets found in roster.
- VM is successfully destroyed
With update_cachedir: true:
- Stack trace below.
- VMs destruction fails with No machines were found to be destroyed

[ERROR   ] An un-handled exception was caught by salt's global exception handler:
TypeError: a bytes-like object is required, not 'str'
Traceback (most recent call last):
  File "/home/bar/Dev/foo/build7/_venv/bin/salt-ssh", line 11, in <module>
    load_entry_point('salt==2017.7.2', 'console_scripts', 'salt-ssh')()
  File "/home/bar/Dev/foo/build7/_venv/lib/python3.5/site-packages/salt/scripts.py", line 416, in salt_ssh
    client.run()
  File "/home/bar/Dev/foo/build7/_venv/lib/python3.5/site-packages/salt/cli/ssh.py", line 20, in run
    ssh = salt.client.ssh.SSH(self.config)
  File "/home/bar/Dev/foo/build7/_venv/lib/python3.5/site-packages/salt/client/ssh/__init__.py", line 230, in __init__
    self.tgt_type)
  File "/home/bar/Dev/foo/build7/_venv/lib/python3.5/site-packages/salt/roster/__init__.py", line 105, in targets
    targets.update(self.rosters[f_str](tgt, tgt_type))
  File "/home/bar/Dev/foo/build7/_venv/lib/python3.5/site-packages/salt/roster/cloud.py", line 45, in targets
    minions = __runner__['cache.cloud'](tgt)
  File "/home/bar/Dev/foo/build7/_venv/lib/python3.5/site-packages/salt/runners/cache.py", line 402, in cloud
    cloud_cache = __utils__['cloud.list_cache_nodes_full'](opts=opts, provider=provider)
  File "/home/bar/Dev/foo/build7/_venv/lib/python3.5/site-packages/salt/utils/cloud.py", line 2768, in list_cache_nodes_full
    minions[driver][prov][minion_id] = msgpack.load(fh_)
  File "msgpack/_unpacker.pyx", line 164, in msgpack._unpacker.unpack (msgpack/_unpacker.cpp:2622)
  File "msgpack/_unpacker.pyx", line 124, in msgpack._unpacker.unpackb (msgpack/_unpacker.cpp:1918)
TypeError: a bytes-like object is required, not 'str'
Traceback (most recent call last):
  File "/home/bar/Dev/foo/build7/_venv/bin/salt-ssh", line 11, in <module>
    load_entry_point('salt==2017.7.2', 'console_scripts', 'salt-ssh')()
  File "/home/bar/Dev/foo/build7/_venv/lib/python3.5/site-packages/salt/scripts.py", line 416, in salt_ssh
    client.run()
  File "/home/bar/Dev/foo/build7/_venv/lib/python3.5/site-packages/salt/cli/ssh.py", line 20, in run
    ssh = salt.client.ssh.SSH(self.config)
  File "/home/bar/Dev/foo/build7/_venv/lib/python3.5/site-packages/salt/client/ssh/__init__.py", line 230, in __init__
    self.tgt_type)
  File "/home/bar/Dev/foo/build7/_venv/lib/python3.5/site-packages/salt/roster/__init__.py", line 105, in targets
    targets.update(self.rosters[f_str](tgt, tgt_type))
  File "/home/bar/Dev/foo/build7/_venv/lib/python3.5/site-packages/salt/roster/cloud.py", line 45, in targets
    minions = __runner__['cache.cloud'](tgt)
  File "/home/bar/Dev/foo/build7/_venv/lib/python3.5/site-packages/salt/runners/cache.py", line 402, in cloud
    cloud_cache = __utils__['cloud.list_cache_nodes_full'](opts=opts, provider=provider)
  File "/home/bar/Dev/foo/build7/_venv/lib/python3.5/site-packages/salt/utils/cloud.py", line 2768, in list_cache_nodes_full
    minions[driver][prov][minion_id] = msgpack.load(fh_)
  File "msgpack/_unpacker.pyx", line 164, in msgpack._unpacker.unpack (msgpack/_unpacker.cpp:2622)
  File "msgpack/_unpacker.pyx", line 124, in msgpack._unpacker.unpackb (msgpack/_unpacker.cpp:1918)
TypeError: a bytes-like object is required, not 'str'
EXIT 1

Setup

MINION_NAME='foobar-08'

AWS_ID='redacted'
AWS_KEY='redacted'
AWS_KEYNAME='redacted'
AWS_PRIVATE_KEY='redacted'
AWS_PROFILE_NAME='redacted'
AWS_SECURITYGROUP='redacted'

LOG_LVL=all

ROOT_DIR=$(mktemp -d)
CONFIG_DIR=$ROOT_DIR/etc/salt
FILE_ROOT_BASE=$(mktemp -d)

# config

mkdir -p $CONFIG_DIR

cat <<EOF > $CONFIG_DIR/cloud
root_dir: $ROOT_DIR
log_file: $ROOT_DIR/var/log/salt/cloud
update_cachedir: True
EOF

cat <<EOF > $CONFIG_DIR/cloud.providers
my-aws-default:
  id: '$AWS_ID'
  key: '$AWS_KEY'
  keyname: '$AWS_KEYNAME'
  private_key: '$AWS_PRIVATE_KEY'
  securitygroup: '$AWS_SECURITYGROUP'
  driver: ec2
EOF

cat <<EOF > $CONFIG_DIR/cloud.profiles
$AWS_PROFILE_NAME:
  provider: my-aws-default
  size: t2.micro
  image: ami-7b4d7900
  script: salt-bootstrap
  ssh_username: ubuntu
EOF

cat <<EOF > $CONFIG_DIR/master
root_dir: $ROOT_DIR
EOF

# run stuff

SLEEP=5

OPTS="--log-file-level=$LOG_LVL --config-dir=$CONFIG_DIR"

salt-master $OPTS --daemon
echo "EXIT $?"
sleep $SLEEP

salt-cloud $OPTS -p $AWS_PROFILE_NAME $MINION_NAME > /dev/null
echo "EXIT $?"
sleep $SLEEP

salt-cloud $OPTS -Q | fgrep $MINION_NAME
echo "EXIT $?"
sleep $SLEEP

salt-ssh $OPTS --roster=cloud -i -r 'uname -a' $MINION_NAME
echo "EXIT $?"
sleep $SLEEP

salt-ssh $OPTS --roster=cache -i -r 'uname -a' $MINION_NAME
echo "EXIT $?"
sleep $SLEEP

salt-cloud $OPTS -d $MINION_NAME -y
echo "EXIT $?"
sleep $SLEEP

pkill -F $ROOT_DIR/var/run/salt-master.pid
rm -rf $ROOT_DIR $FILE_ROOT_BASE

Steps to Reproduce Issue

Run the script with update_cachedir toggled.

Versions Report

Salt Version:
           Salt: 2017.7.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.7
          ioflo: Not Installed
         Jinja2: 2.9.6
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.5.3 (default, Sep 14 2017, 22:58:41)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.3
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.2
            ZMQ: 4.1.6

System Versions:
           dist: Ubuntu 17.04 zesty
         locale: UTF-8
        machine: x86_64
        release: 4.12.0-041200rc6-generic
         system: Linux
        version: Ubuntu 17.04 zesty

Bug P1 Salt-Cloud fixed-pending-your-verification severity-medium team-cloud

Source

brianthelion

All 16 comments

I am not super surprised that the cachedir stuff in salt-cloud is not working with py3 yet.

@techhat you wanna take a look at your msgpack stuff here, it might be time to look at using the cache subsystem for that.

Thanks,
Daniel

gtmanfred on 9 Nov 2017

@techhat I reviewed the source in salt/utils/cloud.py and a problem is that files passed to msgpack aren't being opened as binary. It looks like @rallytime started the Py3 port

https://github.com/saltstack/salt/blob/8e99aeed6ac69461cfb3fa1549800ff4b8260d98/salt/utils/cloud.py#L2590-L2593

but didn't finish it

https://github.com/saltstack/salt/blob/8e99aeed6ac69461cfb3fa1549800ff4b8260d98/salt/utils/cloud.py#L2800-L2803

If the appropriate mode checks are added, the code no longer crashes BUT

The roster still comes up empty
salt-cloud --destroy returns No machines were found to be destroyed

brianthelion on 9 Nov 2017

Digression: Is there a bounty system for salt? I'd be happy to pay to have some of my bugs fixed.

brianthelion on 9 Nov 2017

@gtmanfred @techhat @rallytime Any thoughts here? I'm thoroughly blocked.

brianthelion on 13 Nov 2017

We do not have a bounty program at this time.

As for the real problem, I think that we should move all the cache storage stuff over to using the salt cache subsystem https://github.com/saltstack/salt/tree/2017.7/salt/cache , which comes with a localfs subsystem by default that uses msgpack and does py3 correctly. then we could rip out the extra caching logic in salt-cloud.

But I don't have time right now to do this.

Thanks,
Daniel

gtmanfred on 13 Nov 2017

@gtmanfred Just to clarify, can you confirm that this is indeed a supposed to work as I've described? I want to make sure that this is a bug and that I'm not abusing salt here. From there, I will probably start looking for a workaround. Thanks!

brianthelion on 13 Nov 2017

cachedir and stuff should be supported in py3 yes.

But Some of the py3 stuff that isn't fully tested looks to have slipped through the cracks in salt-cloud. The best way to fix it would be to do the migration to the cache subsystem to replace the stuff doing the cachedir on py3.

gtmanfred on 13 Nov 2017

@gtmanfred Just a reminder to update to the "bug" label. Thanks!

brianthelion on 15 Nov 2017

Thanks sorry about that.

gtmanfred on 15 Nov 2017

👍1

@gtmanfred Any updates on the cloud cache? I am now blocked on this.

brianthelion on 31 Dec 2017

It is not something I am working on right now. I want to migrate to using the caching backend for cloud cache. Until then I would recommend using salt with python 2 for now

gtmanfred on 31 Dec 2017

@gtmanfred I am stuck on Py3 unfortunately. Is there anything I can read up on to learn about how the caching backend is supposed to work? If I can understand what you intend, I may be able to get a serviceable implementation together on my own. A design doc would be super helpful.

brianthelion on 31 Dec 2017

Unfortunately there is not a lot of documentation about it, so it is going to take quite a bit of code diving.

You can try and look through how the minion data cache uses it. https://docs.saltstack.com/en/latest/topics/cache/

Daniel

gtmanfred on 2 Jan 2018

👍1

I've tracked this down to a handful of functions using msgpack in salt/utils/cloud.py. From testing, looks like anyplace calling msgpack.dump needs the stream opened as binary (several examples of this in the same file). Then anywhere msgpack.load is being called needs the handler opened in binary and also encoding='utf-8' keyword argument passed to load. Working on a patch for this now.

dwoz on 7 Jan 2018

@brianthelion Are you able to try this patch? Also, what other bugs would you be looking to throw a bounty on? I may be able to help.

dwoz on 8 Jan 2018

@dwoz Sorry for not watching this thread more closely; I've been away from devops for a few weeks. I can probably put some time on that later in the week. As for bounties, fixing https://github.com/saltstack/salt/issues/42493 would make my life MUCH better.

brianthelion on 8 Jan 2018

Was this page helpful?

0 / 5 - 0 ratings