Salt: Salt-cloud - proxmox driver - returns KeyError: u'name'

[cavepopo]

Description of Issue/Question

When provisioning a vm (either lxc or kvm) on a proxmox cluster, the salt-cloud command fails with this error :

KeyError: u'name'

This happens approximately 1 out of 3 tries, on different proxmox nodes and different site too.

Proxmox are as follow :

Kernel Version Linux 4.15.18-1-pve #1 SMP PVE 4.15.18-15 (Wed, 04 Jul 2018 15:42:56 +0200)
--
PVE Manager Version pve-manager/5.2-5/eb24855a

Steps to Reproduce Issue

just run a standard salt-cloud command based on map files :
sudo /usr/bin/salt-cloud -y -m /etc/salt/map.d/ams_nightly.map

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
           Salt: 2018.3.2

Dependency Versions:
           cffi: 1.11.4
       cherrypy: Not Installed
       dateutil: 2.2
      docker-py: Not Installed
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: 0.26.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.2
   mysql-python: 1.2.3
      pycparser: 2.10
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.26.3
         Python: 2.7.9 (default, Jun 29 2016, 13:08:31)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.4.0
           RAET: Not Installed
          smmap: 0.8.2
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: debian 8.11 
         locale: UTF-8
        machine: x86_64
        release: 4.15.18-1-pve
         system: Linux
        version: debian 8.11 

Logs

Here is the portion of log that seems useful :

18:11:40 [ERROR   ] Failed to create VM ams-nightly.eloquant.build. Configuration value u'name' needs to be set
18:11:40 Traceback (most recent call last):
18:11:40   File "/usr/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 1284, in create
18:11:40     output = self.clouds[func](vm_)
18:11:40   File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/proxmox.py", line 579, in create
18:11:40     if not start(name, vmid, call='action'):
18:11:40   File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/proxmox.py", line 991, in start
18:11:40     if not set_vm_status('start', name, vmid=vmid):
18:11:40   File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/proxmox.py", line 924, in set_vm_status
18:11:40     vmobj = _get_vm_by_id(vmid)
18:11:40   File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/proxmox.py", line 210, in _get_vm_by_id
18:11:40     for vm_name, vm_details in six.iteritems(get_resources_vms(includeConfig=allDetails)):
18:11:40   File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/proxmox.py", line 329, in get_resources_vms
18:11:40     name = resource['name']
18:11:40 KeyError: u'name'

More about this

I can quite easily reproduce this (this has been happening since the very beginning of my salt-cloud experience, around a year ago).

Any ideas or hint about the root cause of this ?

is this due to a mis-communication between the node api and the salt-cloud engine ? some params lost on their way ?

If I launch this map file another time after it has 99% of chance for being correctly handled, sometimes I need 2 or 3 tries before it gets ok.
This issue is particularly blocking in our CI pipeline (we are building container to check the salt states, and this all starts by provisioning a new vm ....)

Thanks

[garethgreenaway]

@cavepopo Thanks for the report. It looks like something may have changed with proxmox from when this was originally implemented. Are you using the same versions across all minions? It looks like the proxmox cloud uses the IPy library, can you see what version is installed on the hosts where you're seeing the issue?

[cavepopo]

Hi, thanks for the feedback, yes I do have the very same versions across all minion/master.
My IPy version is also the same, as :

23:16 $ dpkg -l |grep -i IPy
ii  python-ipy                  1:0.81-1                         all          Python module for handling IPv4 and IPv6 addresses and networks

[cavepopo]

Hi,

Any news about that one ? I am not asking for a solution straight ahead, I am just wondering if there is any ongoing work.

I am not a dev otherwise I'd happily help you guys, let me know if I can be of any help.

Thanks for your great work !

[markjmiller]

I ran into the same issue with salt-cloud and proxmox. The containers would be created through salt-cloud just fine, but would not start. It turns out that I was using invalid MAC addresses in my map file.

On your proxmox server, you can try to start your container in foreground mode and dump the debug output to get more information about what is failing.

lxc-start -n <CONTAINER_NAME> -l DEBUG -F -o /tmp/lxc-start.log

[cavepopo]

Thanks for the input @markjmiller , I do not specify any mac address in my map / profile or provider files, so I doubt a little this to be the root cause...
I did not think about trying to start the container manually (although I use this often for debugging purposes), thanks a lot for reminding me about this ! I'll give it a try to ensure this is not mac address related, and perhaps finding the root cause for this issue.

[cavepopo]

I ran into this just now, and as I suspected the error is not manually reproducible, right after the error I can start the vm through the proxmox webui without any problem.

I highly suspect this issue being caused by proxmox api not getting the "start" command in a correct way, or not handling it the correct way... (note that this is only a feeling and absolutely not based on a analysis :( )

[cavepopo]

Hi, any news or update on this one ?

Thanks

[cavepopo]

Hi,
Please let me know if I can be of any help regarding this issue.

[markjmiller]

Scratch what I said it my last comment. It did help me solve a different issue, but not this one specifically. So this morning I was greeted with my weekly staging server redeploy failing with the exact same error you are seeing. To investigate, I changed line 329 in _/usr/lib/python2.7/dist-packages/salt/cloud/clouds/proxmox.py_ to:

try:
  name = resource['name']
except:
  print(resource)
  exit(0)

And then ran:
salt-cloud -f get_resources_vms <my_proxmox_provider> -l trace

One-by-one it queried each vm in my cluster and then failed on a bad one that returned json without a "name" key. I deleted the bad vm and the issue went away. But I have occasionally seen this issue without a bad vm present, too. I assume there is a race condition where proxmox has not set a name when salt-cloud queries it and the response json doesn't have a "name" key yet.

[cavepopo]

Oh good ! I can't wait to test the new PR.

Thanks a lot @BrianSidebotham

[cavepopo]

Hi,
Would you happen to have eta for this patch ? I saw it was not in 2019.2

Thanks