salt-cloud : not able to create new container on proxmox

[cavepopo]

Description of Issue/Question

Based on the salt-cloud example I am trying to create a new container (lxc), but this failed with a :

The following exception was thrown when trying to run the initial deployment: 
400 Client Error: Parameter verification failed.

Setup

root@salt [~]: cat  /etc/salt/cloud.providers.d/proxmox.conf 
my-proxmox-config:
  # Set up the location of the salt master
  #
  minion:
    master: salt.com

  # Set the PROXMOX access credentials (see below)
  #
  user: root@pam
  password: password

  # Set the access URL for your PROXMOX host
  #
  url: proxmox.com
  driver: proxmox
  verify_ssl: False

cat  /etc/salt/cloud.profiles.d/debian_minimal.conf 
proxmox-debian:
    provider: my-proxmox-config
    image: local:vztmpl/debian-8.0-x86_64-minimal.tar.gz
    technology: lxc
    # host needs to be set to the configured name of the proxmox host
    # and not the ip address or FQDN of the server
    host: proxmox # name visible from the cluster webadmin here
    ip_address: <ip of the proxmox server here>
    password: password

Steps to Reproduce Issue

Try to create a lxc using a basic config based on the salt-cloud doc (https://docs.saltstack.com/en/latest/topics/cloud/proxmox.html#getting-started-with-proxmox), I got this debug log :

[DEBUG   ] Generating minion keys for 'debianTEST'
[DEBUG   ] LazyLoaded cloud.fire_event
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Sending event: tag = salt/cloud/debianTEST/creating; data = {'profile': 'proxmox-debian', 'event': 'starting create', '_stamp': '2017-03-24T10:57:34.927275', 'name': 'debianTEST', 'provider': 'my-proxmox-config:proxmox'}
[INFO    ] Creating Cloud VM debianTEST
[DEBUG   ] Not authenticated yet, doing that now..
[WARNING ] /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:770: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)

[DEBUG   ] get: https://proxmox.com:8006/api2/json/cluster/nextid (None)
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Sending event: tag = salt/cloud/debianTEST/requesting; data = {'_stamp': '2017-03-24T10:57:35.046782', 'event': 'requesting instance', 'kwargs': {'password': 'password', 'hostname': 'debianTEST', 'vmid': 163, 'ostemplate': 'local:vztmpl/debian-8.0-x86_64-minimal.tar.gz', 'net0': 'bridge=vmbr0,ip=192.168.123.236/24,name=eth0,type=veth'}}
[DEBUG   ] Preparing to generate a node using these parameters: {'password': 'password', 'hostname': 'debianTEST', 'vmid': 163, 'ostemplate': 'local:vztmpl/debian-8.0-x86_64-minimal.tar.gz', 'net0': 'bridge=vmbr0,ip=192.168.123.236/24,name=eth0,type=veth'} 
[DEBUG   ] post: https://proxmox:8006/api2/json/nodes/tb1/lxc ({'password': 'password', 'hostname': 'debianTEST', 'vmid': 163, 'ostemplate': 'local:vztmpl/debian-8.0-x86_64-minimal.tar.gz', 'net0': 'bridge=vmbr0,ip=192.168.123.236/24,name=eth0,type=veth'})
[ERROR   ] Error creating debianTEST on PROXMOX

The following exception was thrown when trying to run the initial deployment: 
400 Client Error: Parameter verification failed.
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/proxmox.py", line 547, in create
    data = create_node(vm_, newid)
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/proxmox.py", line 728, in create_node
    node = query('post', 'nodes/{0}/{1}'.format(vmhost, vm_['technology']), newnode)
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/proxmox.py", line 178, in query
    response.raise_for_status()
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 851, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 400 Client Error: Parameter verification failed.
Error: There was a profile error: Failed to deploy VM

Versions Report

root@salt [~]: salt --versions-report
Salt Version:
           Salt: 2016.11.3

Dependency Versions:
           cffi: 0.8.6
       cherrypy: Not Installed
       dateutil: 2.2
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.2
   mysql-python: Not Installed
      pycparser: 2.10
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.9 (default, Jun 29 2016, 13:08:31)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.4.0
           RAET: Not Installed
          smmap: 0.8.2
        timelib: Not Installed
        Tornado: 4.4.2
            ZMQ: 4.0.5

System Versions:
           dist: debian 8.7 
        machine: x86_64
        release: 4.4.35-1-pve
         system: Linux
        version: debian 8.7 

root@salt [~]: dpkg -l |grep -E "python-ipy|python-requests"
ii  python-ipy                  1:0.81-1                  all          Python module for handling IPv4 and IPv6 addresses and networks
ii  python-requests             2.7.0-3                   all          elegant and simple HTTP library for Python2, built for human beings

Any tips are welcome :)

[gtmanfred]

Unfortunately, Proxmox is not a driver that we have an account for, so there is not much I can do to help.

@hobgoblinsmaster it looks like you have been making changes to Proxmox recently, and @marnovdm, you appear to have extended the lxc stuff last year according to the history on that file.

Would either of yall be able to help @cavepopo

Thanks!
Daniel

[cavepopo]

Thanks for you input @gtmanfred , I am actually firstly looking for someone who get this simple feature to work (just to be sure that this is possible at the moment to automatically create a lxc on a proxmox cluster), and then maybe a fix to my config to make it works too.

Additionnal info about my setup :
Virtual Environment 4.4-5/c43015a5

UPDATE :
I tried on my prod env just to ensure the same behaviour , and guess what ? It works down there, here is the PVE version :
Virtual Environment 4.4-12/e71b7a74

Thanks

[cavepopo]

Hi,

After some time I was finally able to deploy a correctly configured LXC, but only on my prod env and still not working on my dev env which I upgraded but this did not fix my issue.

Anyway, for the people looking for some info, I found out that the example given at (https://docs.saltstack.com/en/latest/topics/cloud/proxmox.html#getting-started-with-proxmox) were no really comprehensive regarding the possible and the correct syntax for the LXC parameters. The best place to look out for correct syntax and available parameter (I guess they are not all available but still) is the proxmox API doc which is :
http://pve.proxmox.com/pve-docs/api-viewer/index.html

this did not fix my dev env issue by the way, so I am still looking for the root cause there.

[cavepopo]

Hi,

I finally found the root cause for my not working salt-cloud on my dev env : It was "just" a storage issue.

I did not explicitely declare the storage to be sued by salt-cloud (in the profile.conf), so it default to the "local" storage which does not allow the container.

I did have to use a manual test procedure (curl'ing straight to the proxmox API) to get the actual error :
{"errors":{"storage":"storage 'local' does not support container directories"},"data":null}

This error is actually replaced by a somewhat generic and not-so-explicit :
The following exception was thrown when trying to run the initial deployment: 400 Client Error: Parameter verification failed. Error: There was a profile error: Failed to deploy VM
Dunno if it's possible but returning the actual error (if any) from the proxmox node would be useful in some case.

Bye

[h3]

@cavepopo thanks so much for taking the time to document your progress, I was about to throw the towel.

I agree that the error handling is painfully missing from the Proxmox provider ... it should be improved.

I'll leave my debugging snippets here for the next poor Googler:

import sys
import json
from pygments import highlight, lexers, formatters
formatted_json = json.dumps(json.loads(response.content), indent=4)
colorful_json = highlight(unicode(formatted_json, 'UTF-8'), lexers.JsonLexer(), formatters.TerminalFormatter())
print(colorful_json)

If you add the above snippet to
/usr/lib/python2.7/dist-packages/salt/cloud/clouds/proxmox.py in the query
method, after the response.raise_for_status() call.

Here's a bash one liner that perform the insert, if it does not work make sure the line number is still good (182):

sed -i '182i# hack ..\n    import sys\n    import json\n    from pygments import highlight, lexers, formatters\n    formatted_json = json.dumps(json.loads(response.content), indent=4)\n    colorful_json = highlight(unicode(formatted_json, "UTF-8"), lexers.JsonLexer(), formatters.TerminalFormatter())\n    print(colorful_json)' /usr/lib/python2.7/dist-packages/salt/cloud/clouds/proxmox.py

Now you can see pretty API replies when calling salt-cloud:

# salt-cloud -p ubuntu ubuntu-test
{
    "data": [
        {
            "node": "pve", 
            "uptime": 23908, 
            "maxcpu": 8, 
            "level": "", 
            "mem": 1343168512, 
            "cpu": 0.0012966504992602, 
            "disk": 2141577216, 
            "type": "node", 
            "id": "node/pve", 
            "maxdisk": 12879781888, 
            "maxmem": 8368082944
        }, 
        {
            "node": "pve", 
            "storage": "local-lvm", 
            "disk": 0, 
            "type": "storage", 
            "id": "storage/pve/local-lvm", 
            "maxdisk": 29393682432
        }, 
        {
            "node": "pve", 
            "storage": "local", 
            "disk": 2141577216, 
            "type": "storage", 
            "id": "storage/pve/local", 
            "maxdisk": 12879781888
        }
    ]
}

{
    "data": "100"
}
# [ERROR   ] Error creating ubuntu-test on PROXMOX

# The following exception was thrown when trying to run the initial deployment: 
# 400 Client Error: Parameter verification failed. for url: https://pve.lan:8006/api2/json/nodes/pve/lxc
# Error: There was a profile error: Failed to deploy VM

Useful curl API calls

Testing params via raw REST call is always a good idea:

Get cookie

$ curl --silent --insecure --data "username=root@pam&password=vagrant" \
https://pve.lan:8006/api2/json/access/ticket | python -m json.tool \
| grep "ticket" | awk '{print substr($2,2,length($2)-3)}' | sed 's/^/PVEAuthCookie=/' > cookie

Get csrftoken

curl --silent --insecure --data "username=root@pam&password=vagrant" \
https://pve.lan:8006/api2/json/access/ticket | python -m json.tool \
| grep 'CSRFPreventionToken' | awk '{print substr($2,2,length($2)-3)}' \
| sed 's/^/CSRFPreventionToken:/' > csrftoken

Create CT

curl --silent --insecure  --cookie "$(<cookie)" --header "$(<csrftoken)" -X POST \
--data-urlencode net0="name=testcontainer,bridge=vmbr0" \
--data-urlencode ostemplate="local:vztmpl/ubuntu-16.04-standard_16.04-1_amd64.tar.gz" \
--data-urlencode storage="local-lvm" \
--data vmid=101 \
https://pve.lan:8006/api2/json/nodes/pve/lxc

Task status

curl --silent --insecure  --cookie "$(<cookie)" --header "$(<csrftoken)" -X GET \
https://pve.lan:8006/api2/json/nodes/pve/tasks/UPID:pve:00001055:00314AEF:5BBE3156:vzcreate:101:root@pam:/status \
| python -m json.tool

Task log

curl --silent --insecure  --cookie "$(<cookie)" --header "$(<csrftoken)" -X GET \
https://pve.lan:8006/api2/json/nodes/pve/tasks/UPID:pve:00001055:00314AEF:5BBE3156:vzcreate:101:root@pam:/log \
| python -m json.tool

[psy0rz]

I had to add it BEFORE the response.raise_for_status()

Also i had multiple errors. (too short password, "storage 'local' does not support container directories")

Error handing really needs to be improved! Even -l all didn't show any clue at all. Also the logs on the proxmox server didnt show anything.

This module is basically impossible to use, unless you get everything right 100% on the first try. (or find this post and add these horrible hacks)