Awx: unable to create instance in openstack with scheduler_hints (out of date openstack python libraries)

[druidushka]

ISSUE TYPE

• Bug Report

COMPONENT NAME

SUMMARY

os_server create task will fail if the scheduler_hints parameter is defined

I am trying to create an instance in openstack(Queens) and it used to work fine until I added scheduler_hints parameter to define anti_affinity group.
After that, I got the error below:
"msg": "BadRequestException: Unknown error"

ENVIRONMENT

• AWX version: 2.0.1
• AWX install method: docker on linux
• Ansible version: 2.7.0
• Operating System: Centos 7.4
• Web Browser: Chrome

STEPS TO REPRODUCE

add a new task into your playbook to provision new instance in openstack with scheduler_hints parameter:
os_server:
name: "{{ instance_name }}"
boot_volume: "{{ instance_boot_volume }}"
cloud: "{{ cloud }}"
scheduler_hints:
group: "{{group_uid}}"
key_name: "{{ key_name }}"
meta:
hostname: "{{ instance_name }}"
groups: "{{ instance.0 | json_query(query_meta) }}"
availability_zone: nova
nics: "net-id=MYNETID"
flavor: "m4.xlarge"
auto_ip: yes
security_groups: default
verify: false

EXPECTED RESULTS

An instance should be created

ACTUAL RESULTS

{
"_ansible_parsed": true,
"exception": "WARNING: The below traceback may not be related to the actual failure.\n File \"/tmp/ansible_os_server_payload__8xXz9/__main__.py\", line 734, in main\n _create_server(module, cloud)\n File \"/tmp/ansible_os_server_payload__8xXz9/__main__.py\", line 524, in _create_server\n *bootkwargs\n File \"\", line 2, in create_server\n File \"/var/lib/awx/venv/ansible/lib/python2.7/site-packages/openstack/cloud/_utils.py\", line 374, in func_wrapper\n return func(args, **kwargs)\n File \"/var/lib/awx/venv/ansible/lib/python2.7/site-packages/openstack/cloud/openstackcloud.py\", line 6763, in create_server\n self.compute.post(endpoint, json={'server': kwargs}))\n File \"/var/lib/awx/venv/ansible/lib/python2.7/site-packages/openstack/_adapter.py\", line 95, in _json_response\n exceptions.raise_from_response(response, error_message=error_message)\n File \"/var/lib/awx/venv/ansible/lib/python2.7/site-packages/openstack/exceptions.py\", line 205, in raise_from_response\n http_status=http_status, request_id=request_id\n",
"_ansible_no_log": false,
"changed": false,
"invocation": {
"module_args": {
"auth_type": null,
"availability_zone": "nova",
"image": null,
"image_exclude": "(deprecated)",
"flavor_include": null,
"cacert": null,
"meta": {
"hostname": "worker1",
"groups": "freeipaclient"
},
"flavor": "c5.large",
"security_groups": [
"default"
],
"scheduler_hints": {
"group": "56559db3-9c84-4596-8ea7-03ebc316fb87"
},
"boot_from_volume": false,
"userdata": null,
"network": null,
"nics": [
"net-id=bc27fc27-65b9-4cb3-a749-f0a8ed7743fc"
],
"floating_ips": null,
"flavor_ram": null,
"volume_size": false,
"state": "present",
"auto_ip": true,
"config_drive": false,
"region_name": null,
"key_name": "os-ansible",
"api_timeout": null,
"auth": null,
"boot_volume": "worker1_volume1",
"key": null,
"interface": "public",
"terminate_volume": false,
"wait": true,
"name": "worker1",
"timeout": 180,
"delete_fip": false,
"cert": null,
"volumes": [],
"verify": false,
"floating_ip_pools": null,
"reuse_ips": true
}
},
"msg": "BadRequestException: Unknown error",
"extra_data": null
}

ADDITIONAL INFORMATION

[wenottingham]

This reads as an issue with the Ansible module itself, not AWX. Please file this at https://github.com/ansible/ansible/issues/. Thanks.

[druidushka]

@wenottingham
Looks like the problem isn't with ansible itself but with openstacksdk version(0.12.0) in the awx_task container.
The same task works fine with Ansible 2.7.0 and openstacksdk version 0.17.0 on my mac.

[gtema]

can the openstacksdk version be raised in AWX now? I also face lots of issues due to old version.

[AlanCoding]

Same as https://github.com/ansible/awx/issues/3306?

[juame]

I believe it's the same as #3306. :relaxed:

From Ansible 2.6 Porting Guide:

This section discusses the behavioral changes between Ansible 2.5 and Ansible 2.6.
...
Openstack modules are no longer using shade library. Instead openstacksdk is used. Since openstacksdk should be already present as a dependency to shade no additional actions are required.

So, Ansible in AWX no longer requires shade. But requires openstacksdk directly:
https://github.com/ansible/awx/blob/3.0.1/requirements/requirements_ansible.in#L54-L55
We also have a lot of troubles with v0.12.0 and would really appreciate if there's a new official image with the latest and greatest openstacksdk :relaxed:

[AlanCoding]

the latest and greatest openstacksdk

This is covered by https://github.com/ansible/awx/pull/3266, see the changes to requirements files there. I was looking at removing shade. We could leave it, but I didn't know of a reason to, it's something I could use feedback on.

[juame]

Do you know if there's a workaround for AWX 3.0.1 in Docker Containers?

I followed the instructions in custom_virtualenvs.md but get following error in awx_task container:

bash-4.2$ sudo virtualenv /var/lib/awx/venv/ansible-openstack
sudo: unknown user: root
sudo: unable to initialize policy plugin

When I try to upgrade OpenStack in venv/ansible:

source /var/lib/awx/venv/ansible/bin/activate
pip install --upgrade openstacksdk
...
Installing collected packages: pbr, os-service-types, decorator, dogpile.cache, netifaces, ipaddress, pycparser, cffi, cryptography, munch, urllib3, idna, certifi, requests, stevedore, keystoneauth1, PyYAML, jsonpatch, jmespath, openstacksdk
  Found existing installation: pbr 3.1.1
    Uninstalling pbr-3.1.1:
Exception:
Traceback (most recent call last):
  File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/pip/req/req_set.py", line 778, in install
    requirement.uninstall(auto_confirm=True)
  File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/pip/req/req_install.py", line 754, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/pip/req/req_uninstall.py", line 115, in remove
    renames(path, new_path)
  File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/pip/utils/__init__.py", line 267, in renames
    shutil.move(old, new)
  File "/usr/lib64/python2.7/shutil.py", line 302, in move
    os.unlink(src)
OSError: [Errno 13] Permission denied: '/var/lib/awx/venv/ansible/bin/pbr'

Any hints welcome ☺️

[ryanpetrello]

@juame

Specify -u0 when you exec into the container:

docker exec -u0 -it tools_awx_1 bash

[juame]

@ryanpetrello thank you! That helped for this!

Can I do it like this (docker commit) or not a good idea? Because I can't use pip on the server directly (no repo, no connectivity):

docker run -d --name awx_task_custom ansible/awx_task:3.0.1
docker exec -u0 -ti awx_task_custom bash

export http_proxy=proxy.example.com:8080
export https_proxy=proxy.example.com:8080

source /var/lib/awx/venv/ansible/bin/activate
pip install --upgrade openstacksdk
unset http_proxy https_proxy
exit  # leave container

docker commit awx_task_custom ansible/awx_task:3.0.1-project
docker tag ansible/awx_task:3.0.1-project project-docker.example.com/ansible/awx_task:3.0.1-project
docker push project-docker.example.com/ansible/awx_task:3.0.1-project

Because then I get on the server:

ERROR: for awx_task_1  Cannot start service task: b'linux spec user: unable to find user root: no matching entries in passwd file'

ERROR: for task  Cannot start service task: b'linux spec user: unable to find user root: no matching entries in passwd file'
ERROR: Encountered errors while bringing up the project.

Update

Found a hint for installing the package via Ansible module pip:

---
- name: Provision Resources on OpenStack
  hosts: localhost
  connection: local
  gather_facts: false
  tasks:
    - pip:
        name: openstacksdk
        version: 0.24.0
        virtualenv: /var/lib/awx/venv/ansible
        extra_args: -i https://example.com/api/pypi/pypi-remote/simple --trusted-host example.com
      tags:
        - awx

Not optimal, but helps us a lot.

[clushie]

Hey, @brtknr's commit on https://github.com/ansible/ansible/commit/c6ed5b314dc211684cfd1342577d034b6de20fa3#diff-08a7b15797f6155ec95b515230231ee8 now breaks this even further, making it impossible to use the os_stack ansible module with the parameter "tag".

Workaround forme now is to run:

sudo docker exec -it awx_task /bin/bash -c '. /var/lib/awx/venv/ansible/bin/activate && \
 pip install --upgrade openstacksdk'

which might has unexpected side-effects.

This two requirement files need to be updated in order to fix this.

• requirements/requirements_ansible.in
• requirements/requirements_ansible.txt

[brtknr]

@clushie I addressed the problem with the fix in https://github.com/ansible/ansible/pull/56710 which was merged to Ansible version 2.8.1. I tried to get Ansible gatekeepers to back port this fix to 2.7.* but it wasn't deemed critical. See https://github.com/ansible/ansible/pull/57100

cc @abadger

[clushie]

Hey @brtknr thank you so much for your immediate response and already tackling down the issue with your fixes.

I've also just created a pull-request with the updated openstacksdk version, I don't know if this makes sense or not. I'm scared of that this updated file possible breaks other things that aren't foreseen. Especially I couldn't quickly find a changelog for the openstacksdk https://pypi.org/project/openstacksdk/#history

[clushie]

Fixed with https://github.com/ansible/awx/releases/tag/6.1.0, it should be safe to close this ticket.

[Spredzy]

Verified and working.