awx surveys and valid nested yaml variables don't appear to work as expected

[v1k0d3n]

ISSUE TYPE

• Bug Report

COMPONENT NAME

• UI (I guess?)

SUMMARY

We're currently exploring AWX, and in our existing playbooks we have nested yaml declarations in group_vars that look like this:

# OS API User Login Information:
auth:
  os_auth_url: "http://openstack.domain.io:5000"
  os_username: "username"

# OS Project Details:
project:
  os_project: "admin"
  os_region_name: "RegionOne"
  os_domain_name: "Default"

...and so on. This is valid yaml syntax, and when used in the playbooks, they're declared like the following just fine (without AWX):

- name: creating new instances and attaching to declared networks
  os_server:
       state: present
       auth:
         auth_url: "{{ auth.os_auth_url }}"
         username: "{{ auth.os_username }}"
         password: "{{ os_password }}"
         project_name: "{{ project.os_project }}"
       name: "{{ instance.os_hostname }}"
       image: "{{ instance.os_image_id }}"
       key_name: "{{ instance.os_key_name }}"
       timeout: 200
       flavor: "{{ instance.os_flavor_id }}"
       network: "{{ network.os_network_id }}"
       meta:
         hostname: "{{ meta.os_hostname }}"
         group: "{{ meta.os_group }}"

This works as expected when running Ansible manually, but when trying to use these same variable declarations in surveys I receive an error that states "Please remove the illegal character from the survey question variable name". I believe this occurs because AWX is not interpreting "." as a valid YAML nested delimiter.

Can someone tell me if there's a work around, or do I really need to redraft our playbooks? We have a lot of playbooks with this format, and I would like to avoid that if possible.

Thanks for the help/guidance in advance!

ENVIRONMENT

• AWX version: Master and also tagged version 1.0.2
• AWX install method: docker
• Ansible version: 2.4.x
• Operating System: Linux
• Web Browser: Firefox and Chrome (same results)

STEPS TO REPRODUCE

Details are included above...

EXPECTED RESULTS

Details are included above...

ACTUAL RESULTS

Details are included above...

ADDITIONAL INFORMATION

None

[sonjz]

Running into same problem...

I think its just space-delimiting the survey, so --extra-vars 'os_version=somevalue auth.auth_url=someurl' wouldn't work.

Logic would have to be nested variable aware, so it could be building a JS Object in the back to execute JSON inline --extra-vars '{"os_version": "somevalue", "auth": {"auth_url" : "someurl"}' or create a YAML config that it can run as a temp file --extra-vars '@survey.yaml'

Another thing to consider is if Answer Variable Name would accept auth.auth_url implicitly or would it remain the same, and you introduce sections that you could assign a variable explicitly

Survey Section
_Description:_ Authorization
_Section Variable:_ auth
Questions: ...

Sections would work, but might become clumbersome if your playbooks vars are really nested. And how many nested sections would be allowed ...

[sonjz]

For anyone interested, I created a very ugly, verbose, limited hack, that works ... did I mention ugly?
I didn't want to rebuild the awx_web container, so I decided to go native.

_(I contemplated running a command against node - command: node some.js {{ dictionary }}, as Javascript has a really easy way to build JSON from dot-notation... obviously a py module would be best)_

Setting up survey variables

Since we are limited with characters we can use for survey variables, I'm using __ in place of . for dot notation for classes.

company__environment                    # company.environment
company__ec2_ami__instance_id           # company.ec2_ami.instance_id

roles/common/tasks/main.yml

In my playbook, I created a roles/common/tasks/main.yml. I filter down _all_ the variables to ones with __ and pass them into roles/common/tasks/awx_survey_hack.yml

- name: Create awx_survey from '__' vars with AWX Survey Hack
  include_tasks: awx_survey_hack.yml
  with_items: "{{ hostvars[inventory_hostname].keys() | select('search', '__') | list}}"
  loop_control:
    loop_var: key
    label: "Setting {{ key }}"

roles/common/tasks/awx_survey_hack.yml

Here's the ugly part, I couldn't figure out how to do it recursively with loops, jinja, and sans an eval approach... so I broke down and created something that would have got me a 50% in computer class.

This has a bunch of conditionals for up to 10 levels of nesting. This means lots of skips will be reported.

_(Unfortunately, no_log only works per-action, not at include_tasks level. And skippy plugin is only at ansible host level, not definable in the playbook)_

I split on __ and based on the number of levels, the appropriate set_fact is selected. It is appended to the awx_survey object using combine.

---
- name: "Set key {{ key }}"
  set_fact:
    survey_name_raw: "{{ key }}"
- name: Set key split
  set_fact:
    survey_name: "{{ survey_name_raw.split('__') }}"
- name: Set value
  set_fact:
    survey_value: "{{ hostvars[inventory_hostname][survey_name_raw] }}"

- name: Set value quotes
  set_fact:
    survey_value_quote: "\""
  when: survey_value is not number
- name: Set value quotes
  set_fact:
    survey_value_quote: ""
  when: survey_value is number

- name: Set 1 level
  set_fact:
    survey_result: "{
      '{{ survey_name[0] }}': {{survey_value_quote}}{{ survey_value }}{{survey_value_quote}}
    }"
  when: (survey_name | length) == 1

- name: Set 2 levels
  set_fact:
    survey_result: "{
      '{{ survey_name[0] }}': {
        '{{ survey_name[1] }}': {{survey_value_quote}}{{ survey_value }}{{survey_value_quote}}
    }}"
  when: (survey_name | length) == 2

- name: Set 3 levels
  set_fact:
    survey_result: "{
      '{{ survey_name[0] }}': {
        '{{ survey_name[1] }}': {
          '{{ survey_name[2] }}': {{survey_value_quote}}{{ survey_value }}{{survey_value_quote}}
    }}}"
  when: (survey_name | length) == 3

- name: Set 4 levels
  set_fact:
    survey_result: "{
      '{{ survey_name[0] }}': {
        '{{ survey_name[1] }}': {
          '{{ survey_name[2] }}': {
            '{{ survey_name[3] }}': {{survey_value_quote}}{{ survey_value }}{{survey_value_quote}}
    }}}}"
  when: (survey_name | length) == 4

- name: Set 5 levels
  set_fact:
    survey_result: "{
      '{{ survey_name[0] }}': {
        '{{ survey_name[1] }}': {
          '{{ survey_name[2] }}': {
            '{{ survey_name[3] }}': {
              '{{ survey_name[4] }}': {{survey_value_quote}}{{ survey_value }}{{survey_value_quote}}
    }}}}}"
  when: (survey_name | length) == 5

- name: Set 6 levels
  set_fact:
    survey_result: "{
      '{{ survey_name[0] }}': {
        '{{ survey_name[1] }}': {
          '{{ survey_name[2] }}': {
            '{{ survey_name[3] }}': {
              '{{ survey_name[4] }}': {
                '{{ survey_name[5] }}': {{survey_value_quote}}{{ survey_value }}{{survey_value_quote}}
    }}}}}}"
  when: (survey_name | length) == 6

- name: Set 7 levels
  set_fact:
    survey_result: "{
      '{{ survey_name[0] }}': {
        '{{ survey_name[1] }}': {
          '{{ survey_name[2] }}': {
            '{{ survey_name[3] }}': {
              '{{ survey_name[4] }}': {
                '{{ survey_name[5] }}': {
                  '{{ survey_name[6] }}': {{survey_value_quote}}{{ survey_value }}{{survey_value_quote}}
    }}}}}}}"
  when: (survey_name | length) == 7

- name: Set 8 levels
  set_fact:
    survey_result: "{
      '{{ survey_name[0] }}': {
        '{{ survey_name[1] }}': {
          '{{ survey_name[2] }}': {
            '{{ survey_name[3] }}': {
              '{{ survey_name[4] }}': {
                '{{ survey_name[5] }}': {
                  '{{ survey_name[6] }}': {
                    '{{ survey_name[7] }}': {{survey_value_quote}}{{ survey_value }}{{survey_value_quote}}
    }}}}}}}}"
  when: (survey_name | length) == 8

- name: Set 9 levels
  set_fact:
    survey_result: "{
      '{{ survey_name[0] }}': {
        '{{ survey_name[1] }}': {
          '{{ survey_name[2] }}': {
            '{{ survey_name[3] }}': {
              '{{ survey_name[4] }}': {
                '{{ survey_name[5] }}': {
                  '{{ survey_name[6] }}': {
                    '{{ survey_name[7] }}': {
                      '{{ survey_name[8] }}': {{survey_value_quote}}{{ survey_value }}{{survey_value_quote}}
    }}}}}}}}}"
  when: (survey_name | length) == 9

- name: Set 10 levels
  set_fact:
    survey_result: "{
      '{{ survey_name[0] }}': {
        '{{ survey_name[1] }}': {
          '{{ survey_name[2] }}': {
            '{{ survey_name[3] }}': {
              '{{ survey_name[4] }}': {
                '{{ survey_name[5] }}': {
                  '{{ survey_name[6] }}': {
                    '{{ survey_name[7] }}': {
                      '{{ survey_name[8] }}': {
                        '{{ survey_name[9] }}': {{survey_value_quote}}{{ survey_value }}{{survey_value_quote}}
    }}}}}}}}}}"
  when: (survey_name | length) == 10

- fail:
    msg: This awx_survey hack can handle up to 10 levels, add more if you want...
  when: (survey_name | length) > 10

- debug:
    var: survey_result
    verbosity: 3

- name: Set awx_survey
  set_fact:
    awx_survey: "{{ awx_survey | default({}) | combine(survey_result, recursive=True) }}"

- debug:
    var: awx_survey
    verbosity: 3

In the end you'll be able to access your nested variables from awx_survey. Personally, I have already setup my playbooks to access company__some_variable as mentioned, so I have an extra step in my roles/common/tasks/main.yml to handle the merge

# NOTE: there might be more generalized way to bind to hostvars[inventory_hostname] directly...
- name: Set awx_survey.company to company
  set_fact:
    company: "{{ company | default({}) | combine(awx_survey.company, recursive=True) }}"
  when: awx_survey is defined and awx_survey.company is defined

- debug:
    var: company
    verbosity: 3

[daemenseth]

is there some progression to fix this issue without the ugly hack?

[adpavlov]

Same here with AWX 6.1.0
Though in yaml format EXTRA VARIABLES page shows everything correct

servers: |-
  server1:
    interface1: eth4
    interface2: eth5
  server2:
    interface1: eth4
    interface2: eth5

During task execution it can't access var DEVICE={{ servers[inventory_hostname].interface1 }}

"msg": "'ansible.utils.unsafe_proxy.AnsibleUnsafeText object' has no attribute u'server1'"

Issue is almost 2 years old, are there any plans to fix it?

[givaldolins]

I am using AWX 11.0.0 and I am also getting the same error. Is there any fix in the road map for this?

[lwilliams1990]

Issue still persists in AWX 11.1.0