Awx: Custom env with python 3.6 on AWX 3.0.0

[roman-vynar]

AWX 3.0.0.
Running in Docker.

Trying to create a custom venv:

yum -y install python36-devel gcc
python3.6 -m venv /var/lib/awx/venv/my-custom-venv
source /var/lib/awx/venv/my-custom-venv/bin/activate
pip install docker python-memcached psutil ansible boto3 boto

Running Inventory sync and gettting:

1.570 INFO     Updating inventory 3: roman cluster
    1.584 INFO     Reading Ansible inventory source: /var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/plugins/inventory/ec2.py
    1.586 INFO     Using VIRTUAL_ENV: /var/lib/awx/venv/my-custom-venv/
    1.586 INFO     Using PATH: /var/lib/awx/venv/my-custom-venv/bin:/var/lib/awx/venv/awx/bin:/var/lib/awx/venv/awx/bin:/var/lib/awx/venv/awx/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    1.586 INFO     Using PYTHONPATH: /var/lib/awx/venv/my-custom-venv/lib/python3.6/site-packages:
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/bin/awx-manage", line 11, in <module>
    load_entry_point('awx==3.0.0.0', 'console_scripts', 'awx-manage')()
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/__init__.py", line 150, in manage
    execute_from_command_line(sys.argv)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 1030, in handle
    raise exc
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 930, in handle
    data = AnsibleInventoryLoader(source=source, is_custom=self.is_custom, venv_path=venv_path).load()
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 176, in load
    return self.command_to_json(base_args + ['--list'])
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 159, in command_to_json
    self.method, proc.returncode, stdout, stderr))
RuntimeError: ansible-inventory failed (rc=1) with stdout:
stderr:
Traceback (most recent call last):
  File "/usr/bin/ansible-inventory", line 67, in <module>
    import ansible.constants as C
  File "/var/lib/awx/venv/my-custom-venv/lib/python3.6/site-packages/ansible/constants.py", line 17, in <module>
    from ansible.config.manager import ConfigManager, ensure_type, get_ini_config_value
  File "/var/lib/awx/venv/my-custom-venv/lib/python3.6/site-packages/ansible/config/manager.py", line 16, in <module>
    from yaml import load as yaml_load
  File "/var/lib/awx/venv/my-custom-venv/lib/python3.6/site-packages/yaml/__init__.py", line 284
    class YAMLObject(metaclass=YAMLObjectMetaclass):
                              ^
SyntaxError: invalid syntax

Trying to fix this manually by removing metaclass=.
Getting another error:

  2.215 INFO     Updating inventory 3: roman cluster
    2.237 INFO     Reading Ansible inventory source: /var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/plugins/inventory/ec2.py
    2.238 INFO     Using VIRTUAL_ENV: /var/lib/awx/venv/my-custom-venv/
    2.238 INFO     Using PATH: /var/lib/awx/venv/my-custom-venv/bin:/var/lib/awx/venv/awx/bin:/var/lib/awx/venv/awx/bin:/var/lib/awx/venv/awx/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    2.238 INFO     Using PYTHONPATH: /var/lib/awx/venv/my-custom-venv/lib/python3.6/site-packages:
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/bin/awx-manage", line 11, in <module>
    load_entry_point('awx==3.0.0.0', 'console_scripts', 'awx-manage')()
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/__init__.py", line 150, in manage
    execute_from_command_line(sys.argv)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 1030, in handle
    raise exc
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 930, in handle
    data = AnsibleInventoryLoader(source=source, is_custom=self.is_custom, venv_path=venv_path).load()
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 176, in load
    return self.command_to_json(base_args + ['--list'])
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 159, in command_to_json
    self.method, proc.returncode, stdout, stderr))
RuntimeError: ansible-inventory failed (rc=1) with stdout:
stderr:
Traceback (most recent call last):
  File "/usr/bin/ansible-inventory", line 67, in <module>
    import ansible.constants as C
  File "/var/lib/awx/venv/my-custom-venv/lib/python3.6/site-packages/ansible/constants.py", line 17, in <module>
    from ansible.config.manager import ConfigManager, ensure_type, get_ini_config_value
  File "/var/lib/awx/venv/my-custom-venv/lib/python3.6/site-packages/ansible/config/manager.py", line 16, in <module>
    from yaml import load as yaml_load
  File "/var/lib/awx/venv/my-custom-venv/lib/python3.6/site-packages/yaml/__init__.py", line 8, in <module>
    from .loader import *
  File "/var/lib/awx/venv/my-custom-venv/lib/python3.6/site-packages/yaml/loader.py", line 4, in <module>
    from .reader import *
  File "/var/lib/awx/venv/my-custom-venv/lib/python3.6/site-packages/yaml/reader.py", line 45, in <module>
    class Reader(object):
  File "/var/lib/awx/venv/my-custom-venv/lib/python3.6/site-packages/yaml/reader.py", line 137, in Reader
    NON_PRINTABLE = re.compile('[^\x09\x0A\x0D\x20-\x7E\x85\xA0-\uD7FF\uE000-\uFFFD]')
  File "/usr/lib64/python2.7/re.py", line 190, in compile
    return _compile(pattern, flags)
  File "/usr/lib64/python2.7/re.py", line 242, in _compile
    raise error, v # invalid expression
sre_constants.error: bad character range

For some reason it takes 2.7 file while using 3.6 venv. Checked venv itself, it correctly imports "re" from the right file but AWX is messing with libs.

Thanks,
Roman

[ryanpetrello]

cc @AlanCoding looks like there might be an issue with https://github.com/ansible/awx/pull/3039 ?

This report makes it sound like the ansible-inventory being run is mistakenly using /usr/bin/ansible-inventory instead of the py3 ansible-inventory executable.

[AlanCoding]

Inventory updates have been tested with python2 virtual environments. They have not been tested with python3 virtual environments. I know that @ryanpetrello has tested playbook runs with python3, but I have not gotten around to that for any job type just yet.

I appreciate you filing this issue, and I assure you it is on our agenda. I also want @jladdjr to be aware of this.

[ryanpetrello]

Sounds to me like this just isn't quite there yet, and needs some more work, I'll investigate.

[ryanpetrello]

Yep, I'm able to reproduce this issue in the latest devel:

Thanks for reporting, @roman-vynar I'll see if I can come up with a patch.

[ryanpetrello]

@roman-vynar are you able to give this PR a shot? It resolves the issue for me.

https://github.com/ansible/awx/pull/3058

[roman-vynar]

Sure, let me edit the files straight in the docker containers :D and check.

[roman-vynar]

Currently I am getting

    3.495 INFO     Updating inventory 3: roman cluster
    3.523 INFO     Reading Ansible inventory source: /var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/plugins/inventory/ec2.py
    3.528 INFO     Using VIRTUAL_ENV: /var/lib/awx/venv/my-custom-venv/
    3.531 INFO     Using PATH: /var/lib/awx/venv/my-custom-venv/bin:/var/lib/awx/venv/awx/bin:/var/lib/awx/venv/awx/bin:/var/lib/awx/venv/awx/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    3.531 INFO     Using PYTHONPATH: /var/lib/awx/venv/my-custom-venv/lib/python3.6/site-packages:
    5.857 INFO     Processing JSON output...
    5.870 INFO     Loaded 46 groups, 10 hosts
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/bin/awx-manage", line 11, in <module>
    load_entry_point('awx==3.0.0.0', 'console_scripts', 'awx-manage')()
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/__init__.py", line 150, in manage
    execute_from_command_line(sys.argv)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 1037, in handle
    raise exc
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 982, in handle
    self.load_into_database()
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 831, in load_into_database
    self._delete_hosts()
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 420, in _delete_hosts
    instance_ids = all_instance_ids[offset:(offset + self._batch_size)]
TypeError: 'dict_keys' object is not subscriptable

I have seen this earlier today, not sure how to bypass that.

[ryanpetrello]

Looks like a bug in the inventory_import command such that it doesn't support py3. Let me update the PR.

[ryanpetrello]

@roman-vynar I've just uploaded a new version of the PR, give it another shot?

[roman-vynar]

Perfect 🎉

@ryanpetrello Thank you so much for such fast fixes!
Finally, I have run my playbook fully with python3.

P.S. I have also added ansible_python_interpreter: python3 to the vars of my template so it calls python3 on remote host instead of python which is 2 there.

[ryanpetrello]

@roman-vynar thanks a bunch for helping test out, I'm going to merge that PR.

[ryanpetrello]

Merged. Thanks for reporting, @roman-vynar. Let me know if you see any additional issues w/ this.

[iamroddo]

Any idea when this will make it's way into a release?