Awx: Virtualenv not being used by default for task execution

[geauxvirtual]

ISSUE TYPE

• Bug Report

COMPONENT NAME

• Job Execution

SUMMARY

The AWX Web and Task containers come configured with an ansible virtualenv in the container. Adding any python modules through pip to this virtualenv are not picked up during a job run unless ansible_python_interpreter: /var/lib/awx/venv/ansible/bin/python is set.

ENVIRONMENT

• AWX version: 2.0.1
• AWX install method: Kubernetes
• Ansible version: 2.6.4

STEPS TO REPRODUCE

Build a custom task container, adding a module such as OpenShift to the default ansible virtualenv environment.

Execute a job template that uses the k8s or oc modules.

EXPECTED RESULTS

Job to be executed with the default Ansible virtualenv.

ACTUAL RESULTS

Job runs with the system python in the container, which does not know where the modules are installed.

[matburt]

I'm having some trouble reproducing this, I'll leave this open in a needs_info state

[geauxvirtual]

Here is the dockerfile I'm using to build a custom task container where {{ awx_base_task_image }} and {{ awx_base_task_version }} point to task container built from the AWX image_build role with Ansible 2.6.4 instead of Ansible 2.7. This is the Dockerfile used to add openshift to the Ansible venv.

FROM {{ awx_base_task_image }}:{{ awx_base_task_version }}
USER root
RUN yum -y clean all
RUN source /var/lib/awx/venv/ansible/bin/activate && \
    umask 0022 && \
    pip install openshift && \
    deactivate
COPY jq /usr/bin/jq
COPY kubectl /usr/bin/kubectl
COPY helm /usr/bin/helm
COPY istioctl /usr/bin/istioctl

Here is a playbook I'm running:

- name: Test AWX
  hosts: local
  gather_facts: yes
  tasks:
  - name: Get Ansible
    shell: ansible --version
    register: ansible_output
  - debug: var=ansible_output
  - name: SysPath
    shell: python -c "import sys; print(sys.path)"
    register: syspath_output
  - debug: var=syspath_output
  - name: Python modules
    shell: pip list | grep openshift
    register: pip_output
  - debug: var=pip_output
  - name: Use K8s
    k8s:
      api_version: v1
      kind: Namespace
      name: ansible-demo
      state: present

Here is the AWX output:

PLAY [Test AWX] ****************************************************************
TASK [Gathering Facts] *********************************************************
ok: [127.0.0.1]
TASK [Get Ansible] *************************************************************
changed: [127.0.0.1]
TASK [debug] *******************************************************************
ok: [127.0.0.1] => {
    "ansible_output": {
        "changed": true, 
        "cmd": "ansible --version", 
        "delta": "0:00:01.247243", 
        "end": "2018-10-26 03:21:22.456331", 
        "failed": false, 
        "rc": 0, 
        "start": "2018-10-26 03:21:21.209088", 
        "stderr": "", 
        "stderr_lines": [], 
        "stdout": "ansible 2.6.4\n  config file = None\n  configured module search path = [u'/var/lib/awx/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']\n  ansible python module location = /usr/lib/python2.7/site-packages/ansible\n  executable location = /usr/bin/ansible\n  python version = 2.7.5 (default, Jul 13 2018, 13:06:57) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]", 
        "stdout_lines": [
            "ansible 2.6.4", 
            "  config file = None", 
            "  c…
TASK [SysPath] *****************************************************************
changed: [127.0.0.1]
TASK [debug] *******************************************************************
ok: [127.0.0.1] => {
    "syspath_output": {
        "changed": true, 
        "cmd": "python -c \"import sys; print(sys.path)\"", 
        "delta": "0:00:00.317673", 
        "end": "2018-10-26 03:21:23.163598", 
        "failed": false, 
        "rc": 0, 
        "start": "2018-10-26 03:21:22.845925", 
        "stderr": "", 
        "stderr_lines": [], 
        "stdout": "['', '/var/lib/awx/venv/ansible/lib/python2.7/site-packages', '/var/lib/awx/projects/_13__awx_tester', '/var/lib/awx/venv/ansible/lib64/python27.zip', '/var/lib/awx/venv/ansible/lib64/python2.7', '/var/lib/awx/venv/ansible/lib64/python2.7/plat-linux2', '/var/lib/awx/venv/ansible/lib64/python2.7/lib-tk', '/var/lib/awx/venv/ansible/lib64/python2.7/lib-old', '/var/lib/awx/venv/ansible/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7', '/usr/lib/python2.7', '/usr/lib64/python2.7/site-packages', '/usr/li…
TASK [Python modules] **********************************************************
changed: [127.0.0.1]
TASK [debug] *******************************************************************
ok: [127.0.0.1] => {
    "pip_output": {
        "changed": true, 
        "cmd": "pip list | grep openshift", 
        "delta": "0:00:00.958661", 
        "end": "2018-10-26 03:21:24.524422", 
        "failed": false, 
        "rc": 0, 
        "start": "2018-10-26 03:21:23.565761", 
        "stderr": "DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.", 
        "stderr_lines": [
            "DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning."
        ], 
        "stdout": "openshift (0.7.2)", 
        "stdout_lines"…
TASK [Use K8s] *****************************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "This module requires the OpenShift Python client. Try `pip install openshift`"}
PLAY RECAP *********************************************************************
127.0.0.1                  : ok=8    changed=3    unreachable=0    failed=1

Detailed stdout output from the playbook run
Command: ansible --version

"stdout_lines": [
            "ansible 2.6.4",
            "  config file = None",
            "  configured module search path = [u'/var/lib/awx/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']",
            "  ansible python module location = /usr/lib/python2.7/site-packages/ansible",
            "  executable location = /usr/bin/ansible",
            "  python version = 2.7.5 (default, Jul 13 2018, 13:06:57) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]"
        ]

Command: python -c "import sys; print(sys.path)"

"stdout_lines": [
            "['', '/var/lib/awx/venv/ansible/lib/python2.7/site-packages', '/var/lib/awx/projects/_13__awx_tester', '/var/lib/awx/venv/ansible/lib64/python27.zip', '/var/lib/awx/venv/ansible/lib64/python2.7', '/var/lib/awx/venv/ansible/lib64/python2.7/plat-linux2', '/var/lib/awx/venv/ansible/lib64/python2.7/lib-tk', '/var/lib/awx/venv/ansible/lib64/python2.7/lib-old', '/var/lib/awx/venv/ansible/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7', '/usr/lib/python2.7', '/usr/lib64/python2.7/site-packages', '/usr/lib/python2.7/site-packages']"
        ],

Command: pip list | grep openshift

"stdout_lines": [
            "openshift (0.7.2)"
        ],

Running a task using K8s module:

{
    "_ansible_parsed": true,
    "invocation": {
        "module_args": {
            "username": null,
            "ssl_ca_cert": null,
            "kind": "Namespace",
            "password": null,
            "force": false,
            "name": "ansible-demo",
            "src": null,
            "cert_file": null,
            "namespace": null,
            "verify_ssl": null,
            "kubeconfig": null,
            "host": null,
            "state": "present",
            "context": null,
            "resource_definition": null,
            "key_file": null,
            "api_key": null,
            "api_version": "v1"
        }
    },
    "changed": false,
    "_ansible_no_log": false,
    "msg": "This module requires the OpenShift Python client. Try `pip install openshift`"
}

[grahamn-gr]

I too have come across this issue, I'm running 1.0.7.0. It seems that if you don't set a custom version the project even though you set it under an organisation level it will default to the system venv.
You either have to manually choose the venv under the template or set it at a project level.

[ghost]

@geauxvirtual @matburt @ryanpetrello @willthames @chouseknecht @fabianvf @flaper87 @maxamillion

Yes. Still its issue ... I manually created this playbook and try to execute the playbook. The Job runs with the system python in the container, which does not know where the modules are installed. ( Even though I run the AWX job with with ansible_python_interpreter: '{{ ansible_playbook_python }}' )

(ansible) [root@awx awx]# pip list | grep shift
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
openshift                      0.8.7
(ansible) [root@awx awx]# source /var/lib/awx/venv/awx/bin/activate
(awx) [root@awx awx]# pip list | grep shift
openshift                0.8.7
(awx) [root@awx awx]#

[ryanpetrello]

Anybody encountering this in more recent versions of AWX (e.g., 3.0.x, 4.0.0)? We've made some notable changes here in the recent past.

You might also consider checking out this document, which outlines the way we _recommend_ managing custom Python dependencies in AWX:

https://github.com/ansible/awx/blob/devel/docs/custom_virtualenvs.md

[geauxvirtual]

We're currently using a 3.0.x+ version of the AWX task container, and we still need to add the ansible_python_interpreter to all our jobs for it to pick up modules added to the ansible virtual environment in the task container.

[dglinder]

For what it's worth, I encountered this in our Tower installation too, Ansible 2.7.7, Python 2.7.5, except Tower 3.3.4: https://github.com/ansible/ansible/issues/50529#issuecomment-477309583) and none of the workarounds in that thread worked for us (e.g. using ansible_python_interpreter, etc).

Due to the size of our deployment, I needed to use the virutalenv settings and we finally found a (ugly) hack to get the k8s module to see the openshift module (albeit in the system base environment). I did build a playbook to setup the virtual and system environments so they work, but that's a band-aid.

[europ]

I have the same problem here.

Tower does not use the same python for tasks, which are executed as delegated_to: localhost on host.

This can be fixed via workaround that requires to specify ansible_python_interpreter: /path/to/your/python in host variables for localhost.

[l3ender]

We're facing this using Tower version 3.6 and running a job template specifying a virtual environment. The playbook runs on localhost with connection: local.

It works if we set ansible_python_interpreter: /usr/bin/python3 for the inventory, but is there a way to have it infer the python version/path from the virtual environment?