Awx: Virtualenv in Kubernetes deployment doesn't work

[artmakh]

ISSUE TYPE

• Bug Report

SUMMARY

Error when applying
curl -X PATCH 'https://admin:mypass@myurl/api/v2/organizations/my_org/' \ -d '{"custom_virtualenv": "/opt/custom-venvs/my_env/"}' -H 'Content-Type:application/json'
{"custom_virtualenv":["/opt/custom-venvs/my_env/ is not a valid virtualenv in /var/lib/awx/venv"]}

ENVIRONMENT

• AWX version: 9.3.0
• AWX install method: Kubernetes v1.15.3
• Ansible version: 2.9.5

STEPS TO REPRODUCE

1) Install AWX to kubernetes with similar config

inventory config

localhost ansible_connection=local ansible_python_interpreter="/usr/bin/env python"

[all:vars]

dockerhub_base=ansible

kubernetes_context=mycontext
kubernetes_namespace=awx

kubernetes_ingress_hostname=myingress

awx_task_hostname=awx
awx_web_hostname=awxweb
postgres_data_dir="~/.awx/pgdocker"
host_port=80
host_port_ssl=443
docker_compose_dir="~/.awx/awxcompose"

pg_hostname=mypghost
pg_username=awx
pg_password=mypgpass
pg_database=awx
pg_port=5432

rabbitmq_password=awxpass
rabbitmq_erlang_cookie=cookiemonster

admin_user=admin
admin_password=myadminpass

secret_key=mysecretkey

logos/blob/master/TRADEMARKS.md

venv_vars.yaml

---
custom_venvs:
  - name: my_env
    python: python3
    python_ansible_version: 2.9.6
    python_modules:
      - dnspython
      - zabbix-api
      - jmespath

2) Assigning custom venv according to this documentation
curl -X PATCH 'https://admin:mypass@myurl/api/v2/settings/system/' \ -d '{"CUSTOM_VENV_PATHS": [["/opt/custom-venvs/my_env/"]}' -H 'Content-Type:application/json'

answer

{
   "ACTIVITY_STREAM_ENABLED":true,
   "ACTIVITY_STREAM_ENABLED_FOR_INVENTORY_SYNC":false,
   "ORG_ADMINS_CAN_SEE_ALL_USERS":true,
   "MANAGE_ORGANIZATION_AUTH":true,
   "TOWER_URL_BASE":"myurl",
   "REMOTE_HOST_HEADERS":[
      "HTTP_X_FORWARDED_FOR"
   ],
   "PROXY_IP_WHITELIST":[

   ],
   "LICENSE":{

   },
   "REDHAT_USERNAME":"admin",
   "REDHAT_PASSWORD":"$encrypted$",
   "AUTOMATION_ANALYTICS_URL":"https://example.org",
   "INSTALL_UUID":"c1885c2a-d736-4023-9517-49929f8d77fa",
   "CUSTOM_VENV_PATHS":[
      "/opt/custom-venvs/my_env/"
   ],
   "INSIGHTS_TRACKING_STATE":false,
   "BROKER_DURABILITY":true,
   "AUTOMATION_ANALYTICS_LAST_GATHER":null,
   "AUTOMATION_ANALYTICS_GATHER_INTERVAL":14400
}

3) Assigning custom venv to organization and get and error
curl -X PATCH 'https://admin:mypass@myurl/api/v2/organizations/my_org/' \ -d '{"custom_virtualenv": "/opt/custom-venvs/my_env/"}' -H 'Content-Type:application/json'

error

{"custom_virtualenv":["/opt/custom-venvs/my_env/ is not a valid virtualenv in /var/lib/awx/venv"]}

EXPECTED RESULTS

my_org will use custom venv

ACTUAL RESULTS

my_org raises an error when apllying custom venv

ADDITIONAL INFORMATION

I've checked both containers awx_web and awx_task, they have /opt/custom-venvs/my_env/ dir with working venv.

Even if PATCH api/v2/settings/system/ works properly (according to it answer, and i can see added venv in web interface), GET api/v2/config/ still returns no venv's

api/v2/config/ answer

{
    "time_zone": "UTC",
    "license_info": {
        "license_key": "OPEN",
        "valid_key": true,
        "compliant": true,
        "features": {
            "activity_streams": true,
            "ha": true,
            "ldap": true,
            "multiple_organizations": true,
            "surveys": true,
            "system_tracking": true,
            "rebranding": true,
            "enterprise_auth": true,
            "workflows": true
        },
        "license_type": "open"
    },
    "version": "9.3.0",
    "ansible_version": "2.9.5",
    "eula": "",
    "analytics_status": "off",
    "become_methods": [
        [
            "sudo",
            "Sudo"
        ],
        [
            "su",
            "Su"
        ],
        [
            "pbrun",
            "Pbrun"
        ],
        [
            "pfexec",
            "Pfexec"
        ],
        [
            "dzdo",
            "DZDO"
        ],
        [
            "pmrun",
            "Pmrun"
        ],
        [
            "runas",
            "Runas"
        ],
        [
            "enable",
            "Enable"
        ],
        [
            "doas",
            "Doas"
        ],
        [
            "ksu",
            "Ksu"
        ],
        [
            "machinectl",
            "Machinectl"
        ],
        [
            "sesu",
            "Sesu"
        ]
    ],
    "project_base_dir": "/var/lib/awx/projects",
    "project_local_paths": [],
    "custom_virtualenvs": [
        "/var/lib/awx/venv/ansible/"
    ]
}

After finding that venv doesn't apply even in system, i've tried to make fresh install of awx on another kube cluster, with same results.

[ryanpetrello]

@fantashley you have any ideas here?

[fantashley]

I found that the System API endpoint seems to expect the parent directory of the custom virtual environments, rather than a path directly to the venv. If I update the system settings with /opt/custom-venvs instead of /opt/custom-venvs/my_env, it recognizes the virtual environment and will then display it at the /api/v2/config/ endpoint:

"custom_virtualenvs": [
    "/var/lib/awx/venv/ansible/",
    "/opt/custom-venvs/my_env/"
]

Then the following patch to the organization with the full venv path appears to work:
curl -X PATCH 'http://ashley:[email protected]/api/v2/organizations/2/' -d '{"custom_virtualenv": "/opt/custom-venvs/my_env/"}' -H 'Content-Type:application/json'

[artmakh]

@fantashley thanks, that hellped!

So the problem was that i specified exact folder of venv, not a folder with venv subfolders in it.
Looks like i was mislead by possitive answer of this API

curl -X PATCH 'https://admin:mypass@myurl/api/v2/settings/system/' \ -d '{"CUSTOM_VENV_PATHS": ["/opt/custom-venvs/my_env/"]}' -H 'Content-Type:application/json' .