Awx: Ansible Virtual Environment no longer appears as an option in job templates

[grahamneville]

ISSUE TYPE

• Bug Report

COMPONENT NAME

• UI

SUMMARY

Ansible Virtual Environment no longer appears as an option in job templates

ENVIRONMENT

• AWX version: 1.0.6.0
• AWX install method: docker on linux
• Ansible version: 2.5.2

STEPS TO REPRODUCE

Create a new job template

EXPECTED RESULTS

You can select virtual environment:

ACTUAL RESULTS

No option appears

[ryanpetrello]

@marshmalien @jaredevantabor did we ever add code to hide this field if there are no actual underlying custom virtualenvs?

[marshmalien]

Yes, we did add code to hide this field. If custom_virtualenvs is an empty array, we hide the field on the job template form.

"custom_virtualenvs": [],

[jaredevantabor]

@grahamneville this field will be hidden if you have not configured AWX to accept multiple venvs. Can you confirm that you are seeing your venv options by looking at api/v2/config/ and then the contents of custom_virtualenvs, if you're able to paste that here please.

[roostercactus]

Hello, I'm having trouble adding custom virtualenvs. I'm following the doc (custom_virtualenvs.md) but it always tells me it's not valid when I try to PATCH a job_template from the web API.

From inside the awx_task container (docker exec -it task_id bash)
(as root)
virtualenv /var/lib/awx/venv/two-four
cd /var/lib/awx/venv/two-four/bin
./pip install python-memcached psutil
./pip install ansible==2.4.4

Then I go to the web API...

http://awx-host.example.org/api/v2/job_templates/7/

and in the box at the bottom of the page I put this

{
"custom_virtualenv": "/var/lib/awx/venv/two-four"
}

and hit the PATCH button. It always says it's not a valid virtual env though.

HTTP 400 Bad Request
{
"custom_virtualenv": [
"/var/lib/awx/venv/two-four is not a valid virtualenv in /var/lib/awx/venv"
]
}

Am I doing it wrong? Missing a step? Does it not work to PATCH from the web API client?

Running latest AWX on docker (1.0.6.9 and ansible 2.5.3). Any help appreciated. Thanks.

[roostercactus]

It helps to read the doc carefully. In a docker setup you have to create the virtualenv on both the awx_task and awx_web containers. Then when you PATCH in the web API it actually works.

[ryanpetrello]

If anyone is encountering this issue and they're _sure_ that /var/lib/awx/venv/ is actually populated with one or more custom virtualenvs (on both the awx_web _and_ awx_task containers), say so and we'll reopen this ticket to investigate.

[gnosek]

I'm absolutely positive that I have a custom venv on both containers. PATCH works for me (I see the custom_virtualenv changed on the organization and project), but there's no custom_virtualenvs setting at all in /api/v2/config (not even an empty list) and even after setting the custom virtualenv the inventory script runs using the default venv.

FWIW,

[root@awxweb awx]# awx-manage shell
Python 2.7.5 (default, Jul 13 2018, 13:06:57)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from awx.main.utils.common import *
>>> get_custom_venv_choices()
[u'/var/lib/awx/venv/ansible/', u'/var/lib/awx/venv/mycustomenv/']
>>>

[ryanpetrello]

@gnosek I think we've figured out what's going on here - it's a permission-related bug that's causing certain users to not see the custom venv choices. I'm currently working on a fix for this.

[gnosek]

After adding another job template (apart from the demo job template already present), I see custom_virtualenvs in /api/v2/config but still no dropdown in the UI

[ryanpetrello]

@gnosek are you logged in as a job template admin, or a superuser?

[gnosek]

superuser (the default user created on install)

[gnosek]

BTW, is there a known workaround for this? I can create an extra user with whatever permissions required if that helps

[ryanpetrello]

What version of awx are you using? We fixed a bug related to this from 1.0.6 -> 1.0.7.

[gnosek]

>>> awx.__version__
'1.0.7.2'

Fresh install, based on these images (plus my venv):

ansible/awx_task    latest              67398bc01f40        4 weeks ago         1.03GB
ansible/awx_web     latest              27936ec5a5de        4 weeks ago         997MB

[ryanpetrello]

Looks like you're on the latest and that this is indeed a bug. Even if you hadn't installed virtualenvs on the containers, the API at /api/v2/config/ should still reply with a list of custom_virtualenvs.

[gnosek]

Well, now it does (after I added another job template) but still there's no dropdown in the UI. When I PATCH the project/organization with the right venv it gets saved but it doesn't seem to have any effect.

I have a project with an inventory script that needs a custom venv and an inventory source doesn't seem to have the custom_virtualenv setting. Still, I'd be happy with setting the venv organization-wide.

[ryanpetrello]

Until we can figure out what's up here, in awx-manage shell_plus, are you able to do the following (where N is the ID of the project):

proj = Project.objects.get(id=N)
proj.custom_virtualenv = '/var/lib/awx/venv/mycustomenv/'
proj.save()
print Project.objects.get(id=N).custom_virtualenv

[gnosek]

It's already pointing to my custom venv (after a previous PATCH, I'd expect, or maybe it even got cloned from the organization. I can drop and recreate if you want), but I just reran the inventory sync using a script from this project and it was executed with the default venv.

[ryanpetrello]

Okay, that sounds like a potential bug, I'll see if I can reproduce.

[gnosek]

FWIW, I can see the dropdown now. I literally didn't touch the site at all since my last comment here. Can it be a caching issue?

Still, the inventory script is ran with the wrong venv.

[gnosek]

Job templates run with the right venv but as it's python 3 based, the whole thing explodes when trying to run a script that looks like print '{ massive json }', still trying to find if it comes from my (inherited) scripts or from awx itself. Though the playbooks have always been run with 3.x, so I wouldn't expect them to require (or even work with) py2.

[gnosek]

OK, I think this is the one:

https://github.com/ansible/awx/blob/devel/awx/main/tasks.py#L818

I'll patch it and see what happens

[ryanpetrello]

@gnosek I did some digging, and it looks like custom virtualenvs specified at the organization level don't get used on inventory syncs, so this is just a bug that needs fixing.

[gnosek]

https://github.com/ansible/awx/pull/2260 submitted.

[gnosek]

Can I set the venv somewhere else that will take effect? It's already set in the organization and project settings.

[ryanpetrello]

@gnosek unfortunately no, the code that runs the ansible inventory import current incorrectly assumes the default virtualenv (it doesn't respect any custom venv you may specify at the organization level):

https://github.com/ansible/awx/blob/devel/awx/main/management/commands/inventory_import.py#L84

This is just a bug that needs fixing.

[ryanpetrello]

If you feel like experimenting, this is a diff I'm currently testing, and am likely to turn into a pull request:

diff --git a/awx/main/management/commands/inventory_import.py b/awx/main/management/commands/inventory_import.py
index 692bb56443..70ae08db0f 100644
--- a/awx/main/management/commands/inventory_import.py
+++ b/awx/main/management/commands/inventory_import.py
@@ -66,7 +66,7 @@ class AnsibleInventoryLoader(object):
     If it fails to find this, it uses the backported script instead
     '''

-    def __init__(self, source, group_filter_re=None, host_filter_re=None, is_custom=False):
+    def __init__(self, source, group_filter_re=None, host_filter_re=None, is_custom=False, venv_path=None):
         self.source = source
         self.source_dir = functioning_dir(self.source)
         self.is_custom = is_custom
@@ -74,6 +74,7 @@ class AnsibleInventoryLoader(object):
         self.method = 'ansible-inventory'
         self.group_filter_re = group_filter_re
         self.host_filter_re = host_filter_re
+        self.venv_path = venv_path or settings.ANSIBLE_VENV_PATH

         self.is_vendored_source = False
         if self.source_dir == os.path.join(settings.BASE_DIR, 'plugins', 'inventory'):
@@ -81,13 +82,13 @@ class AnsibleInventoryLoader(object):

     def build_env(self):
         env = dict(os.environ.items())
-        env['VIRTUAL_ENV'] = settings.ANSIBLE_VENV_PATH
-        env['PATH'] = os.path.join(settings.ANSIBLE_VENV_PATH, "bin") + ":" + env['PATH']
+        env['VIRTUAL_ENV'] = self.venv_path
+        env['PATH'] = os.path.join(self.venv_path, "bin") + ":" + env['PATH']
         # Set configuration items that should always be used for updates
         for key, value in STANDARD_INVENTORY_UPDATE_ENV.items():
             if key not in env:
                 env[key] = value
-        venv_libdir = os.path.join(settings.ANSIBLE_VENV_PATH, "lib")
+        venv_libdir = os.path.join(self.venv_path, "lib")
         env.pop('PYTHONPATH', None)  # default to none if no python_ver matches
         if os.path.isdir(os.path.join(venv_libdir, "python2.7")):
             env['PYTHONPATH'] = os.path.join(venv_libdir, "python2.7", "site-packages") + ":"
@@ -223,7 +224,7 @@ class AnsibleInventoryLoader(object):

 def load_inventory_source(source, group_filter_re=None,
                           host_filter_re=None, exclude_empty_groups=False,
-                          is_custom=False):
+                          is_custom=False, venv_path=None):
     '''
     Load inventory from given source directory or file.
     '''
@@ -242,7 +243,8 @@ def load_inventory_source(source, group_filter_re=None,
         source=source,
         group_filter_re=group_filter_re,
         host_filter_re=host_filter_re,
-        is_custom=is_custom).load()
+        is_custom=is_custom,
+        venv_path=venv_path).load()

     logger.debug('Finished loading from source: %s', source)
     # Exclude groups that are completely empty.
@@ -991,7 +993,8 @@ class Command(BaseCommand):
                                                    self.group_filter_re,
                                                    self.host_filter_re,
                                                    self.exclude_empty_groups,
-                                                   self.is_custom)
+                                                   self.is_custom,
+                                                   self.inventory.organization.custom_virtualenv)
             if settings.DEBUG:
                 # depending on inventory source, this output can be
                 # *exceedingly* verbose - crawling a deeply nested

[gnosek]

I'll give the patch a try tomorrow, thanks!

BTW I'm thinking if this should use the per-project venv, not per-organization (though it's 100% fine for me either way).

[ryanpetrello]

@gnosek that probably makes sense for inventory sources that sourced from a project clone.

[gnosek]

yeah, that's exactly what I'm using fwiw

[gnosek]

I tested the patch right now and it works for me, thanks!

[MrMEEE]

@gnosek Did you ever figure out what made the dropdown box show up??

I'm having the same issue in my RPM version...

[gnosek]

@MrMEEE no idea, I just left it running overnight. Maybe some cache had to expire so you might try restarting memcached (just guessing).

[MrMEEE]

@gnosek Tried restarting memcached... no luck.. tried logging in and out several times... no luck...

But then I added a second venv... and.. Voila!! there the dropdown appeared...

I'm thinking.. maybe it's meant not to show up if there is only on venv.. but it forgets to count the default one??

[ryanpetrello]

@MrMEEE as you suggested - the environment selection doesn't appear if there's only one venv available (the base one we ship with awx). Once you add a second custom one, the dropdown should appear.

Another thing I'll suggest is to make sure you're on the latest awx - we _have_ fixed several issues related to the custom virtualenv selection in the past few months.

[MrMEEE]

@ryanpetrello The problem is that the menu first appears when the third venv is added..

I can only speak for the RPM install version, but other docker users seem to have had the same issue

[ryanpetrello]

@MrMEEE which version of awx are you using?

[MrMEEE]

2.0.1

[ryanpetrello]

@MrMEEE if you create one custom virtualenv, you don't see the dropdown on the Job Template add/edit form? When that's the case, can you share with me what you see at:

HTTP GET /api/v2/config/

[angystardust]

@ryanpetrello Hi, same problem here: dropdown remains hidden with just one venv. It magically appears when i create a second venv