Awx: Adding venv_vars.yaml makes container fail to start

[magick93]

ISSUE TYPE

• Bug Report

SUMMARY

Use of virtual envs for k8s/openshift, as described on https://github.com/ansible/awx/blob/devel/docs/custom_virtualenvs.md isnt working.

ENVIRONMENT

• AWX version: 9.1.1
• AWX install method: openshift
• Ansible version: Same as what ships with AWX 9.1.1
• Operating System: Centos
• Web Browser: Firefox

STEPS TO REPRODUCE

1. Ansible AWX is already installed on Openshift 3.11
2. Create a file called venv_vars.yaml and copy the example yaml from https://github.com/ansible/awx/blob/devel/docs/custom_virtualenvs.md
3. Attempt to modify installed AWX to have some venvs. Run ansible-playbook -i inventory install.yml -e openshift_password=username -e openshift_token=$(oc whoami -t) --extra-vars "@venv_vars.yaml"

EXPECTED RESULTS

Expect to have virtual environments created.

ACTUAL RESULTS

The aws-0 container fails to start, gets stuck in Init Crash Loop Back-off.

ADDITIONAL INFORMATION

I'm running in a corp environment, and am using a proxy. The below looks like the container isnt using the proxy settings that are added to the inventory.

The init-custom-venvs has the following command:

sh -c yum install -y ansible curl python-setuptools epel-release \ openssl openssl-devel gcc python-devel && yum install -y python-virtualenv python36 python36-devel && mkdir -p /opt/custom-venvs && virtualenv -p python3 \ /opt/custom-venvs/dns_team && source /opt/custom-venvs/dns_team/bin/activate && /opt/custom-venvs/dns_team/bin/pip install -U psutil \ "ansible==2.8.1" && /opt/custom-venvs/dns_team/bin/pip install -U \ dnspython infoblox-client && deactivate && virtualenv -p python2 \ /opt/custom-venvs/windows_team && source /opt/custom-venvs/windows_team/bin/activate && /opt/custom-venvs/windows_team/bin/pip install -U psutil \ "ansible==2.8.0" && /opt/custom-venvs/windows_team/bin/pip install -U \ winrm && deactivate && virtualenv -p python2 \ /opt/custom-venvs/vmware_team && source /opt/custom-venvs/vmware_team/bin/activate && /opt/custom-venvs/vmware_team/bin/pip install -U psutil \ "ansible==2.7.10" && /opt/custom-venvs/vmware_team/bin/pip install -U \ pyvmomi && deactivate && :

Logs

From oc logs awx-0 -c init-custom-venvs shows:

Loaded plugins: fastestmirror, ovl
Determining fastest mirrors


 One of the configured repositories failed (Unknown),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Run the command with the repository temporarily disabled
            yum --disablerepo=<repoid> ...

     4. Disable the repository permanently, so yum won't use it by default. Yum
        will then just ignore the repository until you permanently enable it
        again or use --enablerepo for temporary usage:

            yum-config-manager --disable <repoid>
        or
            subscription-manager repos --disable=<repoid>

     5. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true

Cannot find a valid baseurl for repo: base/7/x86_64
Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container error was
14: curl#7 - "Failed to connect to 2604:1580:fe02:2::10: Network is unreachable"

@fantashley

[wenottingham]

It looks like your container can't reach the internet at large (at least over the IPv6 address you got back from the mirror list)

[fantashley]

Can you test against this branch https://github.com/fantashley/awx/tree/support-proxy-venv and tell me if that fixes your issue?

It will set the environment variables http_proxy and https_proxy if they are defined as ansible variables, and yum should pick that up in CentOS 7.

[magick93]

Hi @fantashley - thanks, yes this helped.

I also had to add the following in order to get it work correctly - in particular, note the --trusted-host for the pip installs:

 initContainers:
 158   │         - image: 'centos:7'
 159   │           name: init-custom-venvs
 160   │           command:
 161   │             - sh
 162   │             - '-c'
 163   │             - >-
 164 ~ │               echo "proxy="{{ http_proxy }} >> /etc/yum.conf  && yum install -y ansible curl python-setuptools epel-release \
 165   │                 openssl openssl-devel gcc python-devel &&
 166   │               yum install -y python-virtualenv python36 python36-devel &&
 167   │               mkdir -p {{ custom_venvs_path }} &&
 168   │ {% for custom_venv in custom_venvs %}
 169   │               virtualenv -p {{ custom_venv.python | default(custom_venvs_python) }} \
 170   │                 {{ custom_venvs_path }}/{{ custom_venv.name }} &&
 171   │               source {{ custom_venvs_path }}/{{ custom_venv.name }}/bin/activate &&
 172 ~ │               {{ custom_venvs_path }}/{{ custom_venv.name }}/bin/pip install   --trusted-host pypi.org --trusted-host files.pythonhosted.org  --trusted-host pypi.python.org --proxy http://proxy.wallets:8080" -U psutil \
 173   │                 "ansible=={{ custom_venv.python_ansible_version }}" &&
 174   │ {% if custom_venv.python_modules is defined %}
 175 ~ │               {{ custom_venvs_path }}/{{ custom_venv.name }}/bin/pip install  --trusted-host pypi.org --trusted-host files.pythonhosted.org --trusted-host pypi.python.org --proxy http://proxy.wallets:8080" -U \
 176   │                 {% for module in custom_venv.python_modules %}{{ module }} {% endfor %} &&
 177   │ {% endif %}
 178   │               deactivate &&
 179   │ {% endfor %}
 180   │               :
 181 + │           env:
 182 + │             - name: HTTP_PROXY
 183 + │               value: {{ http_proxy }}
 184 + │             - name: HTTPS_PROXY
 185 + │               value: {{ https_proxy }}

[evertmulder]

@fantashley I also run into this issue as well. Are you planning to make a PR for this?

[fantashley]

@evertmulder yep, I will be able to make a PR by the end of this week. It would be great if you could both test the changes for me since I don't have a similar environment setup. I'll let you know once I've created it!

@magick93 sounds good, I will add in the --trusted-host parameters you provided. Thanks!

[fantashley]

I updated my branch https://github.com/fantashley/awx/tree/support-proxy-venv with the trusted host arguments if someone wants to give it a try!

[evertmulder]

Hi @fantashley. I just tested the proxy support and it works great. The proxy settings are added to the init container and the trusted-host settings are also added to the pip install. The thing missing is the no_proxy setting.

Tested on OKD 3.11

The final statefulset includes the following init-container:

      initContainers:
        - command:
            - sh
            - '-c'
            - >-
              yum install -y ansible curl python-setuptools epel-release \
                openssl openssl-devel gcc python-devel &&
              yum install -y python-virtualenv python36 python36-devel && mkdir
              -p /opt/custom-venvs && virtualenv -p python3 \
                /opt/custom-venvs/team_a &&
              source /opt/custom-venvs/team_a/bin/activate &&
              /opt/custom-venvs/team_a/bin/pip install --trusted-host pypi.org
              --trusted-host files.pythonhosted.org --trusted-host
              pypi.python.org -U psutil \
                "ansible==2.8.4" &&
              /opt/custom-venvs/team_a/bin/pip install --trusted-host pypi.org
              --trusted-host files.pythonhosted.org --trusted-host
              pypi.python.org -U \
                dnspython infoblox-client jmespath  &&
              deactivate && virtualenv -p python2 \
                /opt/custom-venvs/team_b &&
              source /opt/custom-venvs/team_b/bin/activate &&
              /opt/custom-venvs/team_b/bin/pip install --trusted-host
              pypi.org --trusted-host files.pythonhosted.org --trusted-host
              pypi.python.org -U psutil \
                "ansible==2.8.6" &&
              /opt/custom-venvs/team_b/bin/pip install --trusted-host
              pypi.org --trusted-host files.pythonhosted.org --trusted-host
              pypi.python.org -U \
                manageiq_client PyVmomi  &&
              deactivate && :
          env:
            - name: http_proxy
              value: 'http://proxy.company.local:8080'
            - name: https_proxy
              value: 'http://proxy.company.local:8080'
          image: 'centos:7'
          imagePullPolicy: IfNotPresent
          name: init-custom-venvs
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /opt/custom-venvs
              name: custom-venvs

The only thing missing is the no_proxy setting. If this setting could also be included this would be great.

[ryanpetrello]

Sounds like this can be closed once https://github.com/ansible/awx/pull/5805 merges - thanks @fantashley!