Ansible: k8s module throwing 'This module requires the OpenShift Python client. Try `pip install openshift`'

[geerlingguy]

SUMMARY

So, I've already read through all of the following earlier issues:

• k8s_raw module shows openshift missing Python client even all dependencies are satisfied
• Execute k8s task failed

And I've been debugging this issue for my Pi Dramble project (within a Vagrant environment) locally for a couple days; see Switch to geerlingguy.k8s_manifests role. I think I'm close to a solution, but because this road has been windy, and ridiculously weird, I thought I'd post an issue here, and at least document how I solved it (if I can get it solved, ha!) for the benefit of future searchers... and myself, when I inevitably run into the problem again!

ISSUE TYPE

• Bug Report (maybe Support Request-ish)

COMPONENT NAME

k8s

ANSIBLE VERSION

```paste below
ansible 2.7.5
config file = /Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/testing/vagrant/ansible.cfg
configured module search path = [u'/Users/jgeerling/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python2.7/site-packages/ansible
executable location = /usr/local/bin/ansible
python version = 2.7.15 (default, Jul 17 2018, 19:41:20) [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)]

##### CONFIGURATION

ANSIBLE_NOCOWS(/Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/testing/vagrant/ansible.cfg) = True
ANSIBLE_PIPELINING(/Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/testing/vagrant/ansible.cfg) = True
ANSIBLE_SSH_CONTROL_PATH(/Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/testing/vagrant/ansible.cfg) = /tmp/ansible-ssh-%%h-%%p-%%r
DEFAULT_FORKS(/Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/testing/vagrant/ansible.cfg) = 10
DEFAULT_HOST_LIST(/Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/testing/vagrant/ansible.cfg) = [u'/Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/testing/vagrant/inventory']
DEFAULT_LOAD_CALLBACK_PLUGINS(/Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/testing/vagrant/ansible.cfg) = True
DEFAULT_ROLES_PATH(/Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/testing/vagrant/ansible.cfg) = [u'/Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/testing/vagrant/roles']
DEFAULT_STDOUT_CALLBACK(/Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/testing/vagrant/ansible.cfg) = yaml
RETRY_FILES_ENABLED(/Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/testing/vagrant/ansible.cfg) = False

##### OS / ENVIRONMENT

macOS Mojave (10.14)

##### STEPS TO REPRODUCE

Playbook:

```yaml
---
- hosts: localhost
  gather_facts: false
  become: false
  connection: local

  vars_files:
    - vars/main.yml
    - config.yml

  roles:
    - name: geerlingguy.k8s_manifests
      tags: ['kubernetes', 'nfs', 'drupal', 'registry']

Inventory:

[localhost]
127.0.0.1 deploy_target=pi

EXPECTED RESULTS

Kubernetes manifests will be applied to the K8s cluster.

ACTUAL RESULTS

TASK [geerlingguy.k8s_manifests : Deploy the resources defined inside the manifest.] ***
task path: /Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/roles/geerlingguy.k8s_manifests/tasks/deploy-manifest.yml:27
Using module file /usr/local/lib/python2.7/site-packages/ansible/modules/clustering/k8s/k8s.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: jgeerling
<127.0.0.1> EXEC /bin/sh -c '/usr/local/opt/python@2/bin/python2.7 && sleep 0'
Using module file /usr/local/lib/python2.7/site-packages/ansible/modules/clustering/k8s/k8s.py
<127.0.0.1> EXEC /bin/sh -c '/usr/local/opt/python@2/bin/python2.7 && sleep 0'
failed: [127.0.0.1] (item={'rules': [{'apiGroups': [u''], 'verbs': [u'get', u'list', u'watch'], 'resources': [u'services', u'endpoints', u'secrets']}, {'apiGroups': [u'extensions'], 'verbs': [u'get', u'list', u'watch'], 'resources': [u'ingresses']}], 'kind': u'ClusterRole', 'apiVersion': u'rbac.authorization.k8s.io/v1beta1', 'metadata': {'name': u'traefik-ingress-controller'}}) => changed=false 
  invocation:
    module_args:
      api_key: null
      api_version: v1
      cert_file: null
      context: null
      definition:
        apiVersion: rbac.authorization.k8s.io/v1beta1
        kind: ClusterRole
        metadata:
          name: traefik-ingress-controller
      ...
      verbs:
      - get
      - list
      - watch
  msg: This module requires the OpenShift Python client. Try `pip install openshift`

First thing I tried was verifying openshift was installed in the python being used:

$ /usr/local/opt/python@2/bin/python2.7 -m pip freeze | grep openshift
openshift==0.8.0

(I also tried everything with openshift==0.8.1 just to rule out version issues.).

That's odd.

So then I added a debug statement to the playbook:

  pre_tasks:
    - python_requirements_facts:
        dependencies: openshift

And the output:

TASK [python_requirements_facts] ***********************************************
task path: /Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/test.yml:12
Using module file /usr/local/lib/python2.7/site-packages/ansible/modules/system/python_requirements_facts.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: jgeerling
<127.0.0.1> EXEC /bin/sh -c '/usr/local/opt/python@2/bin/python2.7 && sleep 0'
ok: [127.0.0.1] => changed=false 
  invocation:
    module_args:
      dependencies:
      - openshift
  mismatched: {}
  not_found: []
  python: /usr/local/opt/python@2/bin/python2.7
  python_system_path:
  - /var/folders/mt/rp7cb3s95qzfkrh1dk_8ckfc0000gn/T/ansible_python_requirements_facts_payload_JLohA7/ansible_python_requirements_facts_payload.zip
  - ''
  - /usr/local/lib/python2.7/site-packages
  - /usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python27.zip
  - /usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7
  - /usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
  - /usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
  - /usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
  - /usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
  - /usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old
  - /usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload
  - /Users/jgeerling/Library/Python/2.7/lib/python/site-packages
  - /usr/local/lib/python2.7/site-packages
  python_version: |-
    2.7.15 (default, Jul 17 2018, 19:41:20)
    [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)]
  valid:
    openshift:
      desired: null
      installed: 0.8.0

So... what gives?

[ansibot]

Files identified in the description:

• lib/ansible/modules/clustering/k8s/k8s.py

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibot]

cc @chouseknecht @fabianvf @flaper87 @maxamillion @willthames
click here for bot help

[geerlingguy]

Created a test python script:

try:
    import kubernetes
    import openshift
    from openshift.dynamic import DynamicClient
    from openshift.dynamic.exceptions import ResourceNotFoundError, ResourceNotUniqueError
    print "Found it"
except ImportError:
    print "Didnt find it"

Ran it:

$ /usr/local/opt/python@2/bin/python2.7 test.py 
/usr/local/lib/python2.7/site-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.24.1) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
Didnt find it

So... looks like there's some weird error with urllib3 and/or chardet, which tosses an exception, even though openshift and kubernetes _are_ available. And so that fails Ansible's check, but that exception is completely gobbled up by Ansible and there's no way for me to tell that without reproducing the test code and finding the exception on my own.

Is there any way to bubble up the actual exception instead of just blindly setting HAS_K8S_MODULE_HELPER = False (which is not quite true, since openshift _is_ verifiably present)?

[geerlingguy]

So in my case, it was an annoying Requests-related exception (actually just a RequestsDependencyWarning, but somehow it was flagging it as an exception?), and I fixed it with:

$ pip install --upgrade requests

(For context, see https://github.com/requests/requests/issues/4673)

Now everything is working happily.

But I want to leave this open in case it's deemed a bug that the k8s module doesn't bubble up exceptions and has (IMO) a little bit of a fragile check for whether openshift is actually installed (or maybe that error message could be made a little less strict—e.g. 'it could be that openshift is not installed').

[fabianvf]

@geerlingguy thanks a ton for the investigation, that's definitely a weird one. I'll try and get that error properly bubbled up.

[ClaudioDeFacci]

Same here:

Ansible 2.7.5
Python 2.7.15rc1
Openshift 0.8.2

I am simply using the k8s module and get the above error.
Last suggestion of geerlingguy (upgrading requests) did not help.
Hope this is no bigger issue.
At this point I am pretty upset as I do not know Python at all and am unable to contribute to a solution on this.
I would appreciate any help to get this solved.
Thank you!

[ghost]

+1

[ghost]

Tested in 2.6.11 version also ... same error and same issue.

kylix3511  (e) venv260  ~  k8s  learning  cat ansible2.yaml 
---
- hosts: master 
  connection: local

  tasks:
    - name: Create a kubernetes namespace
      k8s_raw:
        name: testing
        api_version: v1
        kind: Namespace
        state: present
 kylix3511  (e) venv260  ~  k8s  learning 

Error Message :

````

 kylix3511  (e) venv260  ~  k8s  learning  ansible-playbook ansible2.yaml -vvv
ansible-playbook 2.6.11
config file = /Users/kylix3511/k8s/learning/ansible.cfg
configured module search path = [u'/Users/kylix3511/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /Users/kylix3511/venv260/lib/python2.7/site-packages/ansible
executable location = /Users/kylix3511/venv260/bin/ansible-playbook
python version = 2.7.10 (default, Aug 17 2018, 19:45:58) [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)]
Using /Users/kylix3511/k8s/learning/ansible.cfg as config file
Parsed /Users/kylix3511/k8s/learning/hosts inventory source with ini plugin

PLAYBOOK: ansible2.yaml ********************************************
1 plays in ansible2.yaml

PLAY [master] **********************************************

TASK [Gathering Facts] *******************************************
task path: /Users/kylix3511/k8s/learning/ansible2.yaml:2
ESTABLISH LOCAL CONNECTION FOR USER: kylix3511
EXEC /bin/sh -c 'echo ~kylix3511 && sleep 0'
EXEC /bin/sh -c '( umask 77 && mkdir -p "echo /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978639.55-171007548805666" && echo ansible-tmp-1546978639.55-171007548805666="echo /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978639.55-171007548805666" ) && sleep 0'
Using module file /Users/kylix3511/venv260/lib/python2.7/site-packages/ansible/modules/system/setup.py
PUT /Users/kylix3511/.ansible/tmp/ansible-local-2599671Udp6/tmpnqhyGB TO /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978639.55-171007548805666/setup.py
EXEC /bin/sh -c 'chmod u+x /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978639.55-171007548805666/ /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978639.55-171007548805666/setup.py && sleep 0'
EXEC /bin/sh -c '/usr/bin/python /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978639.55-171007548805666/setup.py && sleep 0'
EXEC /bin/sh -c 'rm -f -r /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978639.55-171007548805666/ > /dev/null 2>&1 && sleep 0'
ok: [master]
META: ran handlers

TASK [Create a kubernetes namespace] ***************************************
task path: /Users/kylix3511/k8s/learning/ansible2.yaml:6
ESTABLISH LOCAL CONNECTION FOR USER: kylix3511
EXEC /bin/sh -c 'echo ~kylix3511 && sleep 0'
EXEC /bin/sh -c '( umask 77 && mkdir -p "echo /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978640.38-224769996401319" && echo ansible-tmp-1546978640.38-224769996401319="echo /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978640.38-224769996401319" ) && sleep 0'
Using module file /Users/kylix3511/venv260/lib/python2.7/site-packages/ansible/modules/clustering/k8s/_k8s_raw.py
PUT /Users/kylix3511/.ansible/tmp/ansible-local-2599671Udp6/tmpOdDMKi TO /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978640.38-224769996401319/_k8s_raw.py
EXEC /bin/sh -c 'chmod u+x /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978640.38-224769996401319/ /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978640.38-224769996401319/_k8s_raw.py && sleep 0'
EXEC /bin/sh -c '/usr/bin/python /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978640.38-224769996401319/_k8s_raw.py && sleep 0'
EXEC /bin/sh -c 'rm -f -r /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978640.38-224769996401319/ > /dev/null 2>&1 && sleep 0'
fatal: [master]: FAILED! => {
"changed": false,
"invocation": {
"module_args": {
"api_key": null,
"api_version": "v1",
"cert_file": null,
"context": null,
"force": false,
"host": null,
"key_file": null,
"kind": "Namespace",
"kubeconfig": null,
"name": "testing",
"namespace": null,
"password": null,
"resource_definition": null,
"src": null,
"ssl_ca_cert": null,
"state": "present",
"username": null,
"verify_ssl": null
}
},
"msg": "This module requires the OpenShift Python client. Try pip install openshift"
}

PLAY RECAP ***********************************************
master : ok=1 changed=0 unreachable=0 failed=1

 kylix3511  (e) venv260  ~  k8s  learning  

````

[fabianvf]

If you have openshift installed in a virtual environment, you may need to specify the proper python intepreter for Ansible to use (by default it will use your system python). You can do that by setting the ansible_python_interpreter variable to ansible_playbook_python. Your new play would look like:

- hosts: master 
  connection: local
  vars:
    ansible_python_interpreter: '{{ ansible_playbook_python }}'
  tasks:
    - name: Create a kubernetes namespace
      k8s_raw:
        name: testing
        api_version: v1
        kind: Namespace
        state: present

Does that fix your issue @kylix3511 or @ClaudioDeFacci?

Edit: Including Will Thames' excellent blog post on this subject: https://willthames.github.io/2018/07/01/connection-local-vs-delegate_to-localhost.html

[ghost]

Thanks @fabianvf.

vars:
ansible_python_interpreter: '{{ ansible_playbook_python }}'

Thats helpful and worked.

Now my goal is to use ansible AWX to use the same operations. Any idea to how to do that ? Have you done that before ?

On Jan 8, 2019, at 12:27 PM, Fabian von Feilitzsch [e-mail] wrote:

If you have openshift installed in a virtual environment, you may need to specify the proper python intepreter for Ansible to use (by default it will use your system python). You can do that by setting the ansible_python_interpreter variable to ansible_playbook_python. Your new play would look like:

• hosts: master
  connection: local
  vars:
  ansible_python_interpreter: '{{ ansible_playbook_python }}'
  tasks:
  
  
  
  • name: Create a kubernetes namespace
    
    k8s_raw:
    
    name: testing
    
    api_version: v1
    
    kind: Namespace
    
    state: present
    
    Does that fix your issue @kylix3511 https://github.com/kylix3511 or @ClaudioDeFacci https://github.com/ClaudioDeFacci?
  
  

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/ansible/ansible/issues/50529#issuecomment-452439820, or mute the thread https://github.com/notifications/unsubscribe-auth/Ak2Op5Ip7fVoHH24brKpkwZCGn4wwuxnks5vBP87gaJpZM4ZpCBb.

[fabianvf]

No, I haven't used AWX very extensively so I can't really be of much help there. I'd recommend asking in IRC.

[geerlingguy]

Just hit this again today on my other Mac, after I ran a brew upgrade and the brew python was relinked. D'oh! Luckily I googled and found this issue pretty quickly :)

On this computer, the problem _was_ that the system Python (/usr/bin/python) was being used for Ansible, even though my current environment was using the Brew-managed Python (/usr/local/bin/python).

So I ran:

$ /usr/bin/python -m pip install --upgrade --user openshift

And now I'm back in business... after also updating kubernetes to 8.0.1.

[benwsapp]

@kylix3511 we got this working in AWX by setting up a custom virtual environment, like here so you have the proper pip dependencies installed, and by setting the "ansible_python_interpreter" with an extra var in the job template in AWX. then, locally just setting the "ansible_python_interpreter as global group_var to "ansible_playbook_python" as @fabianvf had said.

it seems like it might be a bug within the AWX custom environment feature. like the error checking for the playbook isn't using the custom venv or something.

[willthames]

@geerlingguy - rather than updating system python, it's better to set ansible_python_interpreter to ansible_playbook_python as @fabianvf suggests.

Also, for anyone reporting these issues, it's hugely helpful to put a python_requirements_facts task in your playbook so that you can see what is actually being run.

For example:

- python_requirements_facts:
    dependencies:
      - openshift
      - requests

would be helpful here

[ghost]

Thanks Will. python_requriments_facts is very useful tip. Thanks so much.

I try to use python_requriments_facts, it is not working as expected. Please help if you got some time.

kylix3511  (e) venv280  ~  ansible  helloWorldApp  cat deployHelloWorld.yaml

---

• hosts: localhost
  connection: local

  vars:
  ansible_python_interpreter: '{{ ansible_playbook_python }}'

  • python_requirements_facts:
    dependencies:
    
    
    
    • openshift
    
    
    • requests
    
    

  tasks:

  • name: Deploying the simple HelloWorld App in Kylix3511 Namespace
    k8s:
    state: present
    definition:
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
    annotations:
    deployment.kubernetes.io/revision: "1"
    labels:
    run: myhelloworld10
    name: myhelloworld10
    namespace: “kylix3511"
    spec:
    replicas: 5
    selector:
    matchLabels:
    run: myhelloworld10
    template:
    metadata:
    labels:
    run: myhelloworld10
    spec:
    containers:
    - image: hub.docker.com/images/myhelloworldapp10:v5
    imagePullPolicy: Always
    name: myhelloworld10

On Feb 2, 2019, at 4:10 PM, Will Thames [e-mail] wrote:

@geerlingguy https://github.com/geerlingguy - rather than updating system python, it's better to set ansible_python_interpreter to ansible_playbook_python as @fabianvf https://github.com/fabianvf suggests.

Also, for anyone reporting these issues, it's hugely helpful to put a python_requirements_facts task in your playbook so that you can see what is actually being run.

For example:

• python_requirements_facts:
  dependencies:
  
  
  
  • openshift
  
  
  • requests
    
    would be helpful here
  
  

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/ansible/ansible/issues/50529#issuecomment-460010809, or mute the thread https://github.com/notifications/unsubscribe-auth/Ak2Op2cUxqRbLmZghYlMDWXR5AlBQqDWks5vJikCgaJpZM4ZpCBb.

[benwsapp]

Thanks Will. python_requriments_facts is very useful tip. Thanks so much. I try to use python_requriments_facts, it is not working as expected. Please help if you got some time. kylix3511  (e) venv280  ~  ansible  helloWorldApp  cat deployHelloWorld.yaml
…
--- - hosts: localhost connection: local vars: ansible_python_interpreter: '{{ ansible_playbook_python }}' - python_requirements_facts: dependencies: - openshift - requests tasks: - name: Deploying the simple HelloWorld App in Kylix3511 Namespace k8s: state: present definition: apiVersion: extensions/v1beta1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: "1" labels: run: myhelloworld10 name: myhelloworld10 namespace: “kylix3511" spec: replicas: 5 selector: matchLabels: run: myhelloworld10 template: metadata: labels: run: myhelloworld10 spec: containers: - image: hub.docker.com/images/myhelloworldapp10:v5 imagePullPolicy: Always name: myhelloworld10
On Feb 2, 2019, at 4:10 PM, Will Thames @.*> wrote: @geerlingguy https://github.com/geerlingguy - rather than updating system python, it's better to set ansible_python_interpreter to ansible_playbook_python as @fabianvf https://github.com/fabianvf suggests. Also, for anyone reporting these issues, it's hugely helpful to put a python_requirements_facts task in your playbook so that you can see what is actually being run. For example: - python_requirements_facts: dependencies: - openshift - requests would be helpful here — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#50529 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/Ak2Op2cUxqRbLmZghYlMDWXR5AlBQqDWks5vJikCgaJpZM4ZpCBb.

looks like you have a typo there? - here's the reference for proper syntax.

otherwise, are you getting output at all? or just an error? should show what requirements are and are not being met.

[ghost]

Thanks @benwsapp. Yes. I did try that ... still same error.

````
ansible-playbook deployHelloWorld.yaml -vvvvv
ansible-playbook 2.8.0.dev0
config file = None
configured module search path = [u'/Users/kylix3511/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /Users/kylix3511/venv280/lib/python2.7/site-packages/ansible
executable location = /Users/kylix3511/venv280/bin/ansible-playbook
python version = 2.7.10 (default, Aug 17 2018, 19:45:58) [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)]
No config file found; using defaults
setting up inventory plugins
match 'all'

ERROR! Syntax Error while loading YAML.
expected , but found ''

The error appears to have been in '/Users/kylix3511/ansible/helloWorldApp/deployHelloWorld.yaml': line 10, column 5, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

tasks:
^ here

```

[willthames]

@kylix3511 - the python_requirements_facts is a task, so needs to be in the task list:

    tasks:
    - python_requirements_facts:
        dependencies:
          - openshift
          - requests

    - name: Deploying the simple HelloWorld App in Kylix3511 Namespace
      k8s:
        state: present
        ...

[ghost]

Thanks @willthames

It works as expected.

TASK [python_requirements_facts] ***********************************************************************************************************************************************************************************************************
task path: /Users/kylix3511/ansible/helloWorldApp/deployHelloWorld.yaml:11
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: kylix3511
<127.0.0.1> EXEC /bin/sh -c 'echo ~kylix3511 && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/kylix3511/.ansible/tmp/ansible-tmp-1549176705.49-269941163809430 `" && echo ansible-tmp-1549176705.49-269941163809430="` echo /Users/kylix3511/.ansible/tmp/ansible-tmp-1549176705.49-269941163809430 `" ) && sleep 0'
Using module file /Users/kylix3511/venv280/lib/python2.7/site-packages/ansible/modules/system/python_requirements_facts.py
<127.0.0.1> PUT /Users/kylix3511/.ansible/tmp/ansible-local-45759DkpzFE/tmpzttzsf TO /Users/kylix3511/.ansible/tmp/ansible-tmp-1549176705.49-269941163809430/AnsiballZ_python_requirements_facts.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /Users/kylix3511/.ansible/tmp/ansible-tmp-1549176705.49-269941163809430/ /Users/kylix3511/.ansible/tmp/ansible-tmp-1549176705.49-269941163809430/AnsiballZ_python_requirements_facts.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/Users/kylix3511/venv280/bin/python /Users/kylix3511/.ansible/tmp/ansible-tmp-1549176705.49-269941163809430/AnsiballZ_python_requirements_facts.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /Users/kylix3511/.ansible/tmp/ansible-tmp-1549176705.49-269941163809430/ > /dev/null 2>&1 && sleep 0'
ok: [localhost] => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "dependencies": [
                "openshift", 
                "requests"
            ]
        }
    }, 
    "mismatched": {}, 
    "not_found": [], 
    "python": "/Users/kylix3511/venv280/bin/python", 
    "python_system_path": [
        "/var/folders/yq/064210y10wz31j_rhz_zp8p40000gn/T/ansible_python_requirements_facts_payload_MfhH23/ansible_python_requirements_facts_payload.zip", 
        "/Users/kylix3511/venv280/lib/python27.zip", 
        "/Users/kylix3511/venv280/lib/python2.7", 
        "/Users/kylix3511/venv280/lib/python2.7/plat-darwin", 
        "/Users/kylix3511/venv280/lib/python2.7/plat-mac", 
        "/Users/kylix3511/venv280/lib/python2.7/plat-mac/lib-scriptpackages", 
        "/Users/kylix3511/venv280/lib/python2.7/lib-tk", 
        "/Users/kylix3511/venv280/lib/python2.7/lib-old", 
        "/Users/kylix3511/venv280/lib/python2.7/lib-dynload", 
        "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7", 
        "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin", 
        "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk", 
        "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac", 
        "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages", 
        "/Users/kylix3511/venv280/lib/python2.7/site-packages"
    ], 
    "python_version": "2.7.10 (default, Aug 17 2018, 19:45:58) \n[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)]", 
    "valid": {
        "openshift": {
            "desired": null, 
            "installed": "0.8.2"
        }, 
        "requests": {
            "desired": null, 
            "installed": "2.20.1"
        }
    }
}

[ghost]

Did any body tried and deployed a simple HelloWorld application in Kubernetes via ansible ?? I'm trying to do that last few days ...

[ghost]

Hi Ansible Team,

https://docs.ansible.com/ansible/latest/plugins/lookup/k8s.html?highlight=k8s https://docs.ansible.com/ansible/latest/plugins/lookup/k8s.html?highlight=k8s

Both are looks same … seems a typo.

Regards,
Kylix

[willthames]

@kylix, in your original output, you can see that python is /usr/bin/python, but in the response from python_requirements_facts python is /Users/kylix3511/venv280/bin/python. This is presumably because you've since fixed ansible_playbook_python

Can you run the same playbook again with -vvv including the python_requirements_facts and the k8s task - I'm trying to see what problem you're hitting now.

[willthames]

Could you report the docs issue as a separate bug - I'd like to keep this one focused on the problem you're having

[Asgoret]

same error.
Version:
ansible - 2.7.2
openshift module - 0.8.4
Python - 2.7.10

Don't work:
Set python_requirements_facts
Set ansible_playbook_python
Set ansible_python_interpreter

Any workarounds?

[fabianvf]

This usually isn't an issue with the Python client or with the Ansible module, it's a core behavior of Ansible, and if you try to run any other modules that require Python dependencies against localhost, you should see the same behavior. When Ansible connects to a machine, even localhost, it will default to using the system installation of Python (ie, /usr/bin/python). If you've installed modules in a different environment than the system installation, they will not be present in the system Python path. To get the behavior you expect (run Ansible, detect modules installed in environment), you need to ensure that Ansible is detecting the proper Python interpreter when it connects. Oftentimes this is fixed by setting ansible_python_interpreter: '{{ ansible_playbook_python }}', which should work so long as Ansible is installed in the same environment as your dependencies.

You can use the python_requirements_facts module to determine if Ansible sees the dependencies when connecting to the machine. If you post the output of the following playbook, I can help debug:

- hosts: localhost
  tasks:
  - python_requirements_facts:
      dependencies:
      - openshift

- hosts: localhost
  vars:
    ansible_python_interpreter: '{{ ansible_playbook_python }}'
  tasks:
  - python_requirements_facts:
      dependencies:
      - openshift

For more technical background on the reason for this behavior, I will again link to Will's breakdown: https://willthames.github.io/2018/07/01/connection-local-vs-delegate_to-localhost.html

[ghost]

Thanks Fabian. Thats a super awesome explanation and information.

For some reason, the same thing is not working in Ansible AWX, I’m creaking my head on it … still no luck.

Thanks.

Regards,
kylix3511

Email : [email protected]

Sent from  iPhone

On Feb 8, 2019, at 10:13 AM, Fabian von Feilitzsch [e-mail] wrote:

This usually isn't an issue with the Python client or with the Ansible module, it's a core behavior of Ansible, and if you try to run any other modules that require Python dependencies against localhost, you should see the same behavior. When Ansible connects to a machine, even localhost, it will default to using the system installation of Python (ie, /usr/bin/python). If you've installed modules in a different environment than the system installation, they will not be present in the system Python path. To get the behavior you expect (run Ansible, detect modules installed in environment), you need to ensure that Ansible is detecting the proper Python interpreter when it connects. Oftentimes this is fixed by setting ansible_python_interpreter: '{{ ansible_playbook_python }}', which should work so long as Ansible is installed in the same environment as your dependencies.

You can use the python_requirements_facts module to determine if Ansible sees the dependencies when connecting to the machine. If you post the output of the following playbook, I can help debug:

---

• hosts: localhost
  tasks:

  • python_requirements_facts:
    dependencies:
    
    
    
    • openshift
    
    

• hosts: localhost
  vars:
  ansible_python_interpreter: '{{ ansible_playbook_python }}'
  tasks:

  • python_requirements_facts:
    dependencies:
    
    
    
    • openshift
      
      For more technical background on the reason for this behavior, I will again link to Will's breakdown: https://willthames.github.io/2018/07/01/connection-local-vs-delegate_to-localhost.html https://willthames.github.io/2018/07/01/connection-local-vs-delegate_to-localhost.html
      
      —
      
      You are receiving this because you were mentioned.
      
      Reply to this email directly, view it on GitHub https://github.com/ansible/ansible/issues/50529#issuecomment-461894431, or mute the thread https://github.com/notifications/unsubscribe-auth/Ak2Op7IyG6t-coVLuEw9_P06ZjHUJPmkks5vLb4vgaJpZM4ZpCBb.
    
    

[Asgoret]

@fabianvf It doesn't work at this moment.
My task:

- name: Test secret
  hosts: localhost
  connection: local
  vars:
    ansible_python_interpreter: "{{ ansible_playbook_python }}"
  tasks:
    - python_requirements_facts:
        dependencies:
        - openshift
    - name: Set simple object state="{{ state }}"
      delegate_to: localhost
      k8s:
        api_key: "{KEY}"
        host: "{URL}"
        state: present
        definition: secret.yaml.j2

And i get:

If it's matter i work in macOS Mojave(10.14.2)

SOLVED
Solvent:
Got to sudo:

sudo su

Update ALL packages:

pip list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1  | xargs -n1 pip install -U

Then you catch list of errors (like six). For this packages use:

pip install --ignore-installed {package_name}

Take in loop last two commands until all packages in your system will be last stable version.

[fabianvf]

@Asgoret glad you solved it, just fyi I believe the delegate_to: localhost you had in there made the ansible_python_interpreter variable not matter anymore, so it was still running against your system Python at that point.

[Asgoret]

@fabianvf I test simple construction (only hosts: localhost) and it works) so the problem was in some old packages. Is there any opportunity that this behavior will have another error message except pip install openshift?

[willthames]

@Asgoret newer versions of the k8s module have slightly improved error messages showing python path etc.

[dglinder]

@kylix3511 wrote:

For some reason, the same thing is not working in Ansible AWX, I’m creaking my head on it … still no luck.

Same here, RedHat 7.6, ansible 2.7.7, Python 2.7.5, except Tower 3.3.4 and using a custom Ansible Environment that has been setup just for this task (See playbook below) - the Tower documentation and RH support recommend this.

When we specify the custom environment via Tower/AWX we still get the error. The output from python_requirements_facts: for Towers "/var/lib/awx/venv/Kubernetes/" venv shows output like this:

{
    "_ansible_parsed": true,
    "python_version": "2.7.5 (default, Sep 12 2018, 05:31:16) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]",
    "_ansible_delegated_vars": {
        "ansible_delegated_host": "localhost",
        "ansible_host": "localhost"
    },
    "python": "/usr/bin/python2",
    "changed": false,
    "python_system_path": [
        "/tmp/ansible_python_requirements_facts_payload_n4PpL5/ansible_python_requirements_facts_payload.zip",
        "/var/lib/awx/venv/Kubernetes/lib/python2.7/site-packages",
        "/var/lib/awx/projects/_551__dglinder_test_k8s/KubernetesPlaybooks",
        "/usr/lib64/python27.zip",
        "/usr/lib64/python2.7",
        "/usr/lib64/python2.7/plat-linux2",
        "/usr/lib64/python2.7/lib-tk",
        "/usr/lib64/python2.7/lib-old",
        "/usr/lib64/python2.7/lib-dynload",
        "/usr/lib64/python2.7/site-packages",
        "/usr/lib64/python2.7/site-packages/gtk-2.0",
        "/usr/lib/python2.7/site-packages"
    ],
    "valid": {
        "requests": {
            "desired": null,
            "installed": "2.21.0"
        },
        "k8s": {
            "desired": null,
            "installed": "0.11.0"
        },
        "openshift": {
            "desired": null,
            "installed": "0.8.7"
        }
    },
    "mismatched": {},
    "invocation": {
        "module_args": {
            "dependencies": [
                "openshift",
                "requests",
                "k8s"
            ]
        }
    },
    "not_found": [],
    "_ansible_no_log": false
}

The Tower virtual environment is setup on all Tower nodes using this playbook:

#!/usr/bin/env ansible-playbook
---
- hosts: all
  gather_facts: false

  vars:
    venvs:
      # Setup the Kubernetes virtual environment
      - dirname: Kubernetes
        modules:
          - k8s
          - openshift
          - requests
        prereqs:
          - gcc


  tasks:
  - name: "Install Python virtual environment packages"
    yum:
      name:
        - python-virtualenv
      state: latest

  - name: "Install pip module required packages"
    yum:
      name: "{{ item.prereqs }}"
      state: latest
    loop: "{{ venvs }}"
    when: item.prereqs is defined

  - name: "Create the venv skeleton directory"
    file:
      path: "/var/lib/awx/venv/{{ item.dirname }}"
      state: directory
    become: true
    loop: "{{ venvs }}"

  - name: "Install base Tower packages"
    pip:
      name:
        - python-memcached
        - psutil
      umask: "0022"
      virtualenv: "/var/lib/awx/venv/{{ item.dirname }}"
    loop: "{{ venvs }}"

  - name: "Install requested packages in virtual env"
    pip:
      name: "{{ item.modules }}"
      umask: "0022"
      state: latest
      virtualenv: "/var/lib/awx/venv/{{ item.dirname }}"
    loop: "{{ venvs }}"

[ghost]

+1

[mukeshmurali]

My problem was resolved with

pip install openshift==0.8.4 --ignore-installed

Havent worked out whats happening behind scenes :(

[dglinder]

@mukeshmurali was your fix for the Tower/AWX installation, or another command line environment fix? I tried your --ignore-installed option on my Tower virtual environment (see above) but it didn't resolve things for us.

[dglinder]

Two questions for those more knowledgeable in "pip" and Python modules:
1: Instead of running pip install openshift in the venv, how about running it in the main OS environment? Will those modules then be available to the Tower execution environment?
2: How is the check for openshift or other modules done? Is it possible that some Tower code has a hard-coded path that ignores the venv?

P.S. I'm still learning about Python and 'pip' so excuse me in advance if those questions are bad.

[willthames]

Is https://github.com/ansible/awx/issues/2462#issuecomment-478988817 helpful here?

[dglinder]

So for those who have struggled with this, my Ansible Tower clusters now have a working solution too. Basically we:

1. Use pip to install the openshift Python module into the OS environment
   
   
   
   • I am pretty confident that this was the main solution but I haven't bisected the changes to validate this.
   
   
2. Setup the AWX / Ansible Tower virtual environment in /var/lib/awx/venv/Kubernetes
3. Add the Tower required base packages (python-memcached and psutil)
4. Add the k8s, openshift, and requests packages.

Pip package installation used the --ignore-installed option and used the site packages as well so the openshift module in the first step (above) was visible. (At least thats what I believe it happening.)

This leads me to believe that there's some "leakage" of the virtual environments within the Tower system that are not properly sandboxing the k8s module within the venv built with the openshift module. But I'm too much of a Python novice to successfully track that down.

I hope this helps others, if only as a band-aid until the real solution is found.

Below is the Ansible playbook I wrote to set this up on my Tower servers:

#!/usr/bin/env ansible-playbook
# Installs the company specific virtual environments for Python properly
# into the Tower environment following the Ansible Tower environment notes:
# URL: https://docs.ansible.com/ansible-tower/3.3.4/html/upgrade-migration-guide/virtualenv.html
# 
# NOTE: Requires the EPEL repository to install the python2-pip package.
#
# Usage:
#   $ sudo ./setup_venvs.yml -i inventory.sand_swn01.ini
#
---
- hosts:
    - tower
    - playground
  gather_facts: false

  vars:
    venvs:
      # Setup the Kubernetes virtual environment
      - dirname: Kubernetes
        modules:
          - k8s
          - openshift
          - requests
        prereqs:
          - gcc

  tasks:
  - name: "Ensure pip is installed"
    # NOTE: This is from the EPEL repository.
    yum:
      name: python2-pip
      state: latest

  - name: "Ensure pip has latest tools"
    # Notes taken from the Ansible container page:
    # https://docs.ansible.com/ansible-container/installation.html
    pip:
      name:
        - setuptools
      extra_args: --upgrade

  - name: "Install packages OUTSIDE Tower virtual environments"
    # Necessary to make 'openshift' for the 'k8s' module.
    pip:
      name:
        - openshift
      extra_args: --ignore-installed

  - name: "Install Python virtual environment packages"
    yum:
      name:
        - python-virtualenv
      state: latest

  - name: "Install pip module required packages"
    yum:
      name: "{{ item.prereqs }}"
      state: latest
    loop: "{{ venvs }}"
    when: item.prereqs is defined

  - name: "Clean out old venv directories"
    file:
      path: "/var/lib/awx/venv/{{ item.dirname }}"
      state: absent
    become: true
    loop: "{{ venvs }}"

  - name: "Create the venv skeleton directory"
    file:
      path: "/var/lib/awx/venv/{{ item.dirname }}"
      state: directory
    become: true
    loop: "{{ venvs }}"

  - name: "Install base Tower packages"
    pip:
      name:
        - python-memcached
        - psutil
      umask: "0022"
      state: latest
      virtualenv: "/var/lib/awx/venv/{{ item.dirname }}"
      virtualenv_site_packages: yes
      extra_args: --ignore-installed
    loop: "{{ venvs }}"

  - name: "Install requested packages in virtual env"
    pip:
      name: "{{ item.modules }}"
      umask: "0022"
      state: latest
      virtualenv: "/var/lib/awx/venv/{{ item.dirname }}"
      virtualenv_site_packages: yes
      extra_args: --ignore-installed
    loop: "{{ venvs }}"

Edit 1 - Update playbook to show the "pip update setuptools" step.

[dglinder]

So, the fix here was to add these Python modules to the underlying core OS package "World" and not just to the Tower virtual environments. Is this a problem in the way Python and/or the "k8s" module checks for pre-requisites, or is it a AWX pre-requisite checking failure?

[fabianvf]

As far as I know the k8s module isn't doing anything different with module discovery/loading, so I would guess the issue is something to do with Tower + virtualenv + ansible python interpreter discovery.

[mitchellmaler]

I was having this issue on a Mac using the homebrew version. My fix was to remove the homebrew version of Ansible and install using pip.
brew remove ansible && pip3 install --user ansible

[dglinder]

As far as I know the k8s module isn't doing anything different with module discovery/loading, so I would guess the issue is something to do with Tower + virtualenv + ansible python interpreter discovery.

I'm suspecting that too, but my support engineer at RedHat says that since the "k8s" module is not part of the core that RedHat provides, his options are limited. I guess I'll take my work-around for now and hope that future modules the teams need don't require a conflicting module.

If anyone has any pointers on where to start debugging the version checking within the virtualenv I'm all ears.

[willthames]

python_requirements_facts is really the only thing we have to see what is going on within a virtualenv - but that will only tell you that something is wrong, not why.

There does seem to be a parallel AWX issue for this (ansible/awx#2462) - this problem seems to be more about AWX/Tower than the k8s module, so perhaps it's worth following up there?

[dglinder]

There does seem to be a parallel AWX issue for this (ansible/awx#2462) - this problem seems to be more about AWX/Tower than the k8s module, so perhaps it's worth following up there?

Thanks @willthames that does seem quite familiar.

[OJFord]

Is there even a working workaround for this issue?

I'm not (knowingly, I've never heard of it) using AWX Tower, I'm just running ansible on a Mac against a Linux remote. Initially I was running brew ansible in a pyenv virtual environment with openshift installed, but following above comments I've tried installing both ansible and openshift to system python environment, and that ansible still gives this error.

The error doesn't make it clear if openshift should be installed on local or remote, so before I found this issue I was adding tasks to install pip and openshift on the remote, which hasn't helped either.

[willthames]

@OJFord - usually it's almost always that if you're using connection: local, you need to set ansible_python_interpreter: {{ ansible_playbook_python }}

Openshift should be installed on whatever host ansible is running on.

As discussed earlier in the thread, you can use python_requirements_facts to ensure that you're running the version of python you think you are and the versions of openshift and kubernetes libraries that you think you are

[OJFord]

@willthames Thanks - so k8s[_*] modules should be run with connection: local? Initially I wasn't doing that, before flailing around 'trying to try' everything in this thread, but I suppose it does make most sense.

I still can't make it work though.
hosts:

[local]
localhost ansible_python_interpreter="{{ ansible_playbook_python }}"

python_requirements_facts:

{"valid": {"openshift": {"desired": null, "installed": "0.8.8"}, "requests": {"desired": null, "installed": "2.21.0"}, ...}

and shows the python from pyenv as expected.

/that/path/to/python -m pip install openshift confirms it's installed.

k8s_facts with connection: local still gives:

{"changed": false, "msg": "This module requires the OpenShift Python client. Try `pip install openshift`"}

[willthames]

@OJFord can you run the playbook with -vvv - that should show what python is being run and whether it's the one you expect.

[OJFord]

@willthames That actually shows it's using /usr/bin/python instead of pyenv's - but I previously installed openshift to _that_ python's environment too, and just double-checked that.

It runs:

[IP] EXEC /bin/sh -c '/usr/bin/python [.ansible]/tmp/ansible-tmp-1557386156.751379-236194994805047/AnsiballZ_k8s_facts.py && sleep 0'

and gives the same error about installing openshift.

/usr/bin/python -m pip install openshift

confirms it _is_ installed.

[willthames]

Ok, if you're seeing [IP] I'm pretty sure it's not running locally, it's running on a remote host which itself doesn't have openshift installed. That also explains why ansible_python_interpreter isn't being respected (I think)

Are you trying to run ansible on a target host to make kubernetes api calls? Do you have a reason to do that (either way is fine, but I'm troubleshooting a problem without all of the information)

You're only giving me minimal snippets of information, this would be much better investigated as a new separate issue, with a proper minimal reproducible test case (https://stackoverflow.com/help/mcve) - creating such a thing is often a good path to isolating the issue and finding the cause of the issue yourself.

[willthames]

I'm going to close this issue now. If people are having an issue that appears similar to this one, please open a new issue fully describing your problem.