Awx: K8S Install Fails in AWX 9.0.0 (postgresql version check fails)

[Quantas]

ISSUE TYPE

• Bug Report

SUMMARY

With a fresh copy of the awx repository, on a brand new K8S cluster, with only helm/tiller installed. The installer fails at the "Check if Postgres 9.6 is being used" step with 2 separate failures. The first is that it cannot find the pod because of the wrong label being used. The second is once the label check issue is resolved it cannot connect to postgresql because it doesn't know the password.

ENVIRONMENT

• AWX version: 9.0.0 (devel branch)
• AWX install method: kubernetes (GKE and Rancher custom cluster)
• Ansible version: 2.5.1
• Operating System: Windows with WSL2 to use kubectl/helm/ansible
• Web Browser: Chrome

STEPS TO REPRODUCE

1. Setup a brand new Kubernetes cluster (GKE, Rancher, etc.)
2. Clone a fresh copy of devel (as of 11/3/2019, 5246c842)
3. Ensure Helm is installed on cluster according to official documentation
4. Configure inventory, setting kubernetes_context, kubernetes_namespace, tiller_namespace, leave everything else default
5. ansible-playbook -i inventory install.yml
6. First this fails because it cannot find the kubernetes pod with the label=name=postgrtesql, I altered installer/roles/kubernetes/tasks/main.yml to change this to look for label=app=postgresql
7. The second fail is because it doesn't know the password for postgresql when checking the version, so I altered the check to use a URL and passed in pg_username:

{{ kubectl_or_oc }} exec -ti $POD -n {{ kubernetes_namespace }} -- bash -c "psql \"postgresql://{{ pg_username }}:{{ pg_password }}@localhost\" -tAc 'select version()'"

EXPECTED RESULTS

Clean cluster install with no modifications to the Ansible scripts.

ACTUAL RESULTS

Install fails trying to find postgresql and then trying to connect to it once script was altered.

ADDITIONAL INFORMATION

Using the exact same steps above, the install is successful on the 7.0.0 tag.

[calvinbui]

I was able to fix this by changing the task to:

- name: Check if Postgres 9.6 is being used
  shell: |
    POD=$({{ kubectl_or_oc }} -n {{ kubernetes_namespace }} \
      get pods -l=app=postgresql --field-selector status.phase=Running -o jsonpath="{.items[0].metadata.name}")
    {{ kubectl_or_oc }} exec -ti $POD -n {{ kubernetes_namespace }} -- bash -c "PGPASSWORD={{ pg_password }} psql -U {{ pg_username }} -tAc 'select version()'"
  register: pg_version

[Quantas]

@calvinbui Were you able to then have AWX start successfully? I noticed that even after I changed these values, the dashboard was still inaccessible.

[calvinbui]

No, it will fail due to https://github.com/ansible/awx/pull/5218/files

[dmaple-gci]

This check also fails if the postgressql database is external to AWX. That task should have a conditional to check if pg_hostname is defined. I worked around this with a patch like this:

diff --git a/installer/roles/kubernetes/tasks/main.yml b/installer/roles/kubernetes/tasks/main.yml
index 8f03022..6f72b17 100644
--- a/installer/roles/kubernetes/tasks/main.yml
+++ b/installer/roles/kubernetes/tasks/main.yml
@@ -118,6 +118,7 @@
POD=$({{ kubectl_or_oc }} -n {{ kubernetes_namespace }} \
get pods -l=name=postgresql --field-selector status.phase=Running -o jsonpath="{.items[0].metadata.name}")
{{ kubectl_or_oc }} exec -ti $POD -n {{ kubernetes_namespace }} -- bash -c "psql -U {{ pg_username }} -tAc 'select version()'"

• when: "pg_hostname is not defined or pg_hostname == ''"
  register: pg_version

• name: Upgrade Postgres if necessary
  @@ -164,7 +165,9 @@

  • name: Wait for Postgres to redeploy
    pause:
    seconds: "{{ postgress_activate_wait }}"
    
    
    
    • when: "pg_version is success and '9.6' in pg_version.stdout"
    
    
    • when:
    
    
    • 
      
      
      • "pg_hostname is not defined or pg_hostname == ''"
      
      
      
    
    
    • 
      
      
      • "pg_version is success and '9.6' in pg_version.stdout"
      
      
      
    
    

• name: Set image names if using custom registry
  block:

[shanemcd]

Will be fixed by https://github.com/ansible/awx/pull/5227

[shanemcd]

Fix will be included in AWX 9.0.1 which I'm pushing out now.

[dineshtng]

Removing # from the pg_hostname in Inventory file works for me.

Inventory file shall have following:

pg_hostname=postgresql