Awx: AWX is Upgrading..... Issue

[codesupremo]

ISSUE TYPE

• AWX is Upgrading... (from Browser)

SUMMARY

AWX is Upgrading... and it is taking a very long time, for more than a day now and still not finished.

ENVIRONMENT

• AWX version: 10.0.0
• AWX install method: docker
• Ansible version: 2.9.3
• Operating System: CentOS7
• Web Browser: Firefox in Ubuntu 18.4

STEPS TO REPRODUCE

When Accessing the AWX GUI — It’s stuck in AWX is Upgrading state. I’m not sure when it will be completely upgraded, It’s been a day now.

EXPECTED RESULTS

AWX GUI Login Prompt

ACTUAL RESULTS

Stuck in “AWX is Upgrading”

Unable to Login to AWX GUI because it is stuck in AWX is Upgrading state

ADDITIONAL INFORMATION

[chrismeyersfsu]

@codesupremo can you supply the docker logs i.e. docker logs -f awx_task docker logs -f awx_web

[ghost]

I can confirm that error as well in a docker-compose setup and external postgresql instance.
In my case, it seems like the database was not migrated, as AWX was complaining in the logs about some tables missing.

I solved by manually perform the migration with:

$ awx-manage migrate --noinput

from the awx-web container.

[way31415926]

@ldepandis @chrismeyersfsu
I encountered the same problem, and execute the command 'awx-manage migrate --noinput'.
Then,I could not login in the system.And there is no data in the database.

[ghost]

@way31415926 yes, that's right, because this is not a part of the database migration process.

In the awx-web container run
awx-manage createsuperuser --username admin

Insert the email and password and you're good to go.

Also, if you want the Demo Inventory or the other demo resources, just run
awx-manage create_preload_data

[way31415926]

@ldepandis You are right , thanks a lot.

[jibbajabber]

I can reliable reproduce this symptom for docker compose install method for all releases I've tried (11.0.0, 12.0.0, 13.0.0 and 14.0.0)
What is odd, is that I was happy deploying 11.0.0 for the past week without issues, however now despite restoring my environment to a clean fresh CentOS 7 install with a snapshot, the installer hangs waiting for migrations that never happen.

This is resolved for me by running the migrations manually inside awx_web container as mentioned but I don't know why this is now broken (for me) for so many releases.

My environment:

• ansible 2.9.0
• python 3.6.5

My pyenv python packages:

ansible==2.9.0
ansible-tower-cli==3.3.9
certifi==2020.6.20
cffi==1.14.1
chardet==3.0.4
click==6.7
colorama==0.4.3
cryptography==3.0
idna==2.10
Jinja2==2.11.2
MarkupSafe==1.1.1
pycparser==2.20
PyYAML==5.3.1
requests==2.24.0
six==1.15.0
urllib3==1.25.10

awx_web container 13.0.0 output:

020-08-13 10:59:37,099 ERROR    awx.main.wsbroadcast AWX is currently installing/upgrading.  Trying again in 5s...
2020-08-13 10:59:43,566 INFO exited: wsbroadcast (exit status 0; expected)
2020-08-13 10:59:43,566 INFO exited: wsbroadcast (exit status 0; expected)
2020-08-13 10:59:44,573 INFO spawned: 'wsbroadcast' with pid 507
2020-08-13 10:59:44,573 INFO spawned: 'wsbroadcast' with pid 507
2020-08-13 10:59:45,584 INFO success: wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-08-13 10:59:45,584 INFO success: wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
[pid: 78|app: 0|req: 12/189] x.x.x.x () {54 vars in 2608 bytes} [Thu Aug 13 10:59:45 2020] GET / => generated 0 bytes in 27 msecs (HTTP/1.1 302) 3 headers in 119 bytes (1 switches on core 0)
x.x.x.x - - [13/Aug/2020:10:59:46 +0000] "GET / HTTP/1.1" 302 5 "https://dv-awx001.oursite.com/migrations_notran/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36" "-"
[pid: 78|app: 0|req: 13/190] x.x.x.x () {54 vars in 2707 bytes} [Thu Aug 13 10:59:46 2020] GET /migrations_notran/ => generated 1589 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 171 bytes (1 switches on core 0)
x.x.x.x - - [13/Aug/2020:10:59:46 +0000] "GET /migrations_notran/ HTTP/1.1" 200 1589 "https://dv-awx001.oursite.com/migrations_notran/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36" "-"
x.x.x.x - - [13/Aug/2020:10:59:46 +0000] "POST /csp-violation/ HTTP/1.1" 302 5 "https://dv-awx001.oursite.com/migrations_notran/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36" "-"
[pid: 79|app: 0|req: 100/191] x.x.x.x () {58 vars in 2622 bytes} [Thu Aug 13 10:59:46 2020] POST /csp-violation/ => generated 0 bytes in 31 msecs (HTTP/1.1 302) 3 headers in 119 bytes (1 switches on core 0)
[pid: 81|app: 0|req: 52/192] x.x.x.x () {54 vars in 2492 bytes} [Thu Aug 13 10:59:46 2020] GET /api/ => generated 0 bytes in 28 msecs (HTTP/1.1 302) 3 headers in 119 bytes (1 switches on core 0)
x.x.x.x - - [13/Aug/2020:10:59:46 +0000] "GET /api/ HTTP/1.1" 302 5 "https://dv-awx001.oursite.com/migrations_notran/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36" "-"
[pid: 79|app: 0|req: 101/193] x.x.x.x () {54 vars in 2583 bytes} [Thu Aug 13 10:59:46 2020] GET /migrations_notran/ => generated 1589 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 171 bytes (1 switches on core 0)
x.x.x.x - - [13/Aug/2020:10:59:46 +0000] "GET /migrations_notran/ HTTP/1.1" 200 1589 "https://dv-awx001.oursite.com/migrations_notran/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36" "-"
x.x.x.x - - [13/Aug/2020:10:59:46 +0000] "GET /api/ HTTP/1.1" 302 5 "https://dv-awx001.oursite.com/migrations_notran/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36" "-"
[pid: 81|app: 0|req: 53/194] x.x.x.x () {56 vars in 2544 bytes} [Thu Aug 13 10:59:46 2020] GET /api/ => generated 0 bytes in 32 msecs (HTTP/1.1 302) 3 headers in 119 bytes (1 switches on core 0)
[pid: 79|app: 0|req: 102/195] x.x.x.x () {56 vars in 2635 bytes} [Thu Aug 13 10:59:46 2020] GET /migrations_notran/ => generated 1589 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 171 bytes (1 switches on core 0)
x.x.x.x - - [13/Aug/2020:10:59:46 +0000] "GET /migrations_notran/ HTTP/1.1" 200 1589 "https://dv-awx001.oursite.com/migrations_notran/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36" "-"
2020-08-13 10:59:47,141 ERROR    awx.main.wsbroadcast AWX is currently installing/upgrading.  Trying again in 5s...

I forget which release but I've also seen this awx_web output (where main_instance doesn't appear to be created):

2020-08-05 10:16:25,097 INFO exited: dispatcher (exit status 1; not expected)
2020-08-05 10:16:25,097 INFO exited: dispatcher (exit status 1; not expected)
2020-08-05 10:16:26,103 INFO spawned: 'dispatcher' with pid 2484
2020-08-05 10:16:26,103 INFO spawned: 'dispatcher' with pid 2484
2020-08-05 10:16:27,106 INFO success: dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-08-05 10:16:27,106 INFO success: dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-08-05 10:16:30,257 WARNING  awx.main.dispatch.periodic periodic beat started
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/awx-manage", line 8, in <module>
    sys.exit(manage())
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/__init__.py", line 154, in manage
    execute_from_command_line(sys.argv)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/management/commands/run_dispatcher.py", line 55, in handle
    reaper.reap()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
    (changed, me) = Instance.objects.get_or_register()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/managers.py", line 144, in get_or_register
    return (False, self.me())
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/managers.py", line 100, in me
    if node.exists():
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py", line 766, in exists
    return self.query.has_results(using=self.db)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
    return compiler.has_results()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1110, in has_results
    return bool(self.execute_sql(SINGLE))
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1140, in execute_sql
    cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...

Since I'm close to completion for an ansible role for creating our entire AWX instance (credentials, projects, inventories, job templates, users etc etc), I will add this workaround when waiting for the installer to complete, if relevant search terms found within the awx_web container log, or within a reasonable time reached for the REST API to be available.
I need our deployment to be completely automated, so a workaround is mandatory for us (I'd rather not _have_ to do this though 😃 if we could get this addressed)

Happy to share more info, capture more logs etc if it helps 👍

My ansible for this workaround (two role task files):

wait_for_install_to_complete.yml (runs after installer finishes and waits for REST to be ready):

- name: Wait for AWX to be ready to work
  uri:
    url: "{{ item.url }}"
    user: "{{ item.user | default(omit) }}"
    password: "{{ item.pass | default(omit) }}"
    return_content: yes
    validate_certs: no
    force_basic_auth: "{{ item.basic_auth | default(omit) }}"
  until: awx_ready.status == 200
  retries: 10
  delay: 30
  register: awx_ready
  changed_when: false
  with_items: "{{ awx_ready_urls }}"
  ignore_errors: yes
...

awx_ready_urls used:

awx_ready_urls:
  - { url: "https://{{ inventory_hostname }}/api/v2" }

rescue.yml (tries the migration workaround, waits for all migrations to be applied then tries one more time to wait for REST):

- name: Attempt rescue of bad install
  block:
    - name: Kick migration in case of failure
      command: docker exec -it awx_web bash -c "awx-manage migrate"
      register: migration_status
      retries: 3
      delay: 5
      until: "'No migrations to apply' in migration_status.stdout"
      changed_when: no

    - name: Restart all containers
      docker_compose:
        project_src: "{{ awx_docker_compose_dir }}"
        restarted: yes

    - name: Wait for AWX to be ready
      include: wait_for_install_to_complete.yml
  when: install_status !=200
...