Awx: EXCEPTION DURING MIGRATION - FRESH INSTALL

[chiquiff]

ISSUE TYPE

• Bug Report

COMPONENT NAME

• Installer
• Migrations

SUMMARY

I'm doing a fresh install on top of Kubernetes. I'm using an RDS (aws) Postgres DB (Engine version 12.3). I'm stuck in a loop with the following stacktrace:

[awx-598d769698-b7ts7 awx-task] return self.cursor.execute(sql, params) [awx-598d769698-b7ts7 awx-task] django.db.utils.ProgrammingError: relation "main_instance" does not exist [awx-598d769698-b7ts7 awx-task] LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE ("main_instance... [awx-598d769698-b7ts7 awx-task] ^ [awx-598d769698-b7ts7 awx-task] [awx-598d769698-b7ts7 awx-task] 2020-10-13 20:39:14,498 INFO exited: dispatcher (exit status 1; not expected) [awx-598d769698-b7ts7 awx-task] 2020-10-13 20:39:14,498 INFO exited: dispatcher (exit status 1; not expected)

ENVIRONMENT

• AWX version: 15.0.0
• AWX install method: kubernetes (following the official installing procedure using ansible-playbook)
• Ansible version: 2.9.7
• Operating System: Linux
• Web Browser: Chrome

STEPS TO REPRODUCE

I've followed the steps in this instructions: https://github.com/ansible/awx/blob/15.0.0/INSTALL.md#kubernetes

• Updated the inventory with my variables (pg_host for instance)
• Run ansible-playbook -i inventory install.yml -vvvv

EXPECTED RESULTS

To work.

ACTUAL RESULTS

Stuck in "Ansible is upgrading" with the stacktrace I've attached above.

ADDITIONAL INFORMATION

[chiquiff]

Here's the big trace:

Traceback (most recent call last):
[awx-598d769698-b7ts7 awx-task] File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
[awx-598d769698-b7ts7 awx-task] return self.cursor.execute(sql, params)
[awx-598d769698-b7ts7 awx-task] psycopg2.errors.UndefinedTable: relation "main_instance" does not exist
[awx-598d769698-b7ts7 awx-task] LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE ("main_instance...
[awx-598d769698-b7ts7 awx-task] ^
[awx-598d769698-b7ts7 awx-task]
[awx-598d769698-b7ts7 awx-task]
[awx-598d769698-b7ts7 awx-task] The above exception was the direct cause of the following exception:
[awx-598d769698-b7ts7 awx-task]
[awx-598d769698-b7ts7 awx-task] Traceback (most recent call last):
[awx-598d769698-b7ts7 awx-task] File "/usr/bin/awx-manage", line 8, in
[awx-598d769698-b7ts7 awx-task] sys.exit(manage())
[awx-598d769698-b7ts7 awx-task] File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/__init__.py", line 154, in manage
[awx-598d769698-b7ts7 awx-task] execute_from_command_line(sys.argv)
[awx-598d769698-b7ts7 awx-task] File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
[awx-598d769698-b7ts7 awx-task] utility.execute()
[awx-598d769698-b7ts7 awx-task] File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
[awx-598d769698-b7ts7 awx-task] self.fetch_command(subcommand).run_from_argv(self.argv)
[awx-598d769698-b7ts7 awx-task] File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv
[awx-598d769698-b7ts7 awx-task] self.execute(args, *cmd_options)
[awx-598d769698-b7ts7 awx-task] File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py", line 364, in execute
[awx-598d769698-b7ts7 awx-task] output = self.handle(args, *options)
[awx-598d769698-b7ts7 awx-task] File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/management/commands/run_dispatcher.py", line 55, in handle
[awx-598d769698-b7ts7 awx-task] reaper.reap()
[awx-598d769698-b7ts7 awx-task] File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/dispatch/reaper.py", line 38, in reap
[awx-598d769698-b7ts7 awx-task] (changed, me) = Instance.objects.get_or_register()
[awx-598d769698-b7ts7 awx-task] File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/managers.py", line 146, in get_or_register
[awx-598d769698-b7ts7 awx-task] registered = self.register(ip_address=pod_ip)
[awx-598d769698-b7ts7 awx-task] File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/managers.py", line 120, in register
[awx-598d769698-b7ts7 awx-task] if inst_conflicting_ip.exists():
[awx-598d769698-b7ts7 awx-task] File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py", line 766, in exists
[awx-598d769698-b7ts7 awx-task] return self.query.has_results(using=self.db)
[awx-598d769698-b7ts7 awx-task] File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/query.py", line 522, in has_results
[awx-598d769698-b7ts7 awx-task] return compiler.has_results()
[awx-598d769698-b7ts7 awx-task] File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1112, in has_results
[awx-598d769698-b7ts7 awx-task] return bool(self.execute_sql(SINGLE))
[awx-598d769698-b7ts7 awx-task] File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1142, in execute_sql
[awx-598d769698-b7ts7 awx-task] cursor.execute(sql, params)
[awx-598d769698-b7ts7 awx-task] File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
[awx-598d769698-b7ts7 awx-task] return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
[awx-598d769698-b7ts7 awx-task] File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
[awx-598d769698-b7ts7 awx-task] return executor(sql, params, many, context)
[awx-598d769698-b7ts7 awx-task] File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
[awx-598d769698-b7ts7 awx-task] return self.cursor.execute(sql, params)
[awx-598d769698-b7ts7 awx-task] File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
[awx-598d769698-b7ts7 awx-task] raise dj_exc_value.with_traceback(traceback) from exc_value
[awx-598d769698-b7ts7 awx-task] File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
[awx-598d769698-b7ts7 awx-task] return self.cursor.execute(sql, params)
[awx-598d769698-b7ts7 awx-task] django.db.utils.ProgrammingError: relation "main_instance" does not exist
[awx-598d769698-b7ts7 awx-task] LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE ("main_instance...
[awx-598d769698-b7ts7 awx-task] ^
[awx-598d769698-b7ts7 awx-task]
[awx-598d769698-b7ts7 awx-task] 2020-10-13 20:39:14,498 INFO exited: dispatcher (exit status 1; not expected)
[awx-598d769698-b7ts7 awx-task] 2020-10-13 20:39:14,498 INFO exited: dispatcher (exit status 1; not expected)

[chiquiff]

I'm looking at the DB and there's nothing in there. No tables at all. Seems that no migration is running.

[chiquiff]

Managed to run the migrations manually with:

awx-manage migrate --noinput

Still not sure why this didn't happen automatically...

[geerlingguy]

@chiquiff - I'm finding the same thing with 15.x on Ubuntu and on CentOS installs (not in cluster)—if I run docker exec awx_task awx-manage migrate --noinput manually after install, I can finally get the migration to kick off.

Alternatively, if I wait for awx_task to get stuck in it's startup loop (~10-30s), then restart the Docker daemon, that seems to also finally help AWX trigger its initial migration on its own.

[vpodzime]

then restart the Docker daemon, that seems to also finally help AWX trigger its initial migration on its own.

No need to restart docker itself, docker restart awx_task should be enough. Still, it shouldn't be needed at all, of course.

[chiquiff]

@chiquiff - I'm finding the same thing with 15.x on Ubuntu and on CentOS installs (not in cluster)—if I run docker exec awx_task awx-manage migrate --noinput manually after install, I can finally get the migration to kick off.

Alternatively, if I wait for awx_task to get stuck in it's startup loop (~10-30s), then restart the Docker daemon, that seems to also finally help AWX trigger its initial migration on its own.

Yeah, I´ve managed to make it work that way. The migrations work at the very end, but they should run automatically when the fresh install starts. Just though it was worth mention it. Thanks for the input @geerlingguy

[vpodzime]

I've ended up with this to make sure things are up and running:

docker exec awx_task awx-manage migrate --noinput
docker restart awx_task
docker restart awx_web
retries=6
while [ $retries -gt 0 ]; do
  if awx login --conf.host http://localhost --conf.username admin --conf.password password; then
    break
  else
    echo "Retrying..."
    retrires=$((retries - 1))
    sleep 10s
  fi
done
if [ $retries -eq 0 ]; then
  echo "Failed to log in to AWX"
  exit 1
fi

And usually there's only one retry needed.

[rooftopcellist]

This issue is fixed by the follow PR's:

• https://github.com/ansible/awx/pull/9078
• https://github.com/ansible/awx/pull/9113

The will be available in the next AWX release, but are currently in devel if you want to try it out now.