Today I tried to upgrade AWX from 1.0.1.something to latest. The Ansible Installer script reported no errors, but from looking at the output of docker logs -f awx_task it seems there is some issue with the database schema.
The following message is repeated over and over again:
ERROR/ForkPoolWorker-274] Task awx.main.tasks.awx_periodic_scheduler[8f25cb7e-aa0c-4dd5-8a87-cd3be49aed58] raised unexpected: ProgrammingError('column main_schedule.inventory_id does not exist\nLINE 1: ...hedule"."modified_by_id", "main_schedule"."name", "main_sche...\n ^\n',)
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/trace.py", line 374, in trace_task
R = retval = fun(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/trace.py", line 629, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/awx/main/tasks.py", line 300, in awx_periodic_scheduler
for schedule in old_schedules:
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py", line 250, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py", line 1118, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py", line 53, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 894, in execute_sql
raise original_exception
ProgrammingError: column main_schedule.inventory_id does not exist
LINE 1: ...hedule"."modified_by_id", "main_schedule"."name", "main_sche...
The main_schedule table is indeed missing a column named inventory_id:
created
modified
description
name
enabled
dtstart
dtend
rrule
next_run
extra_data
created_by_id
modified_by_id
unified_job_template_id
The following schema updates have been recorded in the django_migrations table:
....
88 social_django 0001_initial 2017-11-19 14:59:31
89 main 0009_v322_add_setting_field_for_activity_stream 2017-12-15 11:24:23
90 main 0010_v322_add_ovirt4_tower_inventory 2017-12-15 11:24:24
How can I proceed from here? Is starting with a fresh install the only thing I can do?
The Ansible Installer should either fail with an error or produce a working environment.
Ansible Installer tells me everything is OK but awx_task container is stuck in some infinite loop, apparently because some db schema updates have not been applied?
Can you login to the awx_task and execute awx-manage showmigrations and git describe on the awx clone?
Best shot, is to try run awx-manage makemigrations and then awx-manage migrate.
Refer to this PR https://github.com/ansible/awx/pull/535 on how to get it done for upgrade.
May I suggest a Changelog or other file documenting any special procedures before the upgrades? I've been using AWX since 1.0.0 and is not the first time that there are issues with upgrades. Sometimes I find the answer on a closed issue or PR or in the forum. A Changelog or a specific document would be very convenient.
I tried upgrading from 1.0.2.0 to 1.0.2.293. AWX was again stuck on the upgrading screen. So I checked the migration states with the commands suggested by @aizuddin85
An exception happens during 0011_v322_encrypt_survey_passwords.
django.db.utils.ProgrammingError: column main_jobtemplate.credential_id does not exist
Here is the output of the commands:
[root@awx awx]# awx-manage showmigrations
auth
[X] 0001_initial
[X] 0002_alter_permission_name_max_length
[X] 0003_alter_user_email_max_length
[X] 0004_alter_user_username_opts
[X] 0005_alter_user_last_login_null
[X] 0006_require_contenttypes_0002
[X] 0007_alter_validators_add_error_messages
[X] 0008_alter_user_username_max_length
conf
[X] 0001_initial
[X] 0002_v310_copy_tower_settings
[X] 0003_v310_JSONField_changes
[X] 0004_v320_reencrypt
contenttypes
[X] 0001_initial
[X] 0002_remove_content_type_name
django_celery_results
[X] 0001_initial
main
[X] 0001_initial
[X] 0002_squashed_v300_release (18 squashed migrations)
[X] 0003_squashed_v300_v303_updates (9 squashed migrations)
[X] 0004_squashed_v310_release (6 squashed migrations)
[X] 0005_squashed_v310_v313_updates (3 squashed migrations)
[X] 0006_v320_release (2 squashed migrations)
[X] 0007_v320_data_migrations
[X] 0008_v320_drop_v1_credential_fields
[X] 0009_v322_add_setting_field_for_activity_stream
[X] 0010_v322_add_ovirt4_tower_inventory
[ ] 0011_v322_encrypt_survey_passwords
[ ] 0012_v322_update_cred_types
[ ] 0013_v330_multi_credential
[ ] 0014_v330_saved_launchtime_configs
[ ] 0015_v330_blank_start_args
[ ] 0016_v330_non_blank_workflow
[ ] 0017_v330_move_deprecated_stdout
[ ] 0018_auto_20171218_1241
sessions
[X] 0001_initial
sites
[X] 0001_initial
[X] 0002_alter_domain_unique
social_django
[X] 0001_initial (2 squashed migrations)
[X] 0002_add_related_name (2 squashed migrations)
[X] 0003_alter_email_max_length (2 squashed migrations)
[X] 0004_auto_20160423_0400 (2 squashed migrations)
[X] 0005_auto_20160727_2333 (1 squashed migrations)
[X] 0006_partial
[X] 0007_code_timestamp
[X] 0008_partial_timestamp
sso
[X] 0001_initial
[X] 0002_expand_provider_options
taggit
[X] 0001_initial
[X] 0002_auto_20150616_2121
[root@awx awx]# awx-manage showmigrations
auth
[X] 0001_initial
[X] 0002_alter_permission_name_max_length
[X] 0003_alter_user_email_max_length
[X] 0004_alter_user_username_opts
[X] 0005_alter_user_last_login_null
[X] 0006_require_contenttypes_0002
[X] 0007_alter_validators_add_error_messages
[X] 0008_alter_user_username_max_length
conf
[X] 0001_initial
[X] 0002_v310_copy_tower_settings
[X] 0003_v310_JSONField_changes
[X] 0004_v320_reencrypt
contenttypes
[X] 0001_initial
[X] 0002_remove_content_type_name
django_celery_results
[X] 0001_initial
main
[X] 0001_initial
[X] 0002_squashed_v300_release (18 squashed migrations)
[X] 0003_squashed_v300_v303_updates (9 squashed migrations)
[X] 0004_squashed_v310_release (6 squashed migrations)
[X] 0005_squashed_v310_v313_updates (3 squashed migrations)
[X] 0006_v320_release (2 squashed migrations)
[X] 0007_v320_data_migrations
[X] 0008_v320_drop_v1_credential_fields
[X] 0009_v322_add_setting_field_for_activity_stream
[X] 0010_v322_add_ovirt4_tower_inventory
[ ] 0011_v322_encrypt_survey_passwords
[ ] 0012_v322_update_cred_types
[ ] 0013_v330_multi_credential
[ ] 0014_v330_saved_launchtime_configs
[ ] 0015_v330_blank_start_args
[ ] 0016_v330_non_blank_workflow
[ ] 0017_v330_move_deprecated_stdout
[ ] 0018_auto_20171218_1241
sessions
[X] 0001_initial
sites
[X] 0001_initial
[X] 0002_alter_domain_unique
social_django
[X] 0001_initial (2 squashed migrations)
[X] 0002_add_related_name (2 squashed migrations)
[X] 0003_alter_email_max_length (2 squashed migrations)
[X] 0004_auto_20160423_0400 (2 squashed migrations)
[X] 0005_auto_20160727_2333 (1 squashed migrations)
[X] 0006_partial
[X] 0007_code_timestamp
[X] 0008_partial_timestamp
sso
[X] 0001_initial
[X] 0002_expand_provider_options
taggit
[X] 0001_initial
[X] 0002_auto_20150616_2121
[root@awx awx]# awx-manage migrate
Operations to perform:
Apply all migrations: auth, conf, contenttypes, django_celery_results, main, sessions, sites, social_django, sso, taggit
Running migrations:
Applying main.0011_v322_encrypt_survey_passwords...Traceback (most recent call last):
File "/usr/bin/awx-manage", line 9, in <module>
load_entry_point('awx==1.0.2.293', 'console_scripts', 'awx-manage')()
File "/usr/lib/python2.7/site-packages/awx/__init__.py", line 109, in manage
execute_from_command_line(sys.argv)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
utility.execute()
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 204, in handle
fake_initial=fake_initial,
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/migrations/executor.py", line 115, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
state = migration.apply(state, schema_editor)
File "/usr/lib/python2.7/site-packages/awx/main/migrations/__init__.py", line 12, in apply
return Migration.apply(self, project_state, schema_editor, collect_sql)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/migrations/migration.py", line 129, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
self.code(from_state.apps, schema_editor)
File "/usr/lib/python2.7/site-packages/awx/main/migrations/_reencrypt.py", line 102, in encrypt_survey_passwords
apps.get_model('main', 'WorkflowJobTemplate'),
File "/usr/lib/python2.7/site-packages/awx/main/migrations/_reencrypt.py", line 109, in _encrypt_survey_passwords
for jt in _type.objects.exclude(survey_spec={}):
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py", line 250, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py", line 1118, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py", line 53, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 894, in execute_sql
raise original_exception
django.db.utils.ProgrammingError: column main_jobtemplate.credential_id does not exist
LINE 1: ...urvey_enabled", "main_jobtemplate"."survey_spec", "main_jobt...
@ortigo, unfortunately you only able to go from 1.0.1.x to 1.0.2. Direct migration from 1.0.1 to 1.0.2.293 is not possible without fixing database. Refer to #535.
@aizuddin85. The problem is when I'm upgrading from 1.0.2.0 to 1.0.2.293. The upgrade from 1.0.1x to 1.0.2.0 worked fine.
@jakemcdermott I can confirm @ortigo observation that 1.0.2.0 fresh installation then upgrade to 1.0.2.293 thrown below error as well:
[root@amsdc1-n-s00289 awx]# docker logs -f awx_task
Using /etc/ansible/ansible.cfg as config file
[DEPRECATION WARNING]: The sudo command line option has been deprecated in
favor of the "become" command line arguments. This feature will be removed in
version 2.6. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
127.0.0.1 | SUCCESS => {
"changed": false,
"db": "awx"
}
Operations to perform:
Apply all migrations: auth, conf, contenttypes, django_celery_results, main, sessions, sites, social_django, sso, taggit
Running migrations:
Applying main.0009_v322_add_setting_field_for_activity_stream... OK
2017-12-19 05:38:39,345 DEBUG awx.main.models.credential adding Red Hat Virtualization credential type
2017-12-19 05:38:39,357 DEBUG awx.main.models.credential adding Ansible Tower credential type
Applying main.0010_v322_add_ovirt4_tower_inventory... OK
Traceback (most recent call last):
File "/usr/bin/awx-manage", line 9, in <module>
load_entry_point('awx==1.0.2.293', 'console_scripts', 'awx-manage')()
File "/usr/lib/python2.7/site-packages/awx/__init__.py", line 109, in manage
execute_from_command_line(sys.argv)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
utility.execute()
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 204, in handle
fake_initial=fake_initial,
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/migrations/executor.py", line 115, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
state = migration.apply(state, schema_editor)
File "/usr/lib/python2.7/site-packages/awx/main/migrations/__init__.py", line 12, in apply
return Migration.apply(self, project_state, schema_editor, collect_sql)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/migrations/migration.py", line 129, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
self.code(from_state.apps, schema_editor)
File "/usr/lib/python2.7/site-packages/awx/main/migrations/_reencrypt.py", line 102, in encrypt_survey_passwords
apps.get_model('main', 'WorkflowJobTemplate'),
File "/usr/lib/python2.7/site-packages/awx/main/migrations/_reencrypt.py", line 109, in _encrypt_survey_passwords
for jt in _type.objects.exclude(survey_spec={}):
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py", line 250, in __iter__
self._fetch_all()
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py", line 1118, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py", line 53, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 894, in execute_sql
raise original_exception
django.db.utils.ProgrammingError: column main_jobtemplate.credential_id does not exist
LINE 1: ...urvey_enabled", "main_jobtemplate"."survey_spec", "main_jobt...
^
Traceback (most recent call last):
File "/usr/bin/awx-manage", line 9, in <module>
load_entry_point('awx==1.0.2.293', 'console_scripts', 'awx-manage')()
File "/usr/lib/python2.7/site-packages/awx/__init__.py", line 109, in manage
execute_from_command_line(sys.argv)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
utility.execute()
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/commands/shell.py", line 101, in handle
exec(sys.stdin.read())
File "<string>", line 1, in <module>
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/contrib/auth/models.py", line 170, in create_superuser
return self._create_user(username, email, password, **extra_fields)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/contrib/auth/models.py", line 153, in _create_user
user.save(using=self._db)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/contrib/auth/base_user.py", line 80, in save
super(AbstractBaseUser, self).save(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/base.py", line 808, in save
force_update=force_update, update_fields=update_fields)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/base.py", line 838, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/base.py", line 924, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/base.py", line 963, in _do_insert
using=using, raw=raw)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py", line 1076, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1107, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "auth_user_username_key"
DETAIL: Key (username)=(admin) already exists.
Applying main.0011_v322_encrypt_survey_passwords...Instance already registered awx
Instance Group already registered tower
Instance already registered awx
Showmigrations output:
[root@awx awx]# awx-manage showmigrations
auth
[X] 0001_initial
[X] 0002_alter_permission_name_max_length
[X] 0003_alter_user_email_max_length
[X] 0004_alter_user_username_opts
[X] 0005_alter_user_last_login_null
[X] 0006_require_contenttypes_0002
[X] 0007_alter_validators_add_error_messages
[X] 0008_alter_user_username_max_length
conf
[X] 0001_initial
[X] 0002_v310_copy_tower_settings
[X] 0003_v310_JSONField_changes
[X] 0004_v320_reencrypt
contenttypes
[X] 0001_initial
[X] 0002_remove_content_type_name
django_celery_results
[X] 0001_initial
main
[X] 0001_initial
[X] 0002_squashed_v300_release (18 squashed migrations)
[X] 0003_squashed_v300_v303_updates (9 squashed migrations)
[X] 0004_squashed_v310_release (6 squashed migrations)
[X] 0005_squashed_v310_v313_updates (3 squashed migrations)
[X] 0006_v320_release (2 squashed migrations)
[X] 0007_v320_data_migrations
[X] 0008_v320_drop_v1_credential_fields
[X] 0009_v322_add_setting_field_for_activity_stream
[X] 0010_v322_add_ovirt4_tower_inventory
[ ] 0011_v322_encrypt_survey_passwords
[ ] 0012_v322_update_cred_types
[ ] 0013_v330_multi_credential
[ ] 0014_v330_saved_launchtime_configs
[ ] 0015_v330_blank_start_args
[ ] 0016_v330_non_blank_workflow
[ ] 0017_v330_move_deprecated_stdout
sessions
[X] 0001_initial
sites
[X] 0001_initial
[X] 0002_alter_domain_unique
social_django
[X] 0001_initial (2 squashed migrations)
[X] 0002_add_related_name (2 squashed migrations)
[X] 0003_alter_email_max_length (2 squashed migrations)
[X] 0004_auto_20160423_0400 (2 squashed migrations)
[X] 0005_auto_20160727_2333 (1 squashed migrations)
[X] 0006_partial
[X] 0007_code_timestamp
[X] 0008_partial_timestamp
sso
[X] 0001_initial
[X] 0002_expand_provider_options
taggit
[X] 0001_initial
[X] 0002_auto_20150616_2121
Seems like 1.0.2 to 1.0.2.293 is broken due to django model issue?
Here is the summary of the tested scenario:
@aizuddin85 I'm seeing the same thing in my testing with the same failed migration.
1.0.1.323 to 1.0.2.0 - Success
1.0.2.0 to 1.0.2.285 - Failure with django.db.utils.ProgrammingError: column main_jobtemplate.credential_id does not exist
Edit: Referencing #841:
https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/awx-project/PQLxKl5Rj9s/UGy-3VaCCQAJ
Still the same issue from 1.0.2.0 to 1.0.2.301.
Still the same issue from 1.0.2.0 to 1.0.2.303. I opened the issue #884 for reference.
Ok, from the referenced issues I learned that there is no update path from 1.0.1.x to 1.0.2.y with y>0. I will reinstall from scratch.
@terra-nova i tried to update from 1.0.2.0 to 1.0.3.0, having the same symtoms. So there is no update path for the .0 versions either ?!?!
Is it true that there are still no upgrade paths from 1.0.2.0 to 1.0.3.0?
Is it true that there are still no upgrade paths from 1.0.2.0 to 1.0.3.0?
It seems that's not the case: https://github.com/ansible/awx/issues/1707#issuecomment-377351025
In general we don't support major version upgrades, you'll likely need to redeploy.
Most helpful comment
May I suggest a Changelog or other file documenting any special procedures before the upgrades? I've been using AWX since 1.0.0 and is not the first time that there are issues with upgrades. Sometimes I find the answer on a closed issue or PR or in the forum. A Changelog or a specific document would be very convenient.