Awx: upgrade 1.0.5 -> 1.0.6 breaks during db upgrade steps.

[greenscar]

ISSUE TYPE

• Bug Report

COMPONENT NAME

• Installer

SUMMARY

Upgrading 1.0.5 to 1.0.6 DB upgrade locks when auth config change started but not completed.

Currently on 1.0.5. We started setting up LDAP but never finished. However, local auth has continued working. We then upgrade to 1.0.6. Now app is hung in endless loop where web ui says "AWX is upgrading"

ENVIRONMENT

AWX version: 1.0.6
AWX install method: docker on linux
Ansible version: 2.5.0
Operating System: ubuntu
Web Browser: any

STEPS TO REPRODUCE

Have local users.
Partially configure LDAP auth. (input settings but auth still failing due to grouping)
Continue using app with local auth.
Upgrade 105 -> 106

EXPECTED RESULTS

it works

ACTUAL RESULTS

xxx@dddd-01:~$ docker logs awx_task_1
Using /etc/ansible/ansible.cfg as config file
...
Traceback (most recent call last):
  File "/usr/bin/awx-manage", line 9, in <module>
    load_entry_point('awx==1.0.6.0', 'console_scripts', 'awx-manage')()
  File "/usr/lib/python2.7/site-packages/awx/__init__.py", line 109, in manage
    execute_from_command_line(sys.argv)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
Operations to perform:
  Apply all migrations: auth, conf, contenttypes, djcelery, main, network_ui, oauth2_provider, sessions, sites, social_django, sso, taggit
    utility.execute()
...
ValueError: The field oauth2_provider.AccessToken.application was declared with a lazy reference to 'main.oauth2application', but app 'main' doesn't provide model 'oauth2application'.
The field oauth2_provider.Grant.application was declared with a lazy reference to 'main.oauth2application', but app 'main' doesn't provide model 'oauth2application'.
The field oauth2_provider.RefreshToken.access_token was declared with a lazy reference to 'main.oauth2accesstoken', but app 'main' doesn't provide model 'oauth2accesstoken'.
The field oauth2_provider.RefreshToken.application was declared with a lazy reference to 'main.oauth2application', but app 'main' doesn't provide model 'oauth2application'.

...
django.db.utils.IntegrityError: duplicate key value violates unique constraint "auth_user_username_key"
DETAIL:  Key (username)=(admin) already exists.

Instance already registered awx
Instance Group already registered tower
...
2018-04-27 17:58:18,281 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
...

ddddd@xxxxx-01:~$ docker logs -f awx_web_1
....
[pid: 99|app: 0|req: 190/6905] 10.252.185.203 () {44 vars in 3511 bytes} [Fri Apr 27 21:34:50 2018] GET /api/ => generated 0 bytes in 24 msecs (HTTP/1.1 302) 5 headers in 160 bytes (1 switches on core 0)
[pid: 108|app: 0|req: 201/6906] 10.252.185.203 () {42 vars in 3580 bytes} [Fri Apr 27 21:34:50 2018] GET /migrations_notran/ => generated 1406 bytes in 9 msecs (HTTP/1.1 200) 4 headers in 129 bytes (1 switches on core 0)
[pid: 28|app: 0|req: 848/6907] 10.252.185.203 () {42 vars in 3580 bytes} [Fri Apr 27 21:34:51 2018] GET /migrations_notran/ => generated 1406 bytes in 13 msecs (HTTP/1.1 200) 4 headers in 129 bytes (1 switches on core 0)
[pid: 108|app: 0|req: 202/6908] 10.252.185.203 () {44 vars in 3546 bytes} [Fri Apr 27 21:34:51 2018] GET / => generated 0 bytes in 34 msecs (HTTP/1.1 302) 5 headers in 160 bytes (1 switches on core 0)
[pid: 90|app: 0|req: 687/6909] 10.252.185.203 () {42 vars in 3489 bytes} [Fri Apr 27 21:34:51 2018] GET /api/ => generated 0 bytes in 32 msecs (HTTP/1.1 302) 5 headers in 160 bytes (1 switches on core 0)
[pid: 28|app: 0|req: 849/6910] 10.252.185.203 () {44 vars in 3645 bytes} [Fri Apr 27 21:34:51 2018] GET /migrations_notran/ => generated 1406 bytes in 10 msecs (HTTP/1.1 200) 4 headers in 129 bytes (1 switches on core 0)
[pid: 108|app: 0|req: 203/6911] 10.252.185.203 () {44 vars in 3511 bytes} [Fri Apr 27 21:34:52 2018] GET /api/ => generated 0 bytes in 24 msecs (HTTP/1.1 302) 5 headers in 160 bytes (1 switches on core 0)
[pid: 30|app: 0|req: 983/6912] 10.252.185.203 () {42 vars in 3580 bytes} [Fri Apr 27 21:34:52 2018] GET /migrations_notran/ => generated 1406 bytes in 9 msecs (HTTP/1.1 200) 4 headers in 129 bytes (1 switches on core 0)
[pid: 99|app: 0|req: 191/6913] 10.252.185.203 () {42 vars in 3580 bytes} [Fri Apr 27 21:35:00 2018] GET /migrations_notran/ => generated 1406 bytes in 12 msecs (HTTP/1.1 200) 4 headers in 129 bytes (1 switches on core 0)
RESULT 2
OKREADY

ADDITIONAL INFORMATION

[jakemcdermott]

Hello, @greenscar

Upgrades between AWX version 1.0.5 and 1.0.6 are not currently expected to work. However, we have recently added an import/export capability to tower-cli/awx-cli, which allows you to export your job templates and other objects (not including credential secrets) to a JSON file, which you can then re-import to a freshly installed 1.0.6.

If you have any questions please feel free reach out to our official awx mailing list or stop by our irc channel on freenode.

[innocode-devops]

@jakemcdermott sorry to ping in closed issue, but is there any documentation on how to upgrade AWX to 1.0.6 in docker compose setup?

1. Is "Upgrades between AWX version 1.0.5 and 1.0.6 are not currently expected to work" mean that might work in later minor version?
2. Is there any documentation how to do export/import in docker setup? As far as I understand, awx-cli is not included in image and does not export users/permissions etc.

[greenscar]

@jakemcdermott : Upgrades from 1.0.4 -> 1.0.5 were not supported either. Are we to expect this moving forward? Why would anyone use a product if they cannot upgrade?

[gregdek]

AWX is not a product; it's a project. The supported product is Ansible Tower.

Upgrades for AWX are best effort, and that best effort at present is this process: download awx-cli, export your AWX data with awx-cli, install new version, import via awx-cli. We'll be documenting this more clearly in the near future.

[greenscar]

Thanks @gregdek. The fact there is a path forward speaks volumes. Looking forward to the docs. Meanwhile, I will try it out; I learned a lot about the cli when I upgraded 1.0.4 -> 1.0.5.

[greenscar]

@innocode-devops : From what I have gathered thus far, direct upgrades will not be supported in the near future. That said, the cli should handle most of the migration.... I am interested to see if it will migrate auth config (ex: we use LDAP)

[PsychoSid]

I have been unable to setup LDAP from scratch in 1.0.6 the config never gets saved when setting up group mappings etc and then disappears. Will try to investigate further.

[itcrowdsource]

Does anyone know how the tower/awx-cli export and import procedure works? I can't find any proper information about it. I have installed tower-cli Python package to manage AWX (pip install ansible-tower-cli) but I'm unable to find the export import commands.
Am I using the wrong version?

[gregdek]

On Thu, May 3, 2018 at 9:49 AM, IT Crowdsource [e-mail]
wrote:

Does anyone know how the tower/awx-cli export and import procedure works?
I can't find any proper information about it. I have installed tower-cli
Python package to manage AWX (pip install ansible-tower-cli) but I'm unable
to find the export import commands.
Am I using the wrong version?

The new version with the new send/receive commands just landed; I don't
know that the docs are updated yet. We're working on that.

In the meantime, it's basically two commands, as I understand it:
"tower-cli receive all" which will export all possible data, and then
"tower-cli send resulting-file.json" to import the data into the new AWX
instance. Someone correct me if I'm wrong. :)

--g

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ansible/awx/issues/1817#issuecomment-386301848, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAPY3esHIgRAkcRN6VODDN0FK4yPlxvXks5tuwrjgaJpZM4TrC95
.

[cavamagie]

tower-cli receive --all > assets.json
Traceback (most recent call last):
File "/usr/local/bin/tower-cli", line 11, in
sys.exit(cli())
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 722, in __ca ll__
return self.main(args, *kwargs)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python2.7/dist-packages/tower_cli/cli/base.py", line 129, in invoke
return super(TowerCLI, self).invoke(ctx)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 1066, in inv oke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 895, in invo ke
return ctx.invoke(self.callback, *ctx.params)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 535, in invo ke
return callback(args, *kwargs)
File "/usr/local/lib/python2.7/dist-packages/tower_cli/conf.py", line 359, in method_with_context_managed
method(args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tower_cli/cli/misc.py", line 332, in receive
receiver.receive(all=all, asset_input=assets_to_export)
File "/usr/local/lib/python2.7/dist-packages/tower_cli/cli/transfer/receive.py ", line 12, in receive
exported_objects = self.export_assets(all, asset_input)
File "/usr/local/lib/python2.7/dist-packages/tower_cli/cli/transfer/receive.py ", line 81, in export_assets
exported_asset[common.ASSET_RELATION_KEY][relation] = common.extract_workflo w_nodes(asset)
File "/usr/local/lib/python2.7/dist-packages/tower_cli/cli/transfer/common.py" , line 176, in extract_workflow_nodes
del node_to_add["unified_job_template"]
KeyError: 'unified_job_template'

[itcrowdsource]

@gregdek Thanks!
The import/export procedure works very well indeed. The only thing that need to be taken into account is that (logically) the passwords or secret keys of the credentials aren't stored in the json file. In my case the import script couldn't create the job templates because these are pulled from a GIT repository because it needs an account to be setup. So I had to first configure my GIT account to pull the playbooks from the GIT repository. After that I ran the tower-cli import script again to recreate the job templates. I think I'm going to create a small script that handles these sequential actions from the export.

[dnc92301]

What's strange is the export (towercli receive) script was able to retrieve the "encrypted" password for the GIT credentials. This was validated on version 1.0.1.167. During the import (tower send) , it was able to recreate the Job templates sourcing from Git. This is an indicator, the "encrypted" password was successfully exported/imported. However, the Machine credential had to be created.

So going through same steps with version 1.0.4* but this time Job templates did NOT get imported successfully. So ended up re-creating password, and then re-running import (tower send) to pull the playbooks.

My question is - if the job templates were already created in the database, why is there still a need to setup GIT repository (credentials) before-hand to import these templates via tower-cli send.

[Xiol]

Upgrades for AWX are best effort, and that best effort at present is this process: download awx-cli, export your AWX data with awx-cli, install new version, import via awx-cli. We'll be documenting this more clearly in the near future.

I really wish this had been documented somewhere before I started the upgrade. Can't export from something that's already broken.

I appreciate this isn't the supported version, and believe me I do appreciate the work that goes into this project, but this just isn't great.