Awx: AwxTaskError.TaskError(instance, rc) Exception: project_update xxx (failed) encountered an error

Created on 21 Jul 2018  路  10Comments  路  Source: ansible/awx
ISSUE TYPE

  • Bug Report
COMPONENT NAME

  • API
  • UI
SUMMARY


While launching a job an activity tries to retrieve the latest updates from the GIT source repository. This activity fails.

ENVIRONMENT
  • AWX version: 1.0.6.35
  • AWX install method: docker on linux
  • Ansible version: 2.6.1
  • Operating System: Ubuntu Xenial
  • Web Browser: Chrome
STEPS TO REPRODUCE

  • Create a job template
  • Create a GIT source project (SCM update options: Clean, Update Revision On Launch)
  • Configure the job template to use a GIT source
  • Run the job template
EXPECTED RESULTS


Job runs succesfully

ACTUAL RESULTS

Job fails:
Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/awx/main/tasks.py", line 881, in run self.pre_run_hook(instance, **kwargs) File "/usr/lib/python2.7/site-packages/awx/main/tasks.py", line 2123, in pre_run_hook task_instance.run(local_project_sync.id) File "/usr/lib/python2.7/site-packages/awx/main/tasks.py", line 574, in _wrapped return f(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/awx/main/tasks.py", line 1027, in run raise AwxTaskError.TaskError(instance, rc) Exception: project_update 567 (failed) encountered an error (rc=2), please see task stdout for details.

ADDITIONAL INFORMATION
api ui bug
Source
picture itcrowdsource

Most helpful comment

@matburt I checked the API stdout of the project update, there was a problem with the requirements.yml file ... some roles are removed from ansible-galaxy .. the update process couldn't retrieve these roles which caused the failure. Would be nice to see the stdout of project updates also in the job output view in the UI. Maybe a proposal for an enhancement?

picture itcrowdsource on 9 Aug 2018
👍7

All 10 comments

Sooo... what was the error in the project update detail view?

Post the reason for the failure and we can reopen as needed.

matburt picture matburt on 24 Jul 2018

@matburt what information is further needed? I've already provided the details about the error message. I'm currently unable to sync a source. In my case a GIT source. If I go into inventory -> Sources ->Click the sync button I'm getting the following error message in the task details view:

Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/awx/main/tasks.py", line 881, in run self.pre_run_hook(instance, **kwargs) File "/usr/lib/python2.7/site-packages/awx/main/tasks.py", line 2123, in pre_run_hook task_instance.run(local_project_sync.id) File "/usr/lib/python2.7/site-packages/awx/main/tasks.py", line 574, in _wrapped return f(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/awx/main/tasks.py", line 1027, in run raise AwxTaskError.TaskError(instance, rc) Exception: project_update 619 (failed) encountered an error (rc=2), please see task stdout for details. SHOW LESS
itcrowdsource picture itcrowdsource on 25 Jul 2018

Look at the error in the job stdout view for the update itself.

matburt picture matburt on 25 Jul 2018

The job's stdout view is empty
awx_update_error

itcrowdsource picture itcrowdsource on 25 Jul 2018

can you grab the api output for this job?

matburt picture matburt on 25 Jul 2018

API doesn't seem to contain data for this job...
awx_api_623

itcrowdsource picture itcrowdsource on 26 Jul 2018

It's not a job... those are playbook runs. It'll be a project update under /api/v2/project_updates/n/

matburt picture matburt on 27 Jul 2018
👍2 
{
    "id": 623,
    "type": "project_update",
    "url": "/api/v2/project_updates/623/",
    "related": {
        "credential": "/api/v2/credentials/2/",
        "unified_job_template": "/api/v2/projects/6/",
        "stdout": "/api/v2/project_updates/623/stdout/",
        "project": "/api/v2/projects/6/",
        "cancel": "/api/v2/project_updates/623/cancel/",
        "notifications": "/api/v2/project_updates/623/notifications/",
        "scm_inventory_updates": "/api/v2/project_updates/623/scm_inventory_updates/",
        "events": "/api/v2/project_updates/623/events/"
    },
    "summary_fields": {
        "instance_group": {
            "id": 1,
            "name": "tower"
        },
        "credential": {
            "id": 2,
            "name": "Bitbucket",
            "description": "Bitbucket",
            "kind": "scm",
            "cloud": false,
            "credential_type_id": 2
        },
        "unified_job_template": {
            "id": 6,
            "name": "GIT source",
            "description": "GIT source",
            "unified_job_type": "project_update"
        },
        "project": {
            "id": 6,
            "name": "GIT source",
            "description": "GIT source",
            "status": "successful",
            "scm_type": "git"
        },
        "user_capabilities": {
            "start": true,
            "delete": true
        }
    },
    "created": "2018-07-25T14:49:56.103826Z",
    "modified": "2018-07-25T14:49:56.119828Z",
    "name": "GIT source",
    "description": "GIT source",
    "local_path": "_6__git_source",
    "scm_type": "git",
    "scm_url": "https://bitbucket.org/xxxxxxxxxxxxxxxxxx/xxxxxxxxxxxxxx.git",
    "scm_branch": "",
    "scm_clean": true,
    "scm_delete_on_update": false,
    "credential": 2,
    "timeout": 0,
    "unified_job_template": 6,
    "launch_type": "sync",
    "status": "failed",
    "failed": true,
    "started": "2018-07-25T14:49:56.102266Z",
    "finished": "2018-07-25T14:50:02.519523Z",
    "elapsed": 6.417,
    "job_args": "[\"ssh-agent\", \"-a\", \"/tmp/awx_623_8awFkx/ssh_auth.sock\", \"sh\", \"-c\", \"ssh-add /tmp/awx_623_8awFkx/credential_2 && rm -f /tmp/awx_623_8awFkx/credential_2 && ansible-playbook -i localhost, -v -e @/tmp/awx_623_8awFkx/tmpq3eTju project_update.yml\"]",
    "job_cwd": "/usr/lib/python2.7/site-packages/awx/playbooks",
    "job_env": {
        "MEMCACHED_PORT_11211_TCP": "tcp://172.17.0.3:11211",
        "ANSIBLE_RETRY_FILES_ENABLED": "False",
        "AWXWEB_ENV_AWX_ADMIN_USER": "admin",
        "http_proxy": "",
        "POSTGRES_ENV_POSTGRES_PASSWORD": "**********",
        "RABBITMQ_PORT_4369_TCP_ADDR": "172.17.0.4",
        "AWXWEB_ENV_DATABASE_USER": "awx",
        "AWXWEB_PORT_8052_TCP_PROTO": "tcp",
        "AWXWEB_PORT_8052_TCP_ADDR": "172.17.0.5",
        "RABBITMQ_ENV_RABBITMQ_LOGS": "-",
        "VIRTUAL_ENV": "/var/lib/awx/venv/ansible",
        "RABBITMQ_PORT_15671_TCP_PORT": "15671",
        "MEMCACHED_ENV_MEMCACHED_VERSION": "1.5.9",
        "SUPERVISOR_ENABLED": "1",
        "RABBITMQ_PORT_15672_TCP_PORT": "15672",
        "RABBITMQ_PORT_5672_TCP_PROTO": "tcp",
        "DJANGO_PROJECT_DIR": "/usr/lib/python2.7/site-packages",
        "RABBITMQ_PORT": "5672",
        "RABBITMQ_HOST": "rabbitmq",
        "PYTHONPATH": "/var/lib/awx/venv/ansible/lib/python2.7/site-packages:/usr/lib/python2.7/site-packages/awx/lib:",
        "POSTGRES_ENV_POSTGRES_USER": "awx",
        "POSTGRES_ENV_LANG": "en_US.utf8",
        "LC_CTYPE": "en_US.UTF-8",
        "MEMCACHED_PORT_11211_TCP_PROTO": "tcp",
        "_MP_FORK_LOGFILE_": "",
        "RABBITMQ_PORT_4369_TCP": "tcp://172.17.0.4:4369",
        "MEMCACHED_PORT_11211_TCP_PORT": "11211",
        "HOSTNAME": "awx",
        "AWXWEB_ENV_DATABASE_NAME": "awx",
        "ANSIBLE_VENV_PATH": "/var/lib/awx/venv/ansible",
        "MEMCACHED_HOST": "memcached",
        "AWXWEB_ENV_RABBITMQ_HOST": "rabbitmq",
        "PROJECT_UPDATE_ID": "623",
        "RABBITMQ_PORT_25672_TCP_PROTO": "tcp",
        "MEMCACHED_NAME": "/awx_task/memcached",
        "CELERY_LOG_REDIRECT_LEVEL": "WARNING",
        "SUPERVISOR_GROUP_NAME": "tower-processes",
        "SECRET_KEY": "**********",
        "POSTGRES_ENV_PGDATA": "/var/lib/postgresql/data/pgdata",
        "RABBITMQ_ENV_RABBITMQ_ERLANG_COOKIE": "cookiemonster",
        "RABBITMQ_PORT_5672_TCP_ADDR": "172.17.0.4",
        "SUPERVISOR_SERVER_URL": "unix:///tmp/supervisor.sock",
        "LANGUAGE": "en_US.UTF-8",
        "SHLVL": "2",
        "AWX_ADMIN_PASSWORD": "**********",
        "CELERY_LOG_FILE": "",
        "RABBITMQ_PORT_5672_TCP_PORT": "5672",
        "ANSIBLE_HOST_KEY_CHECKING": "False",
        "POSTGRES_PORT_5432_TCP": "tcp://172.17.0.2:5432",
        "LANG": "en_US.UTF-8",
        "ANSIBLE_STDOUT_CALLBACK": "awx_display",
        "AWXWEB_ENV_MEMCACHED_PORT": "11211",
        "SUPERVISOR_PROCESS_NAME": "celery",
        "RABBITMQ_PORT_5671_TCP_ADDR": "172.17.0.4",
        "RABBITMQ_PORT_4369_TCP_PORT": "4369",
        "POSTGRES_ENV_POSTGRES_DB": "awx",
        "POSTGRES_PORT_5432_TCP_PORT": "5432",
        "_": "/usr/bin/supervisord",
        "RABBITMQ_VHOST": "awx",
        "AWXWEB_ENV_RABBITMQ_PASSWORD": "**********",
        "DATABASE_NAME": "awx",
        "RABBITMQ_ENV_RABBITMQ_VERSION": "3.7.4",
        "POSTGRES_PORT_5432_TCP_ADDR": "172.17.0.2",
        "RABBITMQ_ENV_RABBITMQ_GITHUB_TAG": "v3.7.4",
        "DJANGO_SETTINGS_MODULE": "awx.settings.production",
        "AWXWEB_ENV_no_proxy": "",
        "TMP": "/tmp",
        "RABBITMQ_PORT_15671_TCP_PROTO": "tcp",
        "ANSIBLE_BECOME_ASK_PASS": "False",
        "ANSIBLE_PARAMIKO_RECORD_HOST_KEYS": "False",
        "AWX_PRIVATE_DATA_DIR": "/tmp/awx_623_8awFkx",
        "RABBITMQ_PORT_5671_TCP_PORT": "5671",
        "AWXWEB_ENV_MEMCACHED_HOST": "memcached",
        "LC_ALL": "en_US.UTF-8",
        "RABBITMQ_PORT_5671_TCP": "tcp://172.17.0.4:5671",
        "HOME": "/var/lib/awx",
        "DISPLAY": "",
        "RABBITMQ_PORT_15672_TCP_PROTO": "tcp",
        "POSTGRES_ENV_GOSU_VERSION": "1.10",
        "_MP_FORK_LOGFORMAT_": "[%(asctime)s: %(levelname)s/%(processName)s] %(message)s",
        "RABBITMQ_PORT_25672_TCP_PORT": "25672",
        "RABBITMQ_PORT_15672_TCP_ADDR": "172.17.0.4",
        "https_proxy": "",
        "RABBITMQ_PORT_4369_TCP_PROTO": "tcp",
        "RABBITMQ_ENV_RABBITMQ_HOME": "/opt/rabbitmq",
        "CACHE": "memcached:11211",
        "AWXWEB_ENV_DATABASE_PORT": "5432",
        "RABBITMQ_ENV_RABBITMQ_GPG_KEY": "**********",
        "AWXWEB_PORT_8052_TCP": "tcp://172.17.0.5:8052",
        "_MP_FORK_LOGLEVEL_": "10",
        "AWXWEB_ENV_RABBITMQ_USER": "guest",
        "AWXWEB_ENV_RABBITMQ_PORT": "5672",
        "CELERY_LOG_LEVEL": "10",
        "DATABASE_HOST": "postgres",
        "AWXWEB_PORT": "tcp://172.17.0.5:8052",
        "POSTGRES_NAME": "/awx_task/postgres",
        "POSTGRES_ENV_PG_VERSION": "9.6.9-2.pgdg90+1",
        "AWXWEB_ENV_RABBITMQ_VHOST": "awx",
        "POSTGRES_ENV_PG_MAJOR": "9.6",
        "RABBITMQ_PORT_15671_TCP": "tcp://172.17.0.4:15671",
        "RABBITMQ_ENV_RABBITMQ_DEFAULT_VHOST": "awx",
        "DJANGO_LIVE_TEST_SERVER_ADDRESS": "localhost:9013-9199",
        "ANSIBLE_ASK_PASS": "False",
        "CELERY_LOG_REDIRECT": "1",
        "AWXWEB_ENV_AWX_ADMIN_PASSWORD": "**********",
        "RABBITMQ_PORT_25672_TCP_ADDR": "172.17.0.4",
        "AWXWEB_ENV_DATABASE_PASSWORD": "**********",
        "PATH": "/var/lib/awx/venv/ansible/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
        "RABBITMQ_PORT_5672_TCP": "tcp://172.17.0.4:5672",
        "DATABASE_USER": "awx",
        "ANSIBLE_CALLBACK_PLUGINS": "/usr/lib/python2.7/site-packages/awx/plugins/callback",
        "RABBITMQ_PORT_15671_TCP_ADDR": "172.17.0.4",
        "TZ": "UTC",
        "POSTGRES_PORT_5432_TCP_PROTO": "tcp",
        "no_proxy": "",
        "RABBITMQ_ENV_RABBITMQ_SASL_LOGS": "-",
        "AWXWEB_PORT_8052_TCP_PORT": "8052",
        "AWXWEB_NAME": "/awx_task/awxweb",
        "AWXWEB_ENV_SECRET_KEY": "**********",
        "AWXWEB_ENV_DATABASE_HOST": "postgres",
        "MEMCACHED_PORT": "11211",
        "MEMCACHED_ENV_MEMCACHED_SHA1": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
        "RABBITMQ_NAME": "/awx_task/rabbitmq",
        "DATABASE_PORT": "5432",
        "CELERY_LOADER": "djcelery.loaders.DjangoLoader",
        "RABBITMQ_PORT_5671_TCP_PROTO": "tcp",
        "AWX_ADMIN_USER": "admin",
        "MEMCACHED_PORT_11211_TCP_ADDR": "172.17.0.3",
        "ANSIBLE_FORCE_COLOR": "True",
        "RABBITMQ_USER": "guest",
        "POSTGRES_PORT": "tcp://172.17.0.2:5432",
        "AWXWEB_ENV_http_proxy": "",
        "PWD": "/var/lib/awx",
        "AWXWEB_ENV_https_proxy": "",
        "RABBITMQ_PORT_15672_TCP": "tcp://172.17.0.4:15672",
        "DATABASE_PASSWORD": "**********",
        "RABBITMQ_PASSWORD": "**********",
        "RABBITMQ_PORT_25672_TCP": "tcp://172.17.0.4:25672"
    },
    "job_explanation": "",
    "execution_node": "awx",
    "result_traceback": "",
    "event_processing_finished": true,
    "project": 6,
    "job_type": "run"
}
itcrowdsource picture itcrowdsource on 27 Jul 2018

@matburt I checked the API stdout of the project update, there was a problem with the requirements.yml file ... some roles are removed from ansible-galaxy .. the update process couldn't retrieve these roles which caused the failure. Would be nice to see the stdout of project updates also in the job output view in the UI. Maybe a proposal for an enhancement?

itcrowdsource picture itcrowdsource on 9 Aug 2018
👍7

If anybody else gets a blank stdout on inventory update job, the TL;DR is:

  • if your AWX GUI URL looks like i.e. /#/jobs/inventory/9135?
  • go to /api/v2/inventory_updates/9135/
  • you'll see a source_project_update field; follow that link
  • then follow resource stdout/ to see the stdout text.

I'm on 9.2.0 but maybe in a future version AWX will show the stdout directly.

darthwalsh picture darthwalsh on 10 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

shortsteps picture shortsteps  路  3Comments
augabet picture augabet  路  3Comments
astraios picture astraios  路  3Comments
icsm2017 picture icsm2017  路  4Comments
gamuniz picture gamuniz  路  3Comments