Awx: Callback endpoints always give a 500 server error

Created on 12 Jan 2018  路  11Comments  路  Source: ansible/awx
ISSUE TYPE
  • Bug Report
COMPONENT NAME
  • API
SUMMARY

Job callbacks result in 500 error, no jobs are triggered. Exception in logs is Fields set(['launch_type']) are not allowed as overrides.

ENVIRONMENT
  • AWX version: 1.0.2.358
  • AWX install method: docker on linux
  • Ansible version: 2.4.2.0
STEPS TO REPRODUCE
  1. Create a new job template
  2. Check 'Allow Provisioning Callbacks'
  3. Add Host Config Key
  4. POST to job callback URI with Host Config Key
EXPECTED RESULTS

Job should be ran

ACTUAL RESULTS

500 Server Error
{"detail": "A server error has occurred."}

ADDITIONAL INFORMATION
2018-01-12 20:08:02,444 ERROR    django.request Internal Server Error: /api/v2/job_templates/9/callback/
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/views.py", line 3197, in dispatch
    return super(JobTemplateCallback, self).dispatch(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/generics.py", line 253, in dispatch
    return super(APIView, self).dispatch(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 489, in dispatch
    response = self.handle_exception(exc)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 449, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 486, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/views.py", line 3327, in post
    job = job_template.create_job(**kv)
  File "/usr/lib/python2.7/site-packages/awx/main/models/jobs.py", line 309, in create_job
    return self.create_unified_job(**kwargs)
  File "/usr/lib/python2.7/site-packages/awx/main/models/unified_jobs.py", line 358, in create_unified_job
    raise Exception('Fields {} are not allowed as overrides.'.format(unallowed_fields))
Exception: Fields set(['launch_type']) are not allowed as overrides.
172.20.5.57 - - [12/Jan/2018:20:08:02 +0000] "POST /api/v2/job_templates/9/callback/ HTTP/1.1" 500 41 "-" "Mozilla/5.0 (Windows NT; Windows NT 10.0; en-US) WindowsPowerShell/5.1.14393.1944" "-"
[pid: 30|app: 0|req: 140/965] 172.20.5.57 () {36 vars in 602 bytes} [Fri Jan 12 20:08:02 2018] POST /api/v2/job_templates/9/callback/ => generated 41 bytes in 61 msecs (HTTP/1.1 500) 5 headers in 177 bytes (1 switches on core 0)
api high bug
Source
picture rbobrowicz

All 11 comments

I'm also bumping into this as well. I'm using the playbook geerlingguy.awx to got through the versions to see if I can get to one that works. So far I have tried the devel branch(ccb40c8) and tags 1.02, 1.0.1 and 1.0.0 (although that required hacking the install scripts. )

secretbatcave picture secretbatcave on 17 Jan 2018

This should be fixed in devel via https://github.com/ansible/awx/pull/998

ryanpetrello picture ryanpetrello on 17 Jan 2018

This appears to be still happening for me:
I've spun up a fresh instance running from devel (fc5c5400cd1f91891ac2a125d24b005abf23e465) and I'm still seeing what appears to be the same error:

2018-01-18 15:43:50,451 ERROR    django.request Internal Server Error: /api/v2/job_templates/8/callback/
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/views.py", line 3197, in dispatch
    return super(JobTemplateCallback, self).dispatch(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/generics.py", line 253, in dispatch
    return super(APIView, self).dispatch(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 489, in dispatch
    response = self.handle_exception(exc)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 449, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 486, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/views.py", line 3288, in post
    inventory_update = inventory_source.create_inventory_update(launch_type='callback')
  File "/usr/lib/python2.7/site-packages/awx/main/models/inventory.py", line 1443, in create_inventory_update
    return self.create_unified_job(**kwargs)
  File "/usr/lib/python2.7/site-packages/awx/main/models/inventory.py", line 1456, in create_unified_job
    return super(InventorySource, self).create_unified_job(**kwargs)
  File "/usr/lib/python2.7/site-packages/awx/main/models/unified_jobs.py", line 358, in create_unified_job
    raise Exception('Fields {} are not allowed as overrides.'.format(unallowed_fields))
Exception: Fields set(['launch_type']) are not allowed as overrides.

STEPS TO REPRODUCE

  1. Create new awx instance with ansible playbook from devel
  2. Configure so template can run
  3. Check 'Allow Provisioning Callbacks'
  4. Add Host Config Key
  5. POST to job callback URI with Host Config Key
secretbatcave picture secretbatcave on 18 Jan 2018

@secretbatcave yep, looks like a _really_ similar issue (your traceback is _slightly_ different) - I'm looking into it.

ryanpetrello picture ryanpetrello on 18 Jan 2018
👍1

@secretbatcave mind giving it another shot now (devel)?

ryanpetrello picture ryanpetrello on 18 Jan 2018

Hmm, Looks like its still erroring out

using devel(073feb74cb824029a1fe0327c13204f2afebbaf1) for one callback request I get the following two exceptions:

2018-01-18 20:48:08,602 ERROR    django.request Internal Server Error: /api/v2/job_templates/8/callback/
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/views.py", line 3197, in dispatch
    return super(JobTemplateCallback, self).dispatch(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/generics.py", line 253, in dispatch
    return super(APIView, self).dispatch(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 489, in dispatch
    response = self.handle_exception(exc)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 449, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 486, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/views.py", line 3327, in post
    job = job_template.create_job(**kv)
  File "/usr/lib/python2.7/site-packages/awx/main/models/jobs.py", line 309, in create_job
    return self.create_unified_job(**kwargs)
  File "/usr/lib/python2.7/site-packages/awx/main/models/unified_jobs.py", line 358, in create_unified_job
    raise Exception('Fields {} are not allowed as overrides.'.format(unallowed_fields))
Exception: Fields set(['launch_type']) are not allowed as overrides.
2018-01-18 20:48:08,602 ERROR    django.request Internal Server Error: /api/v2/job_templates/8/callback/
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/views.py", line 3197, in dispatch
    return super(JobTemplateCallback, self).dispatch(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/generics.py", line 253, in dispatch
    return super(APIView, self).dispatch(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 489, in dispatch
    response = self.handle_exception(exc)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 449, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 486, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/views.py", line 3327, in post
    job = job_template.create_job(**kv)
  File "/usr/lib/python2.7/site-packages/awx/main/models/jobs.py", line 309, in create_job
    return self.create_unified_job(**kwargs)
  File "/usr/lib/python2.7/site-packages/awx/main/models/unified_jobs.py", line 358, in create_unified_job
    raise Exception('Fields {} are not allowed as overrides.'.format(unallowed_fields))
Exception: Fields set(['launch_type']) are not allowed as overrides.
[pid: 27|app: 0|req: 173/334] 34.243.229.143 () {36 vars in 526 bytes} [Thu Jan 18 20:48:08 2018] POST /api/v2/job_templates/8/callback/ => generated 41 bytes in 61 msecs (HTTP/1.1 500) 5 headers in 177 bytes (1 switches on core 0)
34.243.229.143 - - [18/Jan/2018:20:48:08 +0000] "POST /api/v2/job_templates/8/callback/ HTTP/1.1" 500 41 "-" "curl/7.47.0" "-"
secretbatcave picture secretbatcave on 18 Jan 2018

@secretbatcave are you _certain_ you have the latest code from devel? The line number references in your traceback don't match what I see in HEAD of devel:

  File "/usr/lib/python2.7/site-packages/awx/api/views.py", line 3327, in post
    job = job_template.create_job(**kv)

https://github.com/ansible/awx/blob/43f19cc94b9e6e90111ff153422da2b2a104f379/awx/api/views.py#L3327

ryanpetrello picture ryanpetrello on 18 Jan 2018

It looks to me like you're running an older version of this code from a few commits back:

https://github.com/ansible/awx/blob/1e8c89f5367c5c4060170a07164600b498fa4159/awx/api/views.py#L3327

ryanpetrello picture ryanpetrello on 18 Jan 2018

Ok, Let me redeploy.

secretbatcave picture secretbatcave on 18 Jan 2018

Aha! you are correct, it is the old commit. However I am confused as I'm installing via the built in ansible playbook, using geerlingguy.awx role (it checks out devel, and is on a fresh instance each time). I'm going off line, but I'll have a look tomorrow and _try_ and have a wild stab as to where the problem _might_ be

secretbatcave picture secretbatcave on 18 Jan 2018

I have run from devel, and I can confirm that callbacks work, which is excellent.

Thanks for the fix!

secretbatcave picture secretbatcave on 19 Jan 2018
🎉1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

darkaxl picture darkaxl  路  3Comments
cs35-owncloud picture cs35-owncloud  路  3Comments
beenje picture beenje  路  3Comments
grahamn-gr picture grahamn-gr  路  3Comments
mwiora picture mwiora  路  3Comments