Awx: set_facts is not working properly

[Matz13]

ISSUE TYPE

• Bug Report

COMPONENT NAME

• API

SUMMARY

When working with a workflow, the variables "stored" with set_facts are not recovered by subsequent jobs

ENVIRONMENT

• AWX version: 1.0.2.52
• AWX install method: docker on linux
• Ansible version: 2.4
• Operating System: Centos 7
• Web Browser: Chrome

STEPS TO REPRODUCE

• Create 2 jobs, 1 with set_stats, the other using the viariable defined in 1 ( I used those indicated here for testing: http://docs.ansible.com/ansible-tower/latest/html/userguide/workflows.html#extra-variables)
• add them to a workflow
• run the workflow

EXPECTED RESULTS

The second job of the flow should display the value of the variable set in the first (display page content)

ACTUAL RESULTS

The second job fails saying the variable is undefined:
fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'integration_results_url' is undefined\n\nThe error appears to have been in '/var/lib/awx/projects/_19__mchesnel/playbooks/use_set_stats.yml': line 3, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n tasks:\n - name: \"Get test results from the web\"\n ^ here\n\nexception type: <class 'ansible.errors.AnsibleUndefinedVariable'>\nexception: 'integration_results_url' is undefined"}

ADDITIONAL INFORMATION

[ryanpetrello]

@Matz13 thanks for reporting this - I believe what you're encountering is actually a known issue that I'm working on fixing at the moment. Could you confirm something for me with your installation?

echo stats | nc localhost 11211 | grep evict
STAT slab_reassign_evictions_nomem 0
STAT evicted_unfetched 0
STAT evicted_active 0
STAT evictions 0

I'm curious to see if your memcached instance is going over its memory capacity and evicting records.

[Matz13]

Hi @ryanpetrello
This is what I got:

[root@emea awx]# sudo echo stats | nc localhost 11211 | grep evict
STAT slab_reassign_evictions_nomem 0
STAT evicted_unfetched 0
STAT evicted_active 0
STAT evictions 0

I also see a lot of error(104, 'Connection reset by peer') in docker logs awx_task (full trace below)
and also 2018-02-16 18:40:51,108 ERROR awx.main.scheduler Failed to retrieve active tasks from celery
I'm not sure it is related though

docker logs awx_task

[2018-02-16 18:40:50,100: ERROR/MainProcess] Control command error: error(104, 'Connection reset by peer')
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/worker/pidbox.py", line 42, in on_message
    self.node.handle_message(body, message)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/pidbox.py", line 129, in handle_message
    return self.dispatch(**body)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/pidbox.py", line 112, in dispatch
    ticket=ticket)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/pidbox.py", line 135, in reply
    serializer=self.mailbox.serializer)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/pidbox.py", line 265, in _publish_reply
    **opts
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/messaging.py", line 181, in publish
    exchange_name, declare,
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/messaging.py", line 203, in _publish
    mandatory=mandatory, immediate=immediate,
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/channel.py", line 1734, in _basic_publish
    (0, exchange, routing_key, mandatory, immediate), msg
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/abstract_channel.py", line 50, in send_method
    conn.frame_writer(1, self.channel_id, sig, args, content)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/method_framing.py", line 166, in write_frame
    write(view[:offset])
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/transport.py", line 258, in write
    self._write(s)
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 104] Connection reset by peer

[ryanpetrello]

@Matz13 that issue looks eerily similar to another we've been tracking - it may explain other issues you're encountering (like the set_stat issue). When we've seen this in the past, it's sometimes been due to an under provisioned system. Are you deploying in an environment that matches these minimum requirements?

https://github.com/ansible/awx/commit/cf0cc2e2f2db8446c1f80538a0e504fb0a877efc#diff-7d442b7eb49f5fc377f51e74b291cfc1R59

[Matz13]

@ryanpetrello
The machine its running on has 8Gb ram / 4 cores / 50Gb HD (33Gb free) so it doesn't look alike a perf issue.

[ryanpetrello]

Hey @Matz13, I haven't forgotten about you 😄. In a short while, we're going to merge some bug fixes into awx from a hardening branch, and I suspect something in there _may_ resolve the set_stats issue for you. I'll post here again when that happens.

[ryanpetrello]

Hey @Matz13, I believe this change should resolve the issue you're encountering with set_stats (which is in the very recent https://github.com/ansible/awx/releases/tag/1.0.4 release):

https://github.com/ansible/awx/commit/d57470ce49f80afd6b510667be151b035f7b0e55

Do you mind giving it a try?

[Matz13]

Well, I just did, and now everything works fine!
Thanks @ryanpetrello !

[ryanpetrello]

Hey @Matz13,

Thanks for confirming the fix! The issue was a race condition in the way the artifacts were stored between workflow node runs.

[DavidDE33]

@Matz13 & @ryanpetrello
I'm following this thread, beacause i have the same issue.
I just updated my awx and now everything works too.
Thanks

[steefenh]

Unfortunately facing the same issue
When working with a workflow, the variables "stored" with set_facts are not recovered by subsequent jobs
using
AWX 1.0.6.1
Ansible 2.5.2

[steefenh]

please ignore my last post...
read the documentation again and found out that set_fact variables will actually not persist over different playbook runs..too bad

[wenottingham]

Ansible 2.4 and later introduce the cacheable parameter to set_fact - that may be of use to you. http://docs.ansible.com/ansible/latest/modules/set_fact_module.html

[gpulido-redhat]

Ansible 2.4 and later introduce the cacheable parameter to set_fact - that may be of use to you. http://docs.ansible.com/ansible/latest/modules/set_fact_module.html

Nope, it does not work. It seems there is no way for variables set with set_fact to pass between playbooks when running in a workflow. I'd use set_stats, however whenever I define something using it, later commands complain that the variable is not defined, so it seems set_stats does not work either.