Awx: Job remains in pending

[beatsandpics]

ISSUE TYPE

• Bug Report

SUMMARY

Job remains in pending, In the log I see a lot of "awx.main.tasks Not running scheduler, another task holds lock"

There are some closed issues regarding this one, but didn't find any solution.

Randomly AWX get stuck where all jobs remain in "pending"

In the logs I see
task_1 | 2020-01-09 16:08:59,344 DEBUG awx.main.dispatch task a1c42d7f-17b2-4590-b013-b82f76105f0c starting awx.main.scheduler.tasks.run_task_manager(*[]) task_1 | 2020-01-09 16:08:59,347 DEBUG awx.main.scheduler Running Tower task manager. task_1 | 2020-01-09 16:08:59,359 DEBUG awx.main.scheduler Not running scheduler, another task holds lock task_1 | 2020-01-09 16:08:59,360 DEBUG awx.main.dispatch task a1c42d7f-17b2-4590-b013-b82f76105f0c is finished

After a lot of time (this time 16min) it suddenly starts to execute the task

task_1 | 2020-01-09 16:24:19,814 DEBUG awx.main.dispatch task b832fea9-c31f-4cfb-8ce3-fda16a1ac5cb starting awx.main.scheduler.tasks.run_task_manager(*[]) task_1 | 2020-01-09 16:24:19,816 DEBUG awx.main.scheduler Running Tower task manager. task_1 | 2020-01-09 16:24:19,827 DEBUG awx.main.scheduler Not running scheduler, another task holds lock task_1 | 2020-01-09 16:24:19,828 DEBUG awx.main.dispatch task b832fea9-c31f-4cfb-8ce3-fda16a1ac5cb is finished task_1 | 2020-01-09 16:24:38,979 DEBUG awx.main.dispatch publish awx.main.tasks.cluster_node_heartbeat(aacec6cc-b438-4cb7-b3f8-2a9ba0b3f399, queue=awx) task_1 | [2020-01-09 16:24:38,979: DEBUG/Process-1] publish awx.main.tasks.cluster_node_heartbeat(aacec6cc-b438-4cb7-b3f8-2a9ba0b3f399, queue=awx) rabbitmq_1 | 2020-01-09 16:24:38.989 [info] <0.3775.0> accepting AMQP connection <0.3775.0> (172.19.0.5:48346 -> 172.19.0.3:5672) rabbitmq_1 | 2020-01-09 16:24:38.991 [info] <0.3775.0> connection <0.3775.0> (172.19.0.5:48346 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost 'awx' rabbitmq_1 | 2020-01-09 16:24:38.996 [info] <0.3775.0> closing AMQP connection <0.3775.0> (172.19.0.5:48346 -> 172.19.0.3:5672, vhost: 'awx', user: 'guest') task_1 | 2020-01-09 16:24:39,003 DEBUG awx.main.dispatch delivered aacec6cc-b438-4cb7-b3f8-2a9ba0b3f399 to worker[178] qsize 0 task_1 | 2020-01-09 16:24:39,005 DEBUG awx.main.dispatch task aacec6cc-b438-4cb7-b3f8-2a9ba0b3f399 starting awx.main.tasks.cluster_node_heartbeat(*[]) task_1 | 2020-01-09 16:24:39,008 DEBUG awx.main.tasks Cluster node heartbeat task. task_1 | 2020-01-09 16:24:39,020 DEBUG awx.main.dispatch task aacec6cc-b438-4cb7-b3f8-2a9ba0b3f399 is finished task_1 | 2020-01-09 16:24:39,090 DEBUG awx.main.dispatch publish awx.main.tasks.awx_k8s_reaper(e33c3d96-163b-487f-8101-00dd80f101a7, queue=awx) task_1 | [2020-01-09 16:24:39,090: DEBUG/Process-1] publish awx.main.tasks.awx_k8s_reaper(e33c3d96-163b-487f-8101-00dd80f101a7, queue=awx) rabbitmq_1 | 2020-01-09 16:24:39.100 [info] <0.3788.0> accepting AMQP connection <0.3788.0> (172.19.0.5:48350 -> 172.19.0.3:5672) rabbitmq_1 | 2020-01-09 16:24:39.102 [info] <0.3788.0> connection <0.3788.0> (172.19.0.5:48350 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost 'awx' rabbitmq_1 | 2020-01-09 16:24:39.104 [info] <0.3788.0> closing AMQP connection <0.3788.0> (172.19.0.5:48350 -> 172.19.0.3:5672, vhost: 'awx', user: 'guest') task_1 | 2020-01-09 16:24:39,105 DEBUG awx.main.dispatch delivered e33c3d96-163b-487f-8101-00dd80f101a7 to worker[178] qsize 0 task_1 | 2020-01-09 16:24:39,108 DEBUG awx.main.dispatch task e33c3d96-163b-487f-8101-00dd80f101a7 starting awx.main.tasks.awx_k8s_reaper(*[]) task_1 | 2020-01-09 16:24:39,117 DEBUG awx.main.dispatch task e33c3d96-163b-487f-8101-00dd80f101a7 is finished task_1 | 2020-01-09 16:24:39,344 DEBUG awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(97554fd6-f2c2-4b5f-a25d-bf09743b9105, queue=awx_private_queue) task_1 | [2020-01-09 16:24:39,344: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(97554fd6-f2c2-4b5f-a25d-bf09743b9105, queue=awx_private_queue) rabbitmq_1 | 2020-01-09 16:24:39.354 [info] <0.3801.0> accepting AMQP connection <0.3801.0> (172.19.0.5:48354 -> 172.19.0.3:5672) rabbitmq_1 | 2020-01-09 16:24:39.356 [info] <0.3801.0> connection <0.3801.0> (172.19.0.5:48354 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost 'awx' task_1 | 2020-01-09 16:24:39,359 DEBUG awx.main.dispatch delivered 97554fd6-f2c2-4b5f-a25d-bf09743b9105 to worker[178] qsize 0 rabbitmq_1 | 2020-01-09 16:24:39.359 [info] <0.3801.0> closing AMQP connection <0.3801.0> (172.19.0.5:48354 -> 172.19.0.3:5672, vhost: 'awx', user: 'guest') task_1 | 2020-01-09 16:24:39,361 DEBUG awx.main.dispatch task 97554fd6-f2c2-4b5f-a25d-bf09743b9105 starting awx.main.tasks.awx_periodic_scheduler(*[]) task_1 | 2020-01-09 16:24:39,377 DEBUG awx.main.tasks Starting periodic scheduler task_1 | 2020-01-09 16:24:39,380 DEBUG awx.main.tasks Last scheduler run was: 2020-01-09 16:05:39.310989+00:00 task_1 | 2020-01-09 16:24:39,390 DEBUG awx.main.dispatch task 97554fd6-f2c2-4b5f-a25d-bf09743b9105 is finished

Is there a way to know which other task blocks the scheduler?

ENVIRONMENT

• AWX version: 9.1.0
• AWX install method: docker on linux
• Operating System: Centos 7

[beatsandpics]

what infos do you need?

[wenottingham]

What sort of deployment do you have - what is the topology?

If this is in kubernetes, what is your pod status?

[beatsandpics]

I'm using docker-compose with an external Postgres DB.
It's a AWX 3.x version (where we had the same issue) which I upgraded to 9.x

version: '2'
services:

  nginx:
    image: nginx:stable
    restart: unless-stopped
    volumes:
      - "~/docker_projects/awx/nginx_ssl.conf:/etc/nginx/nginx.conf:ro"
      - "~/docker_projects/awx/cert/:/etc/nginx/certs/"
    ports:
      - "80:80"
      - "443:443"
    depends_on:
      - web

  web:
    image: ansible/awx_web:9.1.0
    container_name: awx_web
    depends_on:
      - rabbitmq
      - memcached
    ports:
      - "8052"
    hostname: awxweb
    user: root
    restart: unless-stopped
    volumes:
      - "~/docker_projects/awx/SECRET_KEY:/etc/tower/SECRET_KEY"
      - "~/docker_projects/awx/environment.sh:/etc/tower/conf.d/environment.sh"
      - "~/docker_projects/awx/credentials.py:/etc/tower/conf.d/credentials.py"
      - "~/docker_projects/awx/nginx.conf:/etc/nginx/nginx.conf:ro"
    environment:
      http_proxy:
      https_proxy:
      no_proxy:


  task:
    image: ansible/awx_task:9.1.0
    container_name: awx_task
    depends_on:
      - rabbitmq
      - memcached
      - web
    hostname: awx
    user: root
    restart: unless-stopped
    volumes:
      - "~/docker_projects/awx/SECRET_KEY:/etc/tower/SECRET_KEY"
      - "~/docker_projects/awx/environment.sh:/etc/tower/conf.d/environment.sh"
      - "~/docker_projects/awx/credentials.py:/etc/tower/conf.d/credentials.py"
    environment:
      http_proxy:
      https_proxy:
      no_proxy:

  rabbitmq:
    image: ansible/awx_rabbitmq:3.7.4
    container_name: awx_rabbitmq
    restart: unless-stopped
    environment:
      RABBITMQ_DEFAULT_VHOST: "awx"
      RABBITMQ_DEFAULT_USER: "guest"
      RABBITMQ_DEFAULT_PASS: "awxpass"
      RABBITMQ_ERLANG_COOKIE: cookiemonster
      http_proxy:
      https_proxy:
      no_proxy:

  memcached:
    image: "memcached:alpine"
    container_name: awx_memcached
    restart: unless-stopped
    environment:
      http_proxy:
      https_proxy:
      no_proxy:

[beatsandpics]

The containers them self run fine

$ docker ps
CONTAINER ID        IMAGE                        COMMAND                  CREATED             STATUS              PORTS                                                 NAMES
24f48f26e685        nginx:stable                 "nginx -g 'daemon of…"   10 days ago         Up 10 days          0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp              awx_nginx_1
caece6b0501a        ansible/awx_web:9.1.0        "/tini -- /bin/sh -c…"   10 days ago         Up 10 days          0.0.0.0:32771->8052/tcp                               awx_web
eddb2f852093        ansible/awx_task:9.1.0       "/tini -- /bin/sh -c…"   10 days ago         Up 10 days          8052/tcp                                              awx_task
8737f57877f1        memcached:alpine             "docker-entrypoint.s…"   10 days ago         Up 10 days          11211/tcp                                             awx_memcached
7a824047110d        ansible/awx_rabbitmq:3.7.4   "docker-entrypoint.s…"   10 days ago         Up 10 days          4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp   awx_rabbitmq

[fischerdr]

I am starting to see this same problem. Has anyone else encounter this?
ENVIRONMENT

AWX version: 9.2.0
AWX install method: docker on linux
Operating System: RHEL 7

[xTrekStorex]

Similar issue here, just that my jobs never resume. Started occuring with the upgrade to 9.2.0 for me. The job that gets stuck is usually my main workflow that runs every 20 minutes. Sometimes its a subworkflow for job slicing. Clicking cancel in AWX web doesn't do anything.
After a restart of the awx_task container I can cancel the job in AWX web and the next ~6 runs (2 hours) run fine. Then it gets stuck again at a random point in the workflow, where new jobs for the next template should be created. Since in my case it never hangs on a job that actually runs a playbook I don't think its related to hosts or tasks.

As for an example, my last running job (767795) started at log timestamp 19:11:47 then it got stuck.
The main workflow started at 19:00:21 UTC (log time) as job 767756.
https://pastebin.com/3E0WZWWb

AWX version: 9.2.0
AWX install method: Local Docker Compose
Operating System: Debian 10 with kernel 5.4 from backports

[ryanpetrello]

@xTrekStorex any chance you could run this in the task container?

awx-manage run_dispatcher --status

...and report back what it prints?

[AlanCoding]

A bit of an extension on that question, if you already have shell. First run awx-manage shell_plus and then

def check_lock():
    import time
    from awx.main.utils.pglock import advisory_lock
    time.sleep(1)
    with advisory_lock('task_manager_lock', wait=False) as acquired:
        return acquired

you can copy and paste functions, so you can past that and hit enter twice.

next run

[check_lock() for i in range(20)]

Expect it to take 20 seconds to run. This assures that we get a full cycle. If any entry returns True, then that indicates that this lock is not held up perpetually.

[xTrekStorex]

@xTrekStorex any chance you could run this in the task container?

awx-manage run_dispatcher --status

...and report back what it prints?

2020-02-26 08:43:04,275 WARNING  awx.main.dispatch checking dispatcher status for awx
listening on ['awx_private_queue [direct]', 'awx [direct]', 'tower_broadcast_all_awx [fanout]']
awx[pid:144] workers total=4 min=4 max=50
.  worker[pid:58556] sent=5 finished=4 qsize=1 rss=121.879MB
     - running a07fbd79-5a41-4572-977c-21198e9269a1 run_task_manager(*[])
.  worker[pid:63835] sent=372 finished=372 qsize=0 rss=128.586MB [IDLE]
.  worker[pid:64365] sent=140 finished=140 qsize=0 rss=129.656MB [IDLE]
.  worker[pid:64373] sent=131 finished=131 qsize=0 rss=129.492MB [IDLE]

A bit of an extension on that question, if you already have shell. First run awx-manage shell_plus and then
[...]

>>> [check_lock() for i in range(20)]
[False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False]

If you need anything else I'd be happy to try things or post more logs etc.

[xTrekStorex]

I just tried killing the worker with the running job. From what I can see in AWX web this causes the next jobs in the workflow to spawn as pending but then gets stuck again once "run_task_manager" runs again:

bash-4.4# kill 58556
bash-4.4# awx-manage run_dispatcher --status
2020-02-26 09:13:18,360 WARNING  awx.main.dispatch checking dispatcher status for awx
listening on ['awx_private_queue [direct]', 'awx [direct]', 'tower_broadcast_all_awx [fanout]']
awx[pid:144] workers total=3 min=4 max=50
.  worker[pid:64851] sent=10 finished=10 qsize=0 rss=130.457MB [IDLE]
.  worker[pid:64860] sent=8 finished=8 qsize=0 rss=130.570MB [IDLE]
.  worker[pid:64862] sent=10 finished=10 qsize=0 rss=130.633MB [IDLE]

bash-4.4# awx-manage run_dispatcher --status
2020-02-26 09:13:44,632 WARNING  awx.main.dispatch checking dispatcher status for awx
listening on ['awx_private_queue [direct]', 'awx [direct]', 'tower_broadcast_all_awx [fanout]']
awx[pid:144] workers total=4 min=4 max=50
.  worker[pid:64851] sent=10 finished=10 qsize=0 rss=130.457MB [IDLE]
.  worker[pid:64860] sent=8 finished=8 qsize=0 rss=130.570MB [IDLE]
.  worker[pid:64862] sent=12 finished=11 qsize=1 rss=137.480MB
     - running 44d2fc0d-58d8-4df6-a5a6-ca961221b7da run_task_manager(*[])
.  worker[pid:64904] sent=1 finished=1 qsize=0 rss=138.359MB [IDLE]

FYI: I ran awx-manage run_dispatcher --reload before that (which didn't have any effect on the issue - as expected) so that is why the other workers have new PIDs

Restarting the entire awx_task container "fixes" the issue. The pending jobs start running shortly afterwards and the workflow completes after "running" for 12 hours - no timeout.
Ignore the failed status, this was due to a temporary issue with a play.

[ryanpetrello]

@xTrekStorex,

We're actively tracking what we think is a regression, and I think you're encountering a version of it. The next time you see this, could you:

• Run awx-manage run_dispatcher --status and find the pid of the process stuck on run_task_manager. In your last comment, this pid would be pid:64862.
• Install and run either (or both) of these tools, and share the output of that pid:

yum install strace
strace -p 64862

yum install strace
gdb -p 64862

[ryanpetrello]

@xTrekStorex,

Also, is there any way you could share with us what your workflow looks like?

[blaisemichel]

sosreport-rgogbabteca01-awxhang-2020-02-26-mugqzxz.tar.xz.zip.001.txt
sosreport-rgogbabteca01-awxhang-2020-02-26-mugqzxz.tar.xz.zip.002.txt
sosreport-rgogbabteca01-awxhang-2020-02-26-mugqzxz.tar.xz.zip.003.txt
sosreport-rgogbabteca01-awxhang-2020-02-26-mugqzxz.tar.xz.zip.004.txt

remove .txt for extract i have the same problem and push to you my sosreport

[blaisemichel]

awx.dump.zip.001.txt
awx.dump.zip.002.txt

remove .txt for extract i have the same problem and push to you my dumpdb

[ryanpetrello]

@blaisemichel,

If you're still encountering a hung install, could you run the commands referenced above and share the output?

https://github.com/ansible/awx/issues/5617#issuecomment-591107161
https://github.com/ansible/awx/issues/5617#issuecomment-591405762

[xTrekStorex]

@xTrekStorex,

We're actively tracking what we think is a regression, and I think you're encountering a version of it. The next time you see this, could you:

• Run awx-manage run_dispatcher --status and find the pid of the process stuck on run_task_manager. In your last comment, this pid would be pid:64862.
• Install and run either (or both) of these tools, and share the output of that pid:
  [...]

Will do as soon as it happens again. For now I encountered a different but maybe related issue of schedules not running anymore. I see a few schedulers waiting:

2020-02-26 17:50:54,983 WARNING  awx.main.dispatch checking dispatcher status for awx
listening on ['awx_private_queue [direct]', 'awx [direct]', 'tower_broadcast_all_awx [fanout]']
awx[pid:150] workers total=18 min=4 max=50
.  worker[pid:26066] sent=1 finished=0 qsize=1 rss=120.391MB
     - running ee25366a-d3db-440b-94a5-5937d227dc7e gather_analytics(*[])
.  worker[pid:106189] sent=1 finished=0 qsize=1 rss=125.023MB
     - running d6669a75-d7cf-4b6f-bfb2-9747bc2332b8 awx_isolated_heartbeat(*[])
.  worker[pid:285021] sent=1 finished=0 qsize=1 rss=125.820MB
     - running 21007516-f287-4af1-b020-723194963400 awx_k8s_reaper(*[])
.  worker[pid:298569] sent=2 finished=1 qsize=1 rss=128.562MB
     - running ebe401c5-d295-48bd-8ac8-0419048f915d awx_periodic_scheduler(*[])
.  worker[pid:341017] sent=1 finished=0 qsize=1 rss=126.910MB
     - running 9e666314-40ca-4ab4-aec9-f6be038ccd4b awx_isolated_heartbeat(*[])
.  worker[pid:383663] sent=1 finished=0 qsize=1 rss=128.758MB
     - running da08c5ba-4ccd-4618-85ef-de6d3e094783 awx_isolated_heartbeat(*[])
.  worker[pid:384184] sent=1 finished=0 qsize=1 rss=129.391MB
     - running 5c4fb992-0f1d-4c73-adb6-f58e84350d93 awx_isolated_heartbeat(*[])
.  worker[pid:384186] sent=1 finished=0 qsize=1 rss=129.391MB
     - running 6ad1a202-6762-4f0e-8cec-bb38c08b435c gather_analytics(*[])
.  worker[pid:385794] sent=1 finished=0 qsize=1 rss=129.508MB
     - running 452cd4dc-489f-4e42-87a6-ae8da70197e8 awx_periodic_scheduler(*[])
.  worker[pid:385894] sent=1 finished=0 qsize=1 rss=129.508MB
     - running dcbde91e-c16f-4953-a2b3-2768e696f6a6 awx_k8s_reaper(*[])
.  worker[pid:386251] sent=1 finished=0 qsize=1 rss=130.410MB
     - running 87c70cce-554f-4723-b64c-850a94206afb awx_isolated_heartbeat(*[])
.  worker[pid:386258] sent=1 finished=0 qsize=1 rss=130.410MB
     - running d6d566f5-9e57-4e5b-ab16-957dfa961904 awx_k8s_reaper(*[])
.  worker[pid:386925] sent=1 finished=0 qsize=1 rss=131.250MB
     - running 0cacc96e-88be-4716-a7c5-11697dd31706 awx_periodic_scheduler(*[])
.  worker[pid:387030] sent=1 finished=0 qsize=1 rss=131.258MB
     - running c3db034d-070b-439c-9fd2-4efcda59fcb8 cluster_node_heartbeat(*[])
.  worker[pid:390159] sent=1 finished=0 qsize=1 rss=133.598MB
     - running e0f5f725-92fd-4357-86f6-33d9154b26dd cluster_node_heartbeat(*[])
.  worker[pid:390316] sent=1 finished=0 qsize=1 rss=134.129MB
     - running b5fc6193-a3a9-4691-8654-b54d4e9ad206 cluster_node_heartbeat(*[])
.  worker[pid:390557] sent=1 finished=0 qsize=1 rss=133.867MB
     - running 18e52063-8342-4dc0-a04b-6f2345b3d2cd awx_k8s_reaper(*[])
.  worker[pid:390898] sent=1 finished=0 qsize=1 rss=135.133MB
     - running 52ee922a-9e06-4b74-851c-0465bd204f88 awx_periodic_scheduler(*[])

Running strace on any of them returns
futex(0x5577f72e6060, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff

However I can run JTs manually for now.

[ryanpetrello]

@xTrekStorex,

Any chance you could run:

gdb -p <pid> on one of those and share the output?

[xTrekStorex]

@xTrekStorex,

Also, is there any way you could share with us what your workflow looks like?

Sure can.

Maybe some context first:
I import our vCenter into inventory (via a modified version of the contrib script adding vCenter tags).
My workflow chains a vCenter sync and my JTs which each are limited to a host group generated from vCenter tags during import. That way I control basic & continuous Ansible deployments via vCenter instead of AWX. Each JT has an error handling path calling itself again to handle plays that failed due to temporary issues like dpkg locks, DNS timeouts etc. If it still fails it terminates the workflow as failed due to missing error handling.

[ryanpetrello]

Is there a specific part of the workflow where you find it hangs? Or does it tend to be random?

[xTrekStorex]

@xTrekStorex,

Any chance you could run:

gdb -p <pid> on one of those and share the output?

https://pastebin.com/9dab4iW3

Is there a specific part of the workflow where you find it hangs? Or does it tend to be random?

Totally random, sometimes nginx, sometimes docker, sometimes common. Can't say I've ever seen it with a pending vCenter sync though.

[ryanpetrello]

@xTrekStorex

Sorry, I gave slightly wrong instructions.

Once you get the (gdb) prompt, could you try running these commands (and sharing output):

bt
py-bt

Thank you so much 😄

[xTrekStorex]

@xTrekStorex

Sorry, I gave slightly wrong instructions.

Once you get the (gdb) prompt, could you try running these commands (and sharing output):

bt
py-bt

Thank you so much 😄

Unfortunately py-bt doesn't work due to missing debuginfos for python.
yum install python36-debug.x86_64 doesn't help and the official docs (https://wiki.python.org/moin/DebuggingWithGdb) with

yum install yum-utils
debuginfo-install glibc
yum install gdb python-debuginfo

fail with

No match for argument: python-debuginfo
Error: Unable to find a match: python-debuginfo

yum update doesn't solve this either.

Here is the bt output though: https://pastebin.com/wnkc9SQn

[ryanpetrello]

@xTrekStorex

Something like this might get you closer, depending on your platform:

dnf debuginfo-install -y --enablerepo='base-debuginfo' python36

[xTrekStorex]

@xTrekStorex

Something like this might get you closer, depending on your platform:

dnf debuginfo-install -y --enablerepo='base-debuginfo' python36

Unfortunately no

bash-4.4# dnf debuginfo-install -y --enablerepo='base-debuginfo' python36
error: rpmdb: damaged header #173 retrieved -- skipping.
enabling epel-debuginfo repository
error: rpmdbNextIterator: skipping h#     173 blob size(55540): BAD, 8 + 16 * il(78) + dl(54284)
Last metadata expiration check: 0:06:50 ago on Wed 26 Feb 2020 07:32:16 PM UTC.
Could not find debuginfo for package: python36-3.6.8-2.module_el8.1.0+245+c39af44f.x86_64
Could not find debuginfo for package: python36-3.6.8-2.module_el8.1.0+245+c39af44f.x86_64
No debuginfo packages available to install
Dependencies resolved.
Nothing to do.
Complete!

My platform is the ansible/awx_task:9.2.0 image from docker hub, which apparently is based on CentOS 8. The host itself is running Debian 10 but even if I install python3-dbg and run gdb there on PIDs matching awx-manage run_dispatcher in the host namespace I don't get py-bt.

The missing debuginfo packages seem to be a common issue from what I can see on Google. An alternative would be to build python and the gdb extension from source but I'd rather not 😄

[ryanpetrello]

After tinkering with an install for about an hour, I've reproduced something just like this, and here's what the Python trace looks like for me (mostly leaving this for my benefit):

(gdb) py-bt
Traceback (most recent call first):
  <built-in method acquire of _thread.lock object at remote 0x7fd0ac08fd78>
  File "<frozen importlib._bootstrap>", line 98, in acquire
  File "<frozen importlib._bootstrap>", line 149, in __enter__
  File "<frozen importlib._bootstrap>", line 968, in _find_and_load
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/kombu/transport/__init__.py", line 13, in supports_librabbitmq
    import librabbitmq  # noqa
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/kombu/connection.py", line 251, in _init_params
    if transport == 'amqp' and supports_librabbitmq():
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/kombu/connection.py", line 199, in __init__
    self._init_params(**params)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/kombu.py", line 17, in __init__
    super(Connection, self).__init__(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/publish.py", line 89, in apply_async
    with Connection(settings.BROKER_URL) as conn:
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/publish.py", line 67, in delay
    return cls.apply_async(args, kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/common.py", line 819, in <lambda>
    connection.on_commit(lambda: run_task_manager.delay())
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/db/backends/base/base.py", line 628, in on_commit
    func()
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/common.py", line 819, in _schedule_task_manager
    connection.on_commit(lambda: run_task_manager.delay())
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/utils/common.py", line 843, in schedule_task_manager
    _schedule_task_manager()
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/models/unified_jobs.py", line 1310, in signal_start
    schedule_task_manager()
  (frame information optimized out)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 62, in run_callable
    return _call(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/task.py", line 86, in perform_work
    result = self.run_callable(body)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/base.py", line 147, in work_loop
    self.perform_work(body, *args)
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python3.6/multiprocessing/popen_fork.py", line 73, in _launch
    code = process_obj._bootstrap()
  File "/usr/lib64/python3.6/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/lib64/python3.6/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/usr/lib64/python3.6/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 67, in start
    self.process.start()
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 229, in up
    worker.start()
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 391, in up
    return super(AutoscalePool, self).up()
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py", line 399, in write
--Type <RET> for more, q to quit, c to continue without paging--
    self.up()
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/base.py", line 103, in process_task
    self.pool.write(queue, body)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/kombu/messaging.py", line 590, in <listcomp>
    [callback(body, message) for callback in callbacks]
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/kombu/messaging.py", line 590, in receive
    [callback(body, message) for callback in callbacks]
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/kombu/messaging.py", line 624, in _receive_callback
    return on_m(message) if on_m else self.receive(decoded, message)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/amqp/channel.py", line 1615, in _on_basic_deliver
    fun(msg)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/amqp/abstract_channel.py", line 145, in dispatch_method
    listener(*args)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/amqp/connection.py", line 518, in on_inbound_method
    method_sig, payload, content,
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/amqp/method_framing.py", line 79, in on_frame
    callback(channel, msg.frame_method, msg.frame_args, msg)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/amqp/connection.py", line 511, in blocking_read
    return self.on_inbound_frame(frame)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/amqp/connection.py", line 505, in drain_events
    while not self.blocking_read(timeout):
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/kombu/transport/pyamqp.py", line 103, in drain_events
    return connection.drain_events(**kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/kombu/connection.py", line 323, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/kombu/mixins.py", line 197, in consume
    conn.drain_events(timeout=safety_interval)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/kombu/mixins.py", line 175, in run
    for _ in self.consume(limit=None, **kwargs):
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/worker/base.py", line 111, in run
    super(AWXConsumer, self).run(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/management/commands/run_dispatcher.py", line 88, in handle
    consumer.run()
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/__init__.py", line 152, in manage
    execute_from_command_line(sys.argv)
  (frame information optimized out)

[ryanpetrello]

@xTrekStorex I'm having a lot of trouble reproducing this in a reliable fashion that allows me to observe and experiment with this issue, but it's a high priority to resolve it, and I have a hunch about what it might be related to in recent commit history.

Are you willing to apply a patch and run it for a day or so to see if your hanging issues go away?

If so, give this a whirl:

https://github.com/ansible/awx/pull/6093

cc @shanemcd @chrismeyersfsu @AlanCoding

[xTrekStorex]

@xTrekStorex I'm having a lot of trouble reproducing this in a reliable fashion that allows me to observe and experiment with this issue, but it's a high priority to resolve it, and I have a hunch about what it might be related to in recent commit history.

Are you willing to apply a patch and run it for a day or so to see if your hanging issues go away?
[...]

Done. Will get back to you asap

[ryanpetrello]

For posterity, I very much suspect this is actually related to a bug in cpython itself (https://bugs.python.org/issue38884) that we've begun encountering due to this recent PR of mine:

https://github.com/ansible/awx/pull/5837

...which moved our periodic heartbeat code (when we removed celery a few weeks ago) to a thread (instead of a distinct process). Details:

https://github.com/ansible/awx/pull/6093#issuecomment-591665229

[ryanpetrello]

@xTrekStorex I've done some testing on my end and am feeling good about my PR. If I hear back from you in the coming days (that you're unable to reproduce with my patch), I'll probably merge https://github.com/ansible/awx/pull/6093 and incorporate it into the next AWX release.

[xTrekStorex]

@xTrekStorex I've done some testing on my end and am feeling good about my PR. If I hear back from you in the coming days (that you're unable to reproduce with my patch), I'll probably merge #6093 and incorporate it into the next AWX release.

I don't want to jinx it but in the past 24 hours I have not encountered the issue again.
Everything ran and keeps running, schedules work - no complaints.
I don't intend to switch back to the official Docker image anytime soon so I'll just leave your PR running.

What I noticed though is that workers seem to be recreated a lot more often now.
When I still encountered the issue the sent and finished values per worker were in the 130-350 range. Now they are more like 10-90 and I didn't restart the task container (24h uptime).
I also just noticed the PIDs increased again so they are definitely new workers.

[ryanpetrello]

@xTrekStorex do you have something on a schedule that's running over, say, the weekend?

With the old 9.2.0 image, were you having trouble encountering the issue often (more than, say, once per day?)

I'm not ready to call this solved, but if you're convinced with a before/after, I'm likely to merge my PR early next week.

[xTrekStorex]

@xTrekStorex do you have something on a schedule that's running over, say, the weekend?

With the old 9.2.0 image, were you having trouble encountering the issue often (more than, say, once per day?)

I'm not ready to call this solved, but if you're convinced with a before/after, I'm likely to merge my PR early next week.

I encountered the issue every 2-6 hours.

There are no major changes planned for the weekend however my workflow runs continuously 24/7, every 20 minutes with a runtime of ~18 minutes to enforce configuration state (I moved to AWX from the Puppet world - it's a habit). So it will create and run plenty of jobs if that is what you're asking.

[ryanpetrello]

@xTrekStorex thanks for helping us test this, I'll check back in with you next week 👍

[blaisemichel]

hello after update of friday my awx is already hang

(InteractiveConsole)

def check_lock():
... import time
... from awx.main.utils.pglock import advisory_lock
... time.sleep(1)
... with advisory_lock('task_manager_lock', wait=False) as acquired:
... return acquired
...
[check_lock() for i in range(20)]
[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]

awx-manage run_dispatcher --status

• queued 47901249-6852-497c-b239-d13cfe6dc567 run_task_manager([])
  
  
  
  • queued e971997f-5054-4808-a980-3eeb49426237 awx_periodic_scheduler([])
  
  
  • queued 9a038bb5-c493-47bb-8667-55ddc8385987 awx_k8s_reaper([])
  
  
  • queued 1b36917d-e5d1-40ff-ba27-e356dabae1e7 awx_periodic_scheduler([])
  
  
  • queued 93fbd67c-5b4f-42e0-a011-8f7d53be9607 awx_periodic_scheduler([])
  
  
  • queued 9910bb10-2479-40fb-8b3c-946ff127ba67 awx_periodic_scheduler([])
  
  
  • queued 3083d3a3-97cf-4791-a40b-2ddf0dfe7b47 awx_k8s_reaper([])
    
    . worker[pid:35212] sent=253 finished=0 qsize=253 rss=151.992MB
  
  
  • running f9c4f29e-704f-4f2a-ba72-d88747f412cf cluster_node_heartbeat([])
  
  
  • queued af671ef7-7220-4b3d-8650-31204006a709 run_task_manager([])
  
  
  • queued 5f098a4f-7424-47e2-b4ad-39de05e7ba09 run_task_manager([])
  
  
  • queued f9fc7049-1060-41fb-8188-8d7bf693deb9 awx_k8s_reaper([])
  
  
  • queued b8cd4350-3f53-48a8-bead-ca0b05787099 awx_periodic_scheduler([])
  
  
  • queued 984a7040-16b8-42a4-975a-b8cec3555239 awx_periodic_scheduler([])
  
  
  • queued 98229f47-9ba6-4853-8fe3-d78542569679 run_task_manager([])
  
  
  • queued e3a69689-f958-472d-9c47-27c66681e399 run_task_manager([])
  
  
  • queued 3fb7b978-2a8e-412f-a7ec-b02585b71099 cluster_node_heartbeat([])
  
  
  • queued a5040010-7b15-4108-9ced-49cf8152b669 cluster_node_heartbeat([])
  
  
  • queued 66d7ee1e-c37e-4121-834f-53f9263ccc69 run_task_manager([])
  
  
  • queued 5225e018-8c30-4eb0-884c-b5d25eb6ab09 cluster_node_heartbeat([])
  
  
  • queued ab327461-2c95-40e0-b57e-75be51a26c99 awx_periodic_scheduler([])
  
  
  • queued 39c8c16c-4b32-454f-b6a8-1c7e24869059 awx_k8s_reaper([])
  
  
  • queued 431182d7-9c40-4994-9f0b-f236eb202d19 run_task_manager([])
  
  
  • queued 7bee902a-717e-4772-94f1-d05e5e7dfb29 run_task_manager([])
  
  
  • queued 0688cf58-bb73-49d3-bec4-85c2123127c9 run_task_manager([])
  
  
  • queued 5ac06666-4b79-42c3-b3ee-2afb0122eed9 awx_periodic_scheduler([])
  
  
  • queued 5eb0b9c5-ac10-4d82-b3b0-df217f3b8669 run_task_manager([])
  
  
  • queued f0a7e56d-0e52-49ee-8b68-8cfb1b75eff9 awx_k8s_reaper([])
  
  
  • queued 9876fcc5-6878-4db3-90bb-2bdc84ad6a09 awx_periodic_scheduler([])
  
  
  • queued 90542787-3f1b-40fe-bf4a-84c8cf4118b9 run_task_manager([])
  
  
  • queued 67f7cfea-a57b-4c01-834c-735d99e1e889 run_task_manager([])
  
  
  • queued 139ea367-8df7-4b4d-8cdb-52017f0762c9 run_task_manager([])
  
  
  • queued 2d9c5c10-356c-4747-90f5-a10b4ba4f439 run_task_manager([])
  
  
  • queued dc7450f9-4bfe-45ba-88be-796bca221819 awx_periodic_scheduler([])
  
  
  • queued 0fc257a7-c195-4b10-9c61-f9550c9d1b09 awx_periodic_scheduler([])
  
  
  • queued 3f53b7c6-0cdb-4b8c-ab35-27fadf4f4c09 gather_analytics([])
  
  
  • queued 0668ce46-3909-4e54-8758-c6f9f1f34529 gather_analytics([])
  
  
  • queued e47d7d19-cd3f-47d0-af71-99922889b319 awx_k8s_reaper([])
  
  
  • queued e1fe67c7-e642-4cae-9dc4-b68b797fed89 run_task_manager([])
  
  
  • queued f269a8ef-0542-40b1-a62b-11e52a8ac929 run_task_manager([])
  
  
  • queued f6c3344f-22ab-4bd7-bc14-7420bb0dce59 awx_isolated_heartbeat([])
  
  
  • queued 331fa3c3-8516-4b73-9ed4-f1f37a7ddeb9 awx_periodic_scheduler([])
  
  
  • queued d30c9e93-559f-43d0-9322-dd560ceca3d9 awx_periodic_scheduler([])
  
  
  • queued 8aa0f67a-3932-4f38-b71b-e10cee62e409 run_task_manager([])
  
  
  • queued 1d8c9736-0472-4240-9c24-2c2c25e414c9 run_task_manager([])
  
  
  • queued 1aee5718-a97b-48e8-967b-7083e330f4d9 awx_periodic_scheduler([])
  
  
  • queued dc22727f-78ab-446b-8926-8d249a6b5be9 awx_periodic_scheduler([])
  
  
  • queued f6caf798-1464-45ff-a027-86212c7ec7d9 run_task_manager([])
  
  
  • queued 66c3975e-3a6d-4292-bc57-a4c44a14c349 awx_periodic_scheduler([])
  
  
  • queued 9a27e34a-808a-40bc-bc9d-9d4aa81e4b39 awx_k8s_reaper([])
  
  
  • queued 0544febb-a9ff-4307-bf30-a098f243d779 run_task_manager([])
  
  
  • queued b0daec78-4015-435a-965b-ebd43d495418 run_task_manager([])
  
  
  • queued 062ceb1e-4707-4171-94c3-d40d65807098 awx_k8s_reaper([])
  
  
  • queued 3b564768-d2eb-488e-98a3-67e3d6ad2018 awx_periodic_scheduler([])
  
  
  • queued 5d187626-9539-45e6-989d-d93db446d6f8 awx_periodic_scheduler([])
  
  
  • queued 83812e38-bb38-47ca-9da3-3cc82c553a98 awx_isolated_heartbeat([])
  
  
  • queued d8d404ee-2433-4116-8592-50592d2df2a8 awx_k8s_reaper([])
  
  
  • queued 03099a70-b556-4f33-b59d-e7d0aca16178 cluster_node_heartbeat([])
  
  
  • queued 65991711-bef9-43eb-a9d5-2550c79560b8 awx_periodic_scheduler([])
  
  
  • queued e76dd5ee-8b06-4623-9956-0c7db051ee28 run_task_manager([])
  
  
  • queued 617b9e74-0c26-4fbf-9194-cbb52d068318 awx_periodic_scheduler([])
  
  
  • queued 7b2c068f-7311-4a2e-873e-a676f04a55b8 awx_periodic_scheduler([])
  
  
  • queued 8dbb7fd7-dc87-48e7-b747-311a793847e8 run_task_manager([])
  
  
  • queued 8578d292-8c41-4028-991f-ddbcd4ae4768 run_task_manager([])
  
  
  • queued efc68acc-ad85-4a96-9c2a-93013d8e53d8 awx_periodic_scheduler([])
  
  
  • queued 27ee9466-ffe7-426b-9eab-0f1342ab6e88 run_task_manager([])
  
  
  • queued f9d226c6-739d-4e6f-9cbc-bb08758f2578 run_task_manager([])
  
  
  • queued d7046d14-4fa5-4861-a30b-d3a5dc7c72f8 run_task_manager([])
  
  
  • queued c640294b-f2bc-44d4-969c-f4f52dacccf8 run_task_manager([])
  
  
  • queued 353c6707-7fe6-409b-861f-2a0330b5bf08 run_task_manager([])
  
  
  • queued 458fa1b2-bed8-467b-a812-5352ed493418 awx_k8s_reaper([])
  
  
  • queued 05cf0d9e-ec63-40ca-b1e2-79612054f588 run_task_manager([])
  
  
  • queued 914a762e-a8f8-44b9-a807-90d8cd3b2838 awx_periodic_scheduler([])
  
  
  • queued 80d47fb1-fdf2-46c9-8981-bd9033856eb8 cluster_node_heartbeat([])
  
  
  • queued f2504e39-0b05-4087-8f89-fbc48fb35b78 awx_periodic_scheduler([])
  
  
  • queued 7a6888e2-b47f-4dec-9c79-b89bb2bbd578 awx_periodic_scheduler([])
  
  
  • queued 549f1a4a-6714-4935-8e99-077d61456ad8 awx_periodic_scheduler([])
  
  
  • queued 49852fd3-ec1e-46e9-99d1-37c064d397a8 awx_periodic_scheduler([])
  
  
  • queued ae135e52-036f-493d-b7fd-440267316d08 run_task_manager([])
  
  
  • queued 1d4e06a5-3350-4abc-afbe-ff1852519848 awx_periodic_scheduler([])
  
  
  • queued 730a3513-4952-4b1d-9776-53964e499d88 awx_k8s_reaper([])
  
  
  • queued 117e9982-cf78-45bf-a22d-f135ae140e28 gather_analytics([])
  
  
  • queued 87d75014-2884-441d-8e33-457d96713c08 run_task_manager([])
  
  
  • queued e872f750-0dd2-49d8-8c75-5a8f4f518978 run_task_manager([])
  
  
  • queued 2ed698da-5ee0-4f17-914c-fbf6c4c731c8 run_task_manager([])
  
  
  • queued 0e385e6e-b529-482c-9b1f-defc42d685e8 run_task_manager([])
  
  
  • queued 32ea0fa9-0a4a-4dfa-8fcd-170524b93698 awx_periodic_scheduler([])
  
  
  • queued aa925bef-2bb1-4982-9028-0b924c9b7518 run_task_manager([])
  
  
  • queued a32bb053-55f9-42ea-b795-4c9c69ab6bd8 cluster_node_heartbeat([])
  
  
  • queued 6b7892b3-e8a2-4384-b37c-ec65188ebf28 awx_periodic_scheduler([])
  
  
  • queued 7c44bca6-27d4-4f1b-8db9-cc5acfa6e778 awx_k8s_reaper([])
  
  
  • queued 97e4b72d-9879-406d-a5a1-3956960ad728 run_task_manager([])
  
  
  • queued a931f244-5d8d-42cd-a65d-bca163854e38 awx_k8s_reaper([])
  
  
  • queued 9ea3946f-b90e-4df5-8f0c-5a463eaa7bd8 run_task_manager([])
  
  
  • queued 59baac63-1e18-4f3f-8edf-2ba841d08be8 run_task_manager([])
  
  
  • queued dd5f90c9-f35c-4435-8a6a-a657fc2156e8 run_task_manager([])
  
  
  • queued 8fd6d8d2-b108-4e43-91a7-1260776e8358 awx_periodic_scheduler([])
  
  
  • queued 900cad0d-396b-49b3-a001-342645eeebd8 run_task_manager([])
  
  
  • queued cec84e4b-4bb9-4f8e-97fe-aa2e9e9c1088 awx_periodic_scheduler([])
  
  
  • queued 53fba48b-b780-4e5a-8f91-308816ffdd38 run_task_manager([])
  
  
  • queued bf9089e4-594b-4258-8100-14550cd9f4e8 run_task_manager([])
  
  
  • queued 98cceebb-18e1-4cc7-9bb8-9aee4bf5e318 awx_k8s_reaper([])
  
  
  • queued 90895ede-da72-4d1d-a374-c18f7de31608 run_task_manager([])
  
  
  • queued 86331aab-9c88-48a0-86d0-cb5fa867ab68 awx_periodic_scheduler([])
  
  
  • queued 96a5cdc0-f1a7-4b73-a693-5afa7e7da788 cluster_node_heartbeat([])
  
  
  • queued c2d2093f-de34-4314-9b87-e8313ac2b628 run_task_manager([])
  
  
  • queued 59191236-9597-451c-89a6-2b078fb9aeb8 run_task_manager([])
  
  
  • queued c9883747-408c-4dab-a613-e9ad20508808 run_task_manager([])
  
  
  • queued d4cfa266-538a-4ac8-9922-d7e7b76b5948 run_task_manager([])
  
  
  • queued 04e79359-8025-47cd-9b2a-ba9f9fa86ac8 awx_periodic_scheduler([])
  
  
  • queued 2db88c1a-1da4-4523-aa10-2779e2f654a8 run_task_manager([])
  
  
  • queued e9f93169-eb7d-4282-ab08-b1c95a19df28 run_task_manager([])
  
  
  • queued ae1c4d91-e824-4db6-87d4-1459b9eceb68 awx_k8s_reaper([])
  
  
  • queued e5d9c762-3bef-4aeb-bf33-87a5f279da68 run_task_manager([])
  
  
  • queued 43ea6300-4026-4291-9869-8ad72c9ade68 cluster_node_heartbeat([])
  
  
  • queued 19a97b33-0116-4701-8a57-c1740f18d058 awx_k8s_reaper([])
  
  
  • queued 4951e973-c351-4bdf-a846-7cba7ed63618 cluster_node_heartbeat([])
  
  
  • queued c7af36e1-ab1a-40be-b8b8-766c99806ae8 awx_k8s_reaper([])
  
  
  • queued e32db75d-61a2-4e51-a64f-91a91826b2e8 run_task_manager([])
  
  
  • queued 8554d657-cadb-48e6-834e-d48858ce3638 awx_periodic_scheduler([])
  
  
  • queued 1d1fcc32-ee20-49f0-8888-646800bcd3c8 awx_periodic_scheduler([])
  
  
  • queued f425775e-afdf-4af5-9721-3d5919c386a8 awx_k8s_reaper([])
  
  
  • queued 2d0bc273-c578-499e-aa65-8be667daa9b8 run_task_manager([])
  
  
  • queued 55c4adab-2708-471f-8af7-5c3d4887fa08 run_task_manager([])
  
  
  • queued 47428ada-cb63-4c8c-92cc-6514916391d8 run_task_manager([])
  
  
  • queued d316b728-6a46-4444-83bf-f9359cea9178 awx_k8s_reaper([])
  
  
  • queued 2a92d76a-9822-46dd-a815-795b4d9b7fd8 run_task_manager([])
  
  
  • queued b5b71293-c92f-40d6-978c-c3641b6696e8 gather_analytics([])
  
  
  • queued d4146b6e-2928-4cb2-a0ab-b324300468d8 awx_periodic_scheduler([])
  
  
  • queued 242b6d77-47ef-4098-9c91-91b22b7ad4f8 cluster_node_heartbeat([])
  
  
  • queued bf11cf42-1cf9-45a9-8f80-04de4b632e18 run_task_manager([])
  
  
  • queued 03ad3f4f-232c-4f7e-8fd8-988e6f224328 awx_periodic_scheduler([])
  
  
  • queued 86b74cfb-e655-49a5-81d3-fc15dc356e18 awx_periodic_scheduler([])
  
  
  • queued e661f3c7-f8a8-4f8b-ae23-4d9b03e5bed8 awx_k8s_reaper([])
  
  
  • queued 15d810a6-ffe9-4065-a245-ca0097a34d38 awx_periodic_scheduler([])
  
  
  • queued 2172acc2-ffc8-44ec-8078-cc3f32c2b168 run_task_manager([])
  
  
  • queued f9f9f57d-bbd3-4fce-b450-b240db9e3608 awx_isolated_heartbeat([])
  
  
  • queued 8ebf8cab-be93-4f37-be08-b90f0392d138 run_task_manager([])
  
  
  • queued 6b1c73be-4674-47c9-9c7f-61b995d0e0f8 cluster_node_heartbeat([])
  
  
  • queued 5bdfa418-0eef-47ec-a713-a4713d7ce3c8 awx_k8s_reaper([])
  
  
  • queued ce6a02a4-ea95-4203-9f31-d9ab5bd90328 run_task_manager([])
  
  
  • queued 44e64169-d643-44e5-9891-b77c03b48e98 awx_k8s_reaper([])
  
  
  • queued aeb78f7d-fcb2-4522-8eb6-efc35f7c47d8 awx_periodic_scheduler([])
  
  
  • queued b919acf8-899d-4187-9684-5a66ea3a9c38 awx_periodic_scheduler([])
  
  
  • queued 08caa6f2-8d55-4717-99b6-529ec6206828 awx_periodic_scheduler([])
  
  
  • queued 6fdf6255-809f-49f8-8f02-0667881922a8 awx_periodic_scheduler([])
  
  
  • queued d2418ef1-d6c2-4f9e-8537-b0c515bfb438 awx_k8s_reaper([])
  
  
  • queued 76d77c3c-0c56-4fa3-974b-d06be5280488 awx_periodic_scheduler([])
  
  
  • queued c245fdfb-f9c5-4855-8c93-065ade0d2278 awx_k8s_reaper([])
  
  
  • queued df14d76d-41e5-4669-a426-028efad64e58 awx_k8s_reaper([])
  
  
  • queued 4d471001-b1c2-4b49-99c0-5b5eff9bbe58 run_task_manager([])
  
  
  • queued 8a93f24b-c3f9-43e5-aa1b-e92ba36d0618 awx_periodic_scheduler([])
  
  
  • queued 955395e7-be58-4226-a59e-52fef3b6f888 run_task_manager([])
  
  
  • queued 33f8ccf2-8d37-4249-bfc0-19d25866eb68 run_task_manager([])
  
  
  • queued 217cfb85-804b-4a95-af3a-8dac94882968 awx_k8s_reaper([])
  
  
  • queued 1cffb860-fb77-41a9-a6ed-ef40de6062f8 run_task_manager([])
  
  
  • queued d5bbd4c5-2b28-4739-b53a-ca185285e8c8 awx_periodic_scheduler([])
  
  
  • queued f9078ecf-b70c-4418-8371-e42113931008 awx_k8s_reaper([])
  
  
  • queued 99924a5c-9d73-462f-bfa5-5f8b1dbee618 awx_periodic_scheduler([])
  
  
  • queued 881a5cca-688c-466e-a3d8-5936759fad38 awx_periodic_scheduler([])
  
  
  • queued 7decc141-1c2d-42b7-a943-ee59d5e45778 cluster_node_heartbeat([])
  
  
  • queued 895195ba-7024-4801-a558-30827f5c99c8 awx_k8s_reaper([])
  
  
  • queued fab16081-7989-4a53-9b19-ec53b6bae018 run_task_manager([])
  
  
  • queued 67dbaab9-a648-4ab3-b98f-f714a229a7b8 gather_analytics([])
  
  
  • queued 18a9d259-d6f3-4a63-967b-482ccfb2b7a8 run_task_manager([])
  
  
  • queued 9e517ff7-7c09-4c5d-a345-02dc6a6b74b8 run_task_manager([])
  
  
  • queued f7c9f33d-5373-4737-a296-c736fd4d6de8 awx_isolated_heartbeat([])
  
  
  • queued db1c2155-8c14-46a3-b7d2-456314987798 awx_periodic_scheduler([])
  
  
  • queued a2a3eb09-6578-42a8-b06a-291fcd5425f8 cluster_node_heartbeat([])
  
  
  • queued a4a39fc0-8ba7-4bbd-bbe2-149c0d4eb4b8 run_task_manager([])
  
  
  • queued 77c54a45-c130-4165-b18e-7c262b0e97e8 awx_periodic_scheduler([])
  
  
  • queued 2d9a4ea5-46c7-4ccd-af17-19707e39b4e8 gather_analytics([])
  
  
  • queued fdd949e1-ef8f-42d0-8b55-a64f960bf828 awx_periodic_scheduler([])
  
  
  • queued 3f3317b5-8175-4540-b50b-f6923e8c4cf8 awx_k8s_reaper([])
  
  
  • queued d4acf7bc-881e-4fd6-b828-a94330ae4f08 awx_periodic_scheduler([])
  
  
  • queued f95cc68a-af8a-4268-acdf-0ba8717be468 awx_periodic_scheduler([])
  
  
  • queued 06b02ac4-f0b5-4b76-a77b-e6478234cc88 awx_periodic_scheduler([])
  
  
  • queued cefa6ffd-b778-4afe-b4b0-f4a0b4591c18 awx_periodic_scheduler([])
  
  
  • queued 68959abf-c0a0-434a-bb84-d64d938f8378 run_task_manager([])
  
  
  • queued 927b8055-9111-41bf-9d5f-e77ae8bfc9a8 run_task_manager([])
  
  
  • queued f6463de3-fe06-46c0-82ef-a9b3ed7555c8 cluster_node_heartbeat([])
  
  
  • queued e7a71afc-a005-4f5a-99d4-e4f8a00cbdc8 cluster_node_heartbeat([])
  
  
  • queued ac0229b1-84e5-4172-a04b-2b39edcc3358 awx_periodic_scheduler([])
  
  
  • queued fb8b988c-bd8a-42dd-911c-085d56decea8 awx_periodic_scheduler([])
  
  
  • queued a72b839a-661f-4430-9d70-38989c5c2038 run_task_manager([])
  
  
  • queued a2ca253a-8a3b-4f9c-ba72-027b83219cd8 awx_periodic_scheduler([])
  
  
  • queued faba938f-df21-42a6-9b12-4240d5d779d8 run_task_manager([])
  
  
  • queued e4cbedde-1965-4bca-a6f4-e55ea1d28d18 cluster_node_heartbeat([])
  
  
  • queued d7a0083f-05ff-42cc-9a89-02728a12a878 awx_periodic_scheduler([])
  
  
  • queued ee34c20e-204f-4c30-9346-15830c701848 cluster_node_heartbeat([])
  
  
  • queued 72d871d5-9146-4ab4-ba89-884164ae6a08 run_task_manager([])
  
  
  • queued 26762ba4-7519-4d23-a7ad-5c4b378ae178 run_task_manager([])
  
  
  • queued f4eadb7d-ef87-4c7c-8137-7d3cf749ced8 awx_k8s_reaper([])
  
  
  • queued 23b9f4b0-840a-47a0-b22b-ed2d48c96278 run_task_manager([])
  
  
  • queued 6b75c790-3a32-4efd-8ece-f26f5bc607b8 awx_k8s_reaper([])
  
  
  • queued fb6abcfe-e03b-4fe3-8629-6bfe87328758 run_task_manager([])
  
  
  • queued 7e1cd3e3-6ff5-4c10-a8b9-da02c725ab98 cluster_node_heartbeat([])
  
  
  • queued 75190e6d-1548-4638-a149-68c9956afec8 gather_analytics([])
  
  
  • queued 20cb1a17-1c0b-4fa6-9c2d-f6c16ead7c98 run_task_manager([])
  
  
  • queued 2017c6c1-b6f7-4008-a24b-1e3a8ea339a8 awx_periodic_scheduler([])
  
  
  • queued b9080caf-f574-44a3-811f-db110c841658 run_task_manager([])
  
  
  • queued 6a0d5501-874d-4b5a-8c42-11245efc0bb8 run_task_manager([])
  
  
  • queued 02ac2013-f7b8-4bd0-ae88-cb024aa2e918 run_task_manager([])
  
  
  • queued 7ce4b5d3-37c6-4878-b6b5-d745d92ce8a8 awx_k8s_reaper([])
  
  
  • queued 3dc40915-b5c6-4e99-aec3-4057790cbc18 awx_periodic_scheduler([])
  
  
  • queued e043de5e-e5b6-4bc1-ba0c-8e9402b23938 cluster_node_heartbeat([])
  
  
  • queued 04cc1d57-02b7-4245-8f53-14c10060de38 cluster_node_heartbeat([])
  
  
  • queued b9959c4a-9f8a-496d-b2c3-7855a25d17f8 awx_periodic_scheduler([])
  
  
  • queued 756b0597-fd3b-4efd-9ef3-e63614f61688 awx_periodic_scheduler([])
  
  
  • queued 1dcd1d84-ec15-4907-9bd1-7c55db49f078 awx_periodic_scheduler([])
  
  
  • queued d67c007a-62f4-4640-a886-f465963c9c58 awx_periodic_scheduler([])
  
  
  • queued 3e5d2ae5-fe80-47f6-a80e-1d61276b4478 run_task_manager([])
  
  
  • queued 6d56387c-1f9e-49a7-abfb-a4704bdd7d68 awx_k8s_reaper([])
  
  
  • queued 11c8b77c-5832-49e8-a207-2d8ef953b998 run_task_manager([])
  
  
  • queued cf4c2c3e-5288-4e7b-ab5d-7f11f7f70228 cluster_node_heartbeat([])
  
  
  • queued 5ab2624b-da52-4e54-b6c0-c5e8c5870e28 awx_periodic_scheduler([])
  
  
  • queued 327d2a79-f9dc-4654-b458-a3411207a118 run_task_manager([])
  
  
  • queued 29c05fad-7e01-4c19-8955-6e49b2b1f558 run_task_manager([])
  
  
  • queued b047f5da-d420-4478-a164-6c1a69437998 run_task_manager([])
  
  
  • queued e6bee596-e95e-45f8-8920-de63e36dfc88 cluster_node_heartbeat([])
  
  
  • queued 8b20e29b-e92d-4918-b122-be405b6d31d8 run_task_manager([])
  
  
  • queued 6ff243c5-cb21-43a7-b56b-64847d469948 awx_periodic_scheduler([])
  
  
  • queued 96d7c4c4-e017-4a24-ad83-25769ba19b58 run_task_manager([])
  
  
  • queued 95d5c3ef-96c7-4e97-b711-70794ebcaf18 run_task_manager([])
  
  
  • queued 4aeb1a2d-29af-430a-9882-725ac020fb58 run_task_manager([])
  
  
  • queued 231271c1-9a06-45eb-92b7-8e8ba9cb31a8 cluster_node_heartbeat([])
  
  
  • queued 045d1c55-c07e-400d-b161-f7cfa1043538 gather_analytics([])
  
  
  • queued 29696ecb-67a5-4680-bc57-1192b5b81558 awx_periodic_scheduler([])
  
  
  • queued 0caea10d-5eac-4f5e-816e-ef344a040ed8 awx_periodic_scheduler([])
  
  
  • queued 2f80cf7b-8b77-465e-a441-554f721d97a8 awx_k8s_reaper([])
  
  
  • queued 4969de58-23a8-4512-8265-066a06a433c8 awx_k8s_reaper([])
  
  
  • queued 7c133157-43e9-4db7-87a9-aade5fb4a908 awx_k8s_reaper([])
  
  
  • queued 17906555-9ee1-4e47-a86e-9bab5c110158 awx_k8s_reaper([])
  
  
  • queued 55c771fd-8a12-47aa-a72b-f130947f0948 awx_periodic_scheduler([])
  
  
  • queued 66e7927a-ac2c-47ed-8ece-5564da7bd438 run_task_manager([])
  
  
  • queued 729d2f70-a271-4ff0-916b-7fe842a3abe8 run_task_manager([])
  
  
  • queued 9ef81261-b336-470b-a99f-14f0c0e69ef8 run_task_manager([])
  
  
  • queued 0e81b573-ace9-4096-8bd8-e8591ffdd358 run_task_manager([])
  
  
  • queued 08ba3327-7f10-489e-9c7e-8e9a30235f18 run_task_manager([])
  
  
  • queued 6b941a1c-de10-4c51-8b37-fec1584eb848 run_task_manager([])
  
  
  • queued cc68f81d-1210-43b8-aa19-52ac3db6b1a8 awx_periodic_scheduler([])
  
  
  • queued 65ebe330-a4cb-4a1b-b9f8-cf8872483558 awx_periodic_scheduler([])
  
  
  • queued 2b6d2201-745f-4c82-8cc3-d841c279e7d8 run_task_manager([])
  
  
  • queued 01f5bec5-555c-4437-96df-8c2946adcac8 run_task_manager([])
  
  
  • queued fca30437-4f2a-4712-b2b0-fa26182d0818 run_task_manager([])
  
  
  • queued ad7b0d3a-7451-4afb-bc26-a1fcff3ff718 run_task_manager([])
  
  
  • queued 19b946f9-0177-4e82-b29b-59bd4b3b84a8 awx_periodic_scheduler([])
  
  
  • queued f664a557-699a-4063-a651-4b1554ff7b48 cluster_node_heartbeat([])
  
  
  • queued 753453b9-c007-4c15-8220-2d26d90e40b8 awx_k8s_reaper([])
  
  
  • queued 38c1c6e0-50c7-4d89-8f84-b36fbf022fa8 run_task_manager([])
  
  
  • queued a3cb31b0-ed48-4f2f-b069-d7e17f2137c8 run_task_manager([])
  
  
  • queued d32012ca-ce68-4ab2-9deb-1113086f8458 awx_periodic_scheduler([])
  
  
  • queued 7565a899-b03c-479e-b313-6be709c54188 awx_k8s_reaper([])
  
  
  • queued 6cb6ac34-d441-4fb6-ab16-6752ce25be68 run_task_manager([])
  
  
  • queued 502a897d-601e-4f0b-a7c5-298b69c2d498 awx_periodic_scheduler([])
  
  
  • queued e6343e99-e6f5-48f2-81c3-970c71ad05f8 cluster_node_heartbeat([])
  
  
  • queued 60e7d2cb-97ae-4f98-9c09-67537578d848 run_task_manager([])
  
  
  • queued 090e3499-8510-465d-b98c-85166b361e38 cluster_node_heartbeat([])
  
  
  • queued 4b4ee9c4-dcab-483e-ab1c-9fc635ff1a28 awx_periodic_scheduler([])
  
  
  • queued c245f7f8-0137-4cdd-baba-fd2fea9b5bb8 awx_k8s_reaper([])
  
  
  • queued 4d8d0870-b2e2-43ec-9b3d-c981269f1a58 run_task_manager([])
    
    . worker[pid:35267] sent=260 finished=0 qsize=260 rss=152.051MB
  
  
  • running bf1b41f6-1807-4730-8e5f-2698711a4c2c cluster_node_heartbeat([])
  
  
  • queued 8390b6fa-d5b9-48b1-9b8f-ac31f330d52a awx_periodic_scheduler([])
  
  
  • queued c023920a-25ba-48e1-8160-59c1bb81ac6a awx_k8s_reaper([])
  
  
  • queued 170ad881-1809-4b67-80f5-c9c3b9b53caa run_task_manager([])
  
  
  • queued 503e92b1-dc22-463e-93f1-c0cd30b7bf6a awx_periodic_scheduler([])
  
  
  • queued f0367a08-36e2-4347-8ba5-c9f223c8e51a run_task_manager([])
  
  
  • queued fab477d9-35de-47f4-9411-ef861685d34a awx_periodic_scheduler([])
  
  
  • queued 130af800-5e93-4b63-9c9d-3b21a00c604a run_task_manager([])
  
  
  • queued 14e6e4e1-220c-4bf2-a031-17c8657a300a awx_periodic_scheduler([])
  
  
  • queued 50a979ee-679d-4680-bcdc-16fdb3ae4d0a run_task_manager([])
  
  
  • queued 77d3162e-21c7-4e3c-a843-02af164d46aa run_task_manager([])
  
  
  • queued 7903280c-9b70-46ac-b66a-e06676e5297a awx_periodic_scheduler([])
  
  
  • queued 0cc283e6-0bf2-421b-9d53-0a89a54017ca awx_periodic_scheduler([])
  
  
  • queued f8723de3-844a-4837-9d2e-830baaf2c9da awx_periodic_scheduler([])
  
  
  • queued 0bc6066a-7584-4a9f-afd1-661a89b45c5a awx_k8s_reaper([])
  
  
  • queued 79d8b4f2-0e16-4d8b-91eb-eeb4e5d147ba run_task_manager([])
  
  
  • queued 33c3aede-539d-4bbb-82cb-3572cd648a7a run_task_manager([])
  
  
  • queued 8d9f83d3-55d6-4251-ac48-4d8ae4af49ba gather_analytics([])
  
  
  • queued a24e8d8f-8af0-492e-992a-f6a4a7c5e0fa run_task_manager([])
  
  
  • queued eebba431-f822-491c-a1a6-2db8f31ad28a awx_k8s_reaper([])
  
  
  • queued c1aa7dff-e077-4259-a802-26beb06a607a cluster_node_heartbeat([])
  
  
  • queued 3a082350-3511-4395-b8ac-ff388e06052a run_task_manager([])
  
  
  • queued 1a0a4a5f-21b1-46fc-8067-f9b0097e8c1a awx_k8s_reaper([])
  
  
  • queued d8c6dfc9-aeaa-45d4-ae99-7a955540f8ba cluster_node_heartbeat([])
  
  
  • queued 9013949b-0e6a-4a5e-b7ab-9632d88d261a awx_periodic_scheduler([])
  
  
  • queued 4f89ebcb-21cd-4d83-b5b7-a98e44b3ebca run_task_manager([])
  
  
  • queued 000ff5a5-9836-4405-b1c3-b58f32776cca cluster_node_heartbeat([])
  
  
  • queued fd3e6667-f8ef-42ed-b372-6c54d451f9ca run_task_manager([])
  
  
  • queued 63ff6423-b432-4907-b377-c36c33080a0a run_task_manager([])
  
  
  • queued e2e2d337-d31e-42de-bfd1-b144f6c6274a awx_periodic_scheduler([])
  
  
  • queued cea42d52-7e73-495e-bb5c-f0a237c0597a run_task_manager([])
  
  
  • queued e9ab9594-c3f3-467c-9fe4-90d1e300ef0a cluster_node_heartbeat([])
  
  
  • queued f5dd164c-bb42-40fb-b385-b4e427bda1da run_task_manager([])
  
  
  • queued 192cb10b-53ca-4bae-994e-befe3268ce8a run_task_manager([])
  
  
  • queued fb5fb23e-eebf-47b4-b7a5-00e7c9dc396a cluster_node_heartbeat([])
  
  
  • queued 6d70a5e9-9bb1-475a-8cc8-49e9cbfe293a awx_periodic_scheduler([])
  
  
  • queued ef75ea6c-84e8-4969-9c3a-070c0925127a awx_periodic_scheduler([])
  
  
  • queued 0f43de75-f981-43a6-a571-ff8d30a216ba cluster_node_heartbeat([])
  
  
  • queued 767a6ef5-4203-4430-8184-a7de8d10e7ba run_task_manager([])
  
  
  • queued 1b8752b8-4708-498c-a7eb-b8dc9de3360a cluster_node_heartbeat([])
  
  
  • queued f411f067-d14b-4b94-ab06-68620541c25a run_task_manager([])
  
  
  • queued b1d5b6b2-15dc-477a-b11c-6a091c6eefda run_task_manager([])
  
  
  • queued 977cc5ad-6216-4d00-9165-1a6ea8cff69a run_task_manager([])
  
  
  • queued e7039d4b-0404-411f-b726-a271a96d00aa cluster_node_heartbeat([])
  
  
  • queued 29e84475-7ec1-42f9-a269-e492c718224a awx_periodic_scheduler([])
  
  
  • queued 8d1b7ff4-623d-4029-afe8-879069650cca run_task_manager([])
  
  
  • queued 9e2a7901-44a5-4eb7-a9e3-ca6e6778fac9 run_task_manager([])
  
  
  • queued 29b28828-3c44-4a8b-a3d3-50478875b819 awx_k8s_reaper([])
  
  
  • queued e60d0798-0d0f-432f-bfea-9bf7075d3329 awx_periodic_scheduler([])
  
  
  • queued 2978a8db-6a45-4b8f-866c-31ed9f8e6a39 gather_analytics([])
  
  
  • queued 7677b3c5-746a-4798-8339-2d9a5dfea669 awx_periodic_scheduler([])
  
  
  • queued 098b1e16-daec-4ba7-b837-167e396ca8d9 run_task_manager([])
  
  
  • queued bdaca4b3-4737-4543-ae47-6bce64814b69 awx_periodic_scheduler([])
  
  
  • queued 6b357af8-128f-43e6-bf9f-fe633b1adb99 awx_periodic_scheduler([])
  
  
  • queued 991b36b2-9e40-45b0-a41d-e8be6b5adf59 run_task_manager([])
  
  
  • queued 78a476db-3557-48b8-93f7-780779b7f489 run_task_manager([])
  
  
  • queued 67e5c594-bda1-4515-8e71-04fd96381449 cluster_node_heartbeat([])
  
  
  • queued 9541b9ee-3773-42fa-8487-6097a97aa9c9 run_task_manager([])
  
  
  • queued e7460382-f0eb-49f7-b8fe-b593513eadd9 awx_periodic_scheduler([])
  
  
  • queued d2c5f2a0-f82b-4bc9-a41a-b776c86e3e59 cluster_node_heartbeat([])
  
  
  • queued 88ec6fc4-8e82-4dff-a733-7836a3f41fb9 run_task_manager([])
  
  
  • queued 93de0985-695d-4ad2-8a12-491d97baa429 run_task_manager([])
  
  
  • queued bb7da223-dfee-48f8-b0a3-bb7e509b60b9 awx_k8s_reaper([])
  
  
  • queued abe48b27-4fd9-4978-955e-a059e3f88f89 run_task_manager([])
  
  
  • queued 3b0deaa2-342e-4f38-b15f-58c8d110aa89 run_task_manager([])
  
  
  • queued c7a7c9bb-998a-43c1-8d69-5573d7879049 awx_k8s_reaper([])
  
  
  • queued 652b4cdb-e15c-404b-9b46-facc40664379 awx_periodic_scheduler([])
  
  
  • queued 5166189d-405b-4871-93f6-af6262a1ff59 run_task_manager([])
  
  
  • queued ac21323f-ac4b-4dc9-8ded-74d2ee2c38e9 awx_periodic_scheduler([])
  
  
  • queued 19c6e0ed-ba54-4993-8d59-f8d493582c69 awx_periodic_scheduler([])
  
  
  • queued edc50484-45eb-460a-a9e3-802fadbefba9 run_task_manager([])
  
  
  • queued c05ed5db-f529-4902-b39c-7c23b39ca339 awx_k8s_reaper([])
  
  
  • queued 1d8d0637-05b0-4a5e-a992-1ff8ad19b799 run_task_manager([])
  
  
  • queued db7c767c-0e6f-48a3-9887-a08cb16ec269 run_task_manager([])
  
  
  • queued 5971eac4-1abe-4484-ba95-b1f2fceeed09 cluster_node_heartbeat([])
  
  
  • queued b5c40e0a-595d-4c54-bc0f-e29d123b0f89 run_task_manager([])
  
  
  • queued 86ca302b-13fc-495f-beac-13813f840a89 run_task_manager([])
  
  
  • queued 3be7d305-7fde-4803-9b43-33b6f2985379 awx_periodic_scheduler([])
  
  
  • queued b92c6c76-2223-4a97-ae75-b053f0f08a29 cluster_node_heartbeat([])
  
  
  • queued 81bd2328-59fc-4101-9757-6eb03998a4c9 cluster_node_heartbeat([])
  
  
  • queued e939c11a-1d10-4442-89c2-7d519ae72439 awx_periodic_scheduler([])
  
  
  • queued 1f229d86-a91d-4b49-8669-7fe69e6bc779 awx_k8s_reaper([])
  
  
  • queued 5b6e2709-987a-45c8-99d0-70b928e1d569 cluster_node_heartbeat([])
  
  
  • queued b6d22921-fbaa-472a-b8cf-0da7d03f7329 awx_periodic_scheduler([])
  
  
  • queued 23cece6c-43ea-4dc8-a257-3539ffe70a69 run_task_manager([])
  
  
  • queued 0ed27b36-010f-4035-b34e-60f979436099 run_task_manager([])
  
  
  • queued 0195d0c9-17cd-48fd-9e9c-f2b3e46e9559 awx_periodic_scheduler([])
  
  
  • queued bf131e6f-93e2-4cb6-9829-f63ca8fddef9 run_task_manager([])
  
  
  • queued 91cc9eff-c378-40d9-adca-ae91be578529 cluster_node_heartbeat([])
  
  
  • queued f7cb105c-a00d-4765-b4e6-e5855a30e7f9 run_task_manager([])
  
  
  • queued ba0cb21a-35fe-4ccc-a3d9-59cfdc60cce9 awx_periodic_scheduler([])
  
  
  • queued 5f3dc2fa-c705-4db5-89e3-0f76a10431b9 run_task_manager([])
  
  
  • queued afe212db-5b88-4005-a6f4-a99ae9403fc9 cluster_node_heartbeat([])
  
  
  • queued a6277a72-b803-44bc-a6ab-d165694cbce9 awx_periodic_scheduler([])
  
  
  • queued a08a94ce-7b2e-45d2-9b0d-3ca03677c019 awx_periodic_scheduler([])
  
  
  • queued 066be578-22e1-4e31-add3-a33b317f6f39 run_task_manager([])
  
  
  • queued 8fbd3ecf-7e93-4c92-9490-2df2d7edd4a9 gather_analytics([])
  
  
  • queued 44303931-1eed-4634-b0ec-6043ce87ea89 awx_k8s_reaper([])
  
  
  • queued 3839ac3f-af4c-42e3-9901-1aee18edce79 run_task_manager([])
  
  
  • queued 3ecb9383-5c1d-4e5c-9415-32012af93fb9 awx_periodic_scheduler([])
  
  
  • queued 79771fb9-d037-4b5b-aa5b-deccefd3ae89 cluster_node_heartbeat([])
  
  
  • queued 65a6a7dd-3572-465b-9905-dbd8aaa8bf49 run_task_manager([])
  
  
  • queued c843ff02-872e-44b5-b3e8-4dbdc79c7ec9 cluster_node_heartbeat([])
  
  
  • queued ea2aea43-c6d7-4e68-9d55-c77ed136bc99 awx_periodic_scheduler([])
  
  
  • queued f7e3b3bb-2e3e-405c-84c8-f3edea518ff9 run_task_manager([])
  
  
  • queued 79ca231f-40ed-49fc-9049-b752ae7bf5a9 awx_periodic_scheduler([])
  
  
  • queued 29907eaf-857d-40be-897e-180d903faef9 run_task_manager([])
  
  
  • queued 7d015ef4-5705-45cf-891d-2b481f881f39 awx_k8s_reaper([])
  
  
  • queued b4fabde1-054d-4147-968b-a67a2a5b30e9 awx_periodic_scheduler([])
  
  
  • queued 8ecd3936-2c1c-4563-b780-19b788d24f29 cluster_node_heartbeat([])
  
  
  • queued 6aa6e23c-bfba-430e-8de7-319d58283c79 awx_isolated_heartbeat([])
  
  
  • queued 17c287e4-6088-4b52-b6e1-f0d50a861829 run_task_manager([])
  
  
  • queued c5a1ea7e-1251-4294-b6aa-6b0d22f93cd9 run_task_manager([])
  
  
  • queued 9282c64d-31ba-4c6f-94cd-b4a5deaafce9 awx_periodic_scheduler([])
  
  
  • queued aaae3038-dde0-4a3c-993c-cee93c3b47a9 run_task_manager([])
  
  
  • queued c6ab97aa-799e-42e5-b463-ff488eb544d9 awx_periodic_scheduler([])
  
  
  • queued 6acdf221-dfdb-4ef3-8c96-b9420d930319 run_task_manager([])
  
  
  • queued d78543a5-35b7-4c34-9c30-f8c91f1ebe69 awx_periodic_scheduler([])
  
  
  • queued 3f977efc-90b1-4da7-aef2-869c6f32dc09 awx_periodic_scheduler([])
  
  
  • queued a090e31e-e447-48e4-849e-5a19fc954049 run_task_manager([])
  
  
  • queued 0a3ddbce-2b60-4f6f-8795-caa414991199 awx_periodic_scheduler([])
  
  
  • queued 4af744b9-383b-4b3e-a99a-f561c4d55d69 awx_periodic_scheduler([])
  
  
  • queued afc0332d-9a9c-4d66-8fbc-6eb6aef16c69 cluster_node_heartbeat([])
  
  
  • queued c58e13ff-57c1-4a66-bdb8-552eee5c9529 run_task_manager([])
  
  
  • queued bf96091c-87f8-4514-9dd2-771d41c00f29 awx_periodic_scheduler([])
  
  
  • queued ab3eb507-2d53-4784-ad6d-b76ecb18a8c9 run_task_manager([])
  
  
  • queued 26808052-8f24-4f60-86d4-4c46bba65c29 run_task_manager([])
  
  
  • queued 8ef80fb1-903a-4bdf-b0c7-37461b7feef9 run_task_manager([])
  
  
  • queued 4ae8cec6-79ac-410e-af6d-63235ab50719 run_task_manager([])
  
  
  • queued e02cfbe5-cbb0-458e-ae40-4daf75fb4229 run_task_manager([])
  
  
  • queued 411f9c0a-49f3-40d7-a735-83b3a0376519 awx_periodic_scheduler([])
  
  
  • queued 6332c205-7fa3-4a47-ba9d-4c3c48817959 cluster_node_heartbeat([])
  
  
  • queued 900d786f-cb32-4c43-a6f1-30ca0bb51c39 awx_periodic_scheduler([])
  
  
  • queued 36a5cb09-3834-4197-bb1a-d37d151139f9 awx_k8s_reaper([])
  
  
  • queued bece88f2-f9e4-4406-9a18-c8f23fd85dc9 run_task_manager([])
  
  
  • queued 8a512c4b-4fb8-45e4-81b1-f4c820585609 cluster_node_heartbeat([])
  
  
  • queued a25574d3-d220-44da-a2b7-6e14bd6a52a9 run_task_manager([])
  
  
  • queued 09631f34-7439-4175-9f5f-10e543526599 run_task_manager([])
  
  
  • queued 6f8a615e-b9a8-47e3-a088-6f951f0bf8c9 run_task_manager([])
  
  
  • queued 5af3d5a9-089d-4490-a8e1-2830e59e5c29 awx_k8s_reaper([])
  
  
  • queued 6180097c-8b5b-4357-9e7a-cb3d2f055799 run_task_manager([])
  
  
  • queued 55209cc4-de6f-416f-9474-5dce0161bf49 awx_periodic_scheduler([])
  
  
  • queued 45f90b4b-4410-427b-a220-0ab963670dd9 cluster_node_heartbeat([])
  
  
  • queued ccb06dce-01ee-4b55-8468-8536b9834899 cluster_node_heartbeat([])
  
  
  • queued 1eabb650-a051-463f-a28a-541eea026929 awx_periodic_scheduler([])
  
  
  • queued 2a1a1458-2b3c-4a8f-b691-d46bdd1d2c89 awx_periodic_scheduler([])
  
  
  • queued 3eb1103a-1f2a-4603-9eef-3412f278e4d9 awx_periodic_scheduler([])
  
  
  • queued e7c9868a-d275-4c9a-9462-26fd8ffdb729 run_task_manager([])
  
  
  • queued 4d0136b8-96aa-4d6c-898b-2651a893f8f9 awx_k8s_reaper([])
  
  
  • queued b6c601aa-1f6b-4b63-a430-0092542dc3e9 run_task_manager([])
  
  
  • queued acdf6766-bb94-457c-a3a5-36bf8245bbc9 run_task_manager([])
  
  
  • queued 5bc6ccaa-887b-4ca1-b4e8-5e359c079ea9 run_task_manager([])
  
  
  • queued 1f088a2f-aead-4b64-a490-c298d42811f9 awx_k8s_reaper([])
  
  
  • queued e1b4a10d-d1e0-4a86-adcd-e6436e4be2d9 cluster_node_heartbeat([])
  
  
  • queued 131f4b6d-7b04-4da1-b30b-c997a517cbc9 awx_k8s_reaper([])
  
  
  • queued 59c4b863-eb15-4a36-aa92-844605ebb269 run_task_manager([])
  
  
  • queued ee4e4c84-7262-41eb-86d5-f3d99bb0ca39 awx_periodic_scheduler([])
  
  
  • queued 7377de9f-ff7b-4219-b53a-e2c62023ac39 run_task_manager([])
  
  
  • queued 7d8bc10b-36d0-450f-b7d0-cc76db71c309 cluster_node_heartbeat([])
  
  
  • queued 25a16535-fa74-4874-9833-fdf90e7bbc39 awx_periodic_scheduler([])
  
  
  • queued 8ca160ae-15b7-4e96-a066-426299000079 run_task_manager([])
  
  
  • queued 70c65257-a983-4d59-9fed-f43694491ee9 awx_k8s_reaper([])
  
  
  • queued c9d507f0-b66c-4738-90d2-819e1415f279 run_task_manager([])
  
  
  • queued 13df5a24-383d-4251-9ec0-d62d969c14c9 awx_periodic_scheduler([])
  
  
  • queued b075cffb-88f9-4f61-9ded-f7b1bb7c5679 awx_periodic_scheduler([])
  
  
  • queued 29253a91-7153-446c-b327-4d9c17117c19 run_task_manager([])
  
  
  • queued 0f7e50d1-4f72-447a-865f-4c35c9e3fe69 awx_k8s_reaper([])
  
  
  • queued 1c57fca0-066d-4d8e-814a-bef1148819d9 awx_k8s_reaper([])
  
  
  • queued d2845a9a-5eb9-44e5-83d4-27f6e7458019 awx_k8s_reaper([])
  
  
  • queued 7b1e8048-2a34-4af6-b4e3-af6f8d45fd69 run_task_manager([])
  
  
  • queued 4641393d-59b2-40bb-a8ba-0604ad3f4369 run_task_manager([])
  
  
  • queued e7c46331-944e-496d-a9a2-3714e6e21719 awx_k8s_reaper([])
  
  
  • queued a4b9e4eb-163d-4eb8-ad86-9df387bfde29 awx_periodic_scheduler([])
  
  
  • queued e4180456-7f6c-47c8-987b-3c91304a0729 awx_periodic_scheduler([])
  
  
  • queued ec713c19-c1fc-444e-9c92-12bf45ebfae9 run_task_manager([])
  
  
  • queued 81db0cbd-0c8d-4419-b893-748c03894539 awx_k8s_reaper([])
  
  
  • queued ac3d0c43-7beb-4ee1-9359-7f83fe6e1489 run_task_manager([])
  
  
  • queued 783bcf83-1460-43e6-b817-58bd125f9629 run_task_manager([])
  
  
  • queued 06ab7942-9bcf-46b0-a2da-4663c9cc98b9 awx_periodic_scheduler([])
  
  
  • queued de7c4471-ef0a-4fb1-9e71-48e823ec46b9 run_task_manager([])
  
  
  • queued dbc9c761-2244-4d31-87fc-746129ced9b9 run_task_manager([])
  
  
  • queued c842d9b5-68f2-46e8-a9e9-6464b0d61d19 awx_periodic_scheduler([])
  
  
  • queued 5078baa8-e56a-4e40-903b-083a045c84b9 awx_periodic_scheduler([])
  
  
  • queued bb45918c-eb07-490c-9ca3-d4b211493ea9 awx_k8s_reaper([])
  
  
  • queued ea08ae18-e511-4f97-98dd-497c310a04f9 awx_k8s_reaper([])
  
  
  • queued d4b41244-ba94-419f-bb35-9f787a4ee2d9 run_task_manager([])
  
  
  • queued 0cb016f5-7fa0-486f-b0e6-671ac6367c39 cluster_node_heartbeat([])
  
  
  • queued 5737a36f-885d-4679-a10b-20ddbecae7f9 cluster_node_heartbeat([])
  
  
  • queued 72ff25bf-8cae-48c4-86a1-11f8fe9faa79 run_task_manager([])
  
  
  • queued af64a592-1983-4f72-9d34-c8804cbf8de9 gather_analytics([])
  
  
  • queued a6fdc448-1c6e-4891-aaad-e1c86b273559 run_task_manager([])
  
  
  • queued d748e268-0ccd-4927-a3e1-174f3150fc79 awx_periodic_scheduler([])
  
  
  • queued 031e720e-1050-4092-bafe-4782f07b46d9 run_task_manager([])
  
  
  • queued cee846df-3875-46d1-b0c0-1de2543c2d49 gather_analytics([])
  
  
  • queued c530cacd-fe36-4ab7-8222-4838fae4a209 awx_periodic_scheduler([])
  
  
  • queued 105ff419-6bb4-405b-86d1-de6e733134b9 awx_k8s_reaper([])
  
  
  • queued bafa6fbc-a451-4942-b02e-cae613613fc9 cluster_node_heartbeat([])
  
  
  • queued d424a349-831a-47c6-b102-051daade63f9 cluster_node_heartbeat([])
  
  
  • queued b861110b-ead0-40d6-a68e-843ad57c5eb9 run_task_manager([])
  
  
  • queued de51f960-3670-4351-bf9b-d47c6c0c1a99 awx_periodic_scheduler([])
  
  
  • queued 7178f301-2fec-4cb4-b004-b05b8a529d99 run_task_manager([])
  
  
  • queued c1affe69-5b96-4825-8e93-25c845de28a9 run_task_manager([])
  
  
  • queued 25563a7f-d13d-4fc1-9977-4ed10ab3e349 run_task_manager([])
  
  
  • queued 5eac917f-008c-44b3-ad41-21b6ccdf77a9 run_task_manager([])
  
  
  • queued 8d48273a-6939-4e19-b4fa-43370d596139 run_task_manager([])
  
  
  • queued 93e991ae-89a7-4327-852d-6de975691029 cluster_node_heartbeat([])
  
  
  • queued 5d58b109-acb9-48f8-9b6b-c83030cb8fb9 run_task_manager([])
  
  
  • queued 1bb573ae-8150-442f-a603-303620cba119 run_task_manager([])
  
  
  • queued 0ce25c3c-1835-4a53-b3fd-46cf41bd6ec9 cluster_node_heartbeat([])
  
  
  • queued 3613d48c-0d6e-49bd-8e82-5261273f1f59 awx_k8s_reaper([])
  
  
  • queued 0c3ebda8-10f3-4d55-adef-9dfc1aece5a9 awx_k8s_reaper([])
  
  
  • queued 77ee146f-d797-43e2-90ab-286cbd4b19f9 cluster_node_heartbeat([])
  
  
  • queued 7e862914-7283-4f69-83e7-c7b93e7ce8a9 awx_k8s_reaper([])
  
  
  • queued 135df22f-046f-4102-ae1b-87972f5e9599 run_task_manager([])
  
  
  • queued 5bb43198-4f07-482a-bea6-02af8aeb9d99 cluster_node_heartbeat([])
  
  
  • queued e69bd4f0-884d-4005-8b09-5bc635b3c1d9 awx_periodic_scheduler([])
  
  
  • queued cab710b6-7f92-4e40-9dd1-ffda26e22979 run_task_manager([])
  
  
  • queued 1e0c5bc1-59af-417c-802d-c6581ee560b9 awx_periodic_scheduler([])
  
  
  • queued 2575aa5c-3743-4757-bdd7-800248de4609 cluster_node_heartbeat([])
  
  
  • queued 9c8ac029-5c50-4df7-98a8-fc8b7ffe1449 awx_isolated_heartbeat([])
  
  
  • queued 74e242d0-3549-42b1-8a18-b91cf5c0e5d9 run_task_manager([])
  
  
  • queued f234d3de-8e9b-4e08-924e-be83b5273bf9 run_task_manager([])
  
  
  • queued abca5c55-f7ef-4d9e-97e8-eda92c4e6799 cluster_node_heartbeat([])
  
  
  • queued 71c0c365-56ab-4a02-8ee0-23186408fca9 run_task_manager([])
  
  
  • queued e1296d8d-4537-4aca-a629-d3a6267b6b19 awx_k8s_reaper([])
  
  
  • queued b1743a84-771f-4919-9262-96f14e60e3c9 awx_k8s_reaper([])
  
  
  • queued d96f5ab3-ba61-4fd5-a6fa-c378005e5209 awx_periodic_scheduler([])
  
  
  • queued 8a3b48c5-c23e-4193-8e14-92f20f2c9079 awx_periodic_scheduler([])
  
  
  • queued 6e07ba0c-ee8c-4082-bdde-b8e211f4c9e9 run_task_manager([])
  
  
  • queued bbbd2ff7-1510-468a-91c4-0e83e1974c69 run_task_manager([])
  
  
  • queued 7a14d981-f004-4ffd-a353-369cd72e2b59 awx_periodic_scheduler([])
  
  
  • queued e86b6fdc-33b1-41fc-9cb3-55e9af592be9 cluster_node_heartbeat([])
  
  
  • queued dfb7e820-1eac-4119-9ecc-44608c19ec59 awx_k8s_reaper([])
  
  
  • queued a530a7ab-5967-4cb8-85c2-d5dc1a48c799 run_task_manager([])
  
  
  • queued 38b6096e-41a7-42a7-a348-7bb3721d5f79 awx_k8s_reaper([])
  
  
  • queued 974b8396-f76e-424c-a2fa-167460b6d169 run_task_manager([])
  
  
  • queued 93f2e653-dfee-4ef5-8b3c-46781834cb09 cluster_node_heartbeat([])
  
  
  • queued 6b677feb-7b50-4f36-9264-c9c4c7e302f9 awx_periodic_scheduler([])
  
  
  • queued e9882904-df4e-464a-b9dd-9a0999f451d9 run_task_manager([])
  
  
  • queued 20b313a2-e74c-413a-82a8-7bd6007e4f29 run_task_manager([])
  
  
  • queued ccff0389-0365-42de-8cda-61d756a84789 run_task_manager([])
  
  
  • queued 955ffbdc-77d5-4883-bc3f-e25da5419009 cluster_node_heartbeat([])
  
  
  • queued e373f4fc-3fe8-4878-8a0c-dc76abcc8c69 awx_periodic_scheduler([])
  
  
  • queued c57b2c0f-b22a-45e7-8003-ec9286451d59 gather_analytics([])
  
  
  • queued 7e47acd5-b4fc-485d-b7cd-cef0038001e9 cluster_node_heartbeat([])
  
  
  • queued 52f3c895-788f-448a-b82c-1963713da2d9 run_task_manager([])
  
  
  • queued 3bc2d22f-fb5c-4db4-88e8-478b48c0d6d9 awx_periodic_scheduler([])
  
  
  • queued d7adc519-f00d-4b17-983b-8829525378a9 awx_periodic_scheduler([])
  
  
  • queued 6e285c0e-b9ba-4f2a-bfca-c949a78502c9 run_task_manager([])
  
  
  • queued 82d9213a-5aae-4831-bd79-3778e3d2aef9 awx_periodic_scheduler([])
  
  
  • queued 0ec5c0f5-5c5d-4603-98aa-ac00fadbfe69 cluster_node_heartbeat([])
  
  
  • queued c0dfe0ca-bf46-4f8c-b506-1c99f0153159 awx_periodic_scheduler([])
  
  
  • queued 41da2ea8-3766-468e-b761-f19e04c39ba9 run_task_manager([])
  
  
  • queued 77be5651-6b14-481e-8d9c-ac175ab89639 awx_periodic_scheduler([])
  
  
  • queued 6b34f44b-f91f-407d-8773-caa9c56c4c59 awx_k8s_reaper([])
  
  
  • queued ad9df671-0401-4e94-a557-b1394e19ca89 run_task_manager([])
  
  
  • queued 0c67926f-89d0-4270-b14b-d7fba1608109 run_task_manager([])
  
  
  • queued 7b7d631f-dfc6-442c-9769-efbc788ca0a9 cluster_node_heartbeat([])
  
  
  • queued 24ea2d1d-dda1-4fa5-a4ab-1cf98cca52d9 cluster_node_heartbeat([])
  
  
  • queued 0518290a-90ff-4875-83a8-da741fb43049 run_task_manager([])
    
    . worker[pid:35581] sent=282 finished=0 qsize=282 rss=152.566MB
  
  
  • running 7d925bcc-ab79-4b05-b3d0-2f84d882ac65 run_task_manager([])
  
  
  • queued de3c6a61-f025-4d2d-bf1e-523418c897ab run_task_manager([])
  
  
  • queued aa031ba7-e815-4cc7-bf3e-2f15906abbbb awx_k8s_reaper([])
  
  
  • queued 81c87877-4f8f-4cd1-8cc8-8d7a75cfbb5b awx_k8s_reaper([])
  
  
  • queued 21ba060c-1c72-45c7-a1ca-b473434f968b awx_isolated_heartbeat([])
  
  
  • queued 4c19c71e-4ec8-4e62-ac8d-2457b31c71eb cluster_node_heartbeat([])
  
  
  • queued e740dbe9-0bdc-4809-a238-1dab2b05feab awx_periodic_scheduler([])
  
  
  • queued d8d3e1a2-98a6-4c85-9878-d9953ab821ab run_task_manager([])
  
  
  • queued 3321a3c3-15d9-4b71-81e3-0c7eeccb62fb awx_periodic_scheduler([])
  
  
  • queued 2b35614f-bda4-4e10-baf0-f2691278135b run_task_manager([])
  
  
  • queued ce37650d-7436-4662-ab3a-43aef7248f0b run_task_manager([])
  
  
  • queued 2a7b01fd-b74f-404c-8d68-7e061565045b run_task_manager([])
  
  
  • queued 1347c163-3e05-4b85-8b1f-a97e6b46c94b cluster_node_heartbeat([])
  
  
  • queued e2f9cadc-fad4-4b8c-b0d0-b4bdf4e03abb awx_k8s_reaper([])
  
  
  • queued ddf2c25b-2a41-42fb-af98-a38430625e0b run_task_manager([])
  
  
  • queued 1e1b30ae-0388-49cb-9657-76fb1268110b run_task_manager([])
  
  
  • queued 83080bce-e0f8-4e26-87db-29116b25fdbb awx_periodic_scheduler([])
  
  
  • queued 4cf0d7cd-8e6d-4f5a-a521-4c4e91b550ab awx_periodic_scheduler([])
  
  
  • queued a797732a-dd7b-448f-b397-d29f2b27a35b awx_k8s_reaper([])
  
  
  • queued 25d97d82-76fa-4d25-8737-6725a28cf8db awx_periodic_scheduler([])
  
  
  • queued c6ec1c71-d3d3-4b82-bd09-3eaf0c671e2b run_task_manager([])
  
  
  • queued dd6a3513-a898-4d30-80f4-afde172a187b awx_periodic_scheduler([])
  
  
  • queued cf782f3c-edd2-4818-818a-d7bce8b3b05b run_task_manager([])
  
  
  • queued dd8e6dac-3df2-4ac1-a1cf-6440ce51d49b awx_periodic_scheduler([])
  
  
  • queued 6d36ff93-b186-4bd7-91f1-0f6b72d4121b awx_periodic_scheduler([])
  
  
  • queued 567a99e5-e6b0-4ae4-bcc0-8b1a725a06bb awx_periodic_scheduler([])
  
  
  • queued 6c29deea-83ca-4df8-be44-7ab7fcdd7fdb cluster_node_heartbeat([])
  
  
  • queued 0362238c-8903-4ee3-94bb-f54abfc35d0b run_task_manager([])
  
  
  • queued 7282cda1-c82c-494c-a4f2-a31fcaf9343b run_task_manager([])
  
  
  • queued d31533f5-9431-4335-8f5a-7045452478ab run_task_manager([])
  
  
  • queued 2269cb88-6d2f-4f3b-ba50-8d08f7681dcb cluster_node_heartbeat([])
  
  
  • queued 80d32b5b-7058-438f-aa52-27fa54a47b9b run_task_manager([])
  
  
  • queued ec151762-27e9-42eb-96eb-15a1c9f941bb awx_k8s_reaper([])
  
  
  • queued 56ba980f-3f58-4790-ab71-14f0686ac17b run_task_manager([])
  
  
  • queued 7332b18e-0558-4f5f-a67f-a6bac533e6ab awx_periodic_scheduler([])
  
  
  • queued 7b3bc217-f9f1-45e9-ac76-c9a8d123bd1b run_task_manager([])
  
  
  • queued 46ac52e1-b796-4e90-b8b7-d43ee5358bdb gather_analytics([])
  
  
  • queued cb6276c4-34a5-4f32-a3eb-74b20a080c6b awx_periodic_scheduler([])
  
  
  • queued 3fe8ade1-9631-4e5c-b117-dc6d3334753b run_task_manager([])
  
  
  • queued 0174aa9a-04c5-4ae9-9172-c44cb2b0537b awx_k8s_reaper([])
  
  
  • queued e0d01854-b035-4514-9abd-b4d2a34d2b0b run_task_manager([])
  
  
  • queued 283ac678-c4f8-439a-be51-9c9a2d56bc8b run_task_manager([])
  
  
  • queued 9fec882a-5ffe-4920-a1e1-e00926af69fb awx_k8s_reaper([])
  
  
  • queued 7b8e6dd7-1c1c-4fe5-9183-4f524a69807b gather_analytics([])
  
  
  • queued 872f507c-e58c-4e5a-967f-9b1932e9d26b awx_periodic_scheduler([])
  
  
  • queued d87d60a9-dc1f-4e5e-9297-f72142c5e79b run_task_manager([])
  
  
  • queued 5620a849-70b6-4659-8056-9a436b11dc3b run_task_manager([])
  
  
  • queued bf9a5579-08a4-42be-97b3-73a5f1564d1b run_task_manager([])
  
  
  • queued b5925ae9-266d-4965-9e0e-19f82b9d945b run_task_manager([])
  
  
  • queued fa19a5e0-e1ff-4f60-871b-daed78c9561b cluster_node_heartbeat([])
  
  
  • queued c50ce42a-a482-4119-bc74-013e78062a6b run_task_manager([])
  
  
  • queued 4b1f8f5c-60ca-40d1-8376-634bf013492b run_task_manager([])
  
  
  • queued ab8c8a11-1046-4262-917e-5829a549b95b cluster_node_heartbeat([])
  
  
  • queued 11d05161-b2de-4d9e-b60f-66b22001a06b run_task_manager([])
  
  
  • queued 1031c3d6-aeb4-4bb9-a022-44f3c6a3de3b awx_periodic_scheduler([])
  
  
  • queued 76786621-0e49-4941-9a39-80971dc9817a awx_k8s_reaper([])
  
  
  • queued f784f3ed-fd64-4049-88d1-4741eae1113a awx_periodic_scheduler([])
  
  
  • queued 1fce2d9c-9e22-4fa1-bb97-5834905df2ca awx_periodic_scheduler([])
  
  
  • queued 954b1346-2d6d-42c4-baf2-521560b0dc2a run_task_manager([])
  
  
  • queued 5683f008-fa2c-4f70-978d-35f60d694d3a run_task_manager([])
  
  
  • queued f5c9dcb5-9165-4265-9abe-5770bd9edb0a cluster_node_heartbeat([])
  
  
  • queued aec708db-7dea-4964-94be-64d353ed53fa awx_k8s_reaper([])
  
  
  • queued 4d55d6fa-e54f-4b8b-a557-ba5b406d063a awx_periodic_scheduler([])
  
  
  • queued b5a7480c-82cf-42cf-95ac-74f81ef7e74a awx_periodic_scheduler([])
  
  
  • queued 0e670f38-310d-428e-ba89-eb3a99a41b1a awx_periodic_scheduler([])
  
  
  • queued 3f58f33b-a1c1-4810-b8ff-10bf35b26c9a gather_analytics([])
  
  
  • queued 8838b0d4-a335-4833-bedb-accd05f37f9a run_task_manager([])
  
  
  • queued 4eab1621-8cec-4f3e-b60b-b3f51867450a run_task_manager([])
  
  
  • queued 64a9892e-db12-4e25-a709-b852e83d474a awx_isolated_heartbeat([])
  
  
  • queued aa109867-1855-42c6-9829-136756db6bda run_task_manager([])
  
  
  • queued 0eab1565-00a6-4269-8c4e-d163319b078a run_task_manager([])
  
  
  • queued 654d7a8e-f3ac-4eb2-a271-3956d51daf6a run_task_manager([])
  
  
  • queued 7c479087-6271-4dc9-89b9-a9a85ecfafba awx_periodic_scheduler([])
  
  
  • queued 926d6c91-5577-4e6c-9748-d2c368f5aefa cluster_node_heartbeat([])
  
  
  • queued 35963bba-6876-453b-b6da-094d1145f95a awx_periodic_scheduler([])
  
  
  • queued 1a233c84-a24a-4311-a839-66849bd56d5a awx_periodic_scheduler([])
  
  
  • queued 3e7d3aa3-4439-4443-a502-3e3b25cb9eaa awx_k8s_reaper([])
  
  
  • queued 2fea3d76-00b7-4abc-b69d-626e4e40e24a awx_periodic_scheduler([])
  
  
  • queued 0fb59103-a57c-43e8-ad39-02fcb84e37ba run_task_manager([])
  
  
  • queued ffa0c43f-141a-4c5c-b424-a3255d5ed05a run_task_manager([])
  
  
  • queued e60e9f5a-5d67-4c64-bae8-fa48dd1ab06a awx_periodic_scheduler([])
  
  
  • queued 3e6e7c3d-22b9-4849-8099-02cae933de7a awx_periodic_scheduler([])
  
  
  • queued a038bdd9-b7b3-48ba-8352-aac1ca06ba2a run_task_manager([])
  
  
  • queued 620a122a-3718-4047-be74-ec938f7ae22a run_task_manager([])
  
  
  • queued c1796b7f-4a91-4acd-a213-fb193cfb677a awx_k8s_reaper([])
  
  
  • queued b140962f-b2b4-4fa9-8410-8bb47a79b1da awx_periodic_scheduler([])
  
  
  • queued c32d74a7-dfed-4be4-8fec-f40a811c36ea run_task_manager([])
  
  
  • queued 1c541023-ca82-4776-b245-38b9686e030a run_task_manager([])
  
  
  • queued 80b491ca-ddfd-4cb3-8926-2820fdf9303a run_task_manager([])
  
  
  • queued 5ee68d00-eea9-450a-bd80-2c3816ca697a awx_periodic_scheduler([])
  
  
  • queued c7163f95-72b7-4e40-8740-4f1b880adeea awx_periodic_scheduler([])
  
  
  • queued be282584-e2bf-44e4-b032-6a979ac9fd6a run_task_manager([])
  
  
  • queued df7aefa2-ec8a-4048-9933-de8088cfbb4a awx_periodic_scheduler([])
  
  
  • queued 9238bb17-b0e8-4c00-8d01-9a8e9eeb098a awx_k8s_reaper([])
  
  
  • queued ac94f29c-1005-4530-95ee-0e38538e790a run_task_manager([])
  
  
  • queued bc488607-7c1a-4051-90cf-8d3b49b747aa awx_periodic_scheduler([])
  
  
  • queued c805d2ac-0a6c-4ce0-ba1b-7400d115460a awx_k8s_reaper([])
  
  
  • queued f1e9ef22-ff56-4675-b936-6a0ab72d37ba run_task_manager([])
  
  
  • queued a93202f5-2438-45be-883e-649371bb3f9a awx_k8s_reaper([])
  
  
  • queued efffc597-5d4b-4a12-85fc-c36a44a9315a awx_k8s_reaper([])
  
  
  • queued 40bae419-681f-4e4c-8f14-14b81859f48a awx_periodic_scheduler([])
  
  
  • queued 4d9f9785-5fbc-4546-8f23-2723d706257a run_task_manager([])
  
  
  • queued 228cab8f-9206-482a-b550-677b216510da cluster_node_heartbeat([])
  
  
  • queued 92021b2c-c55d-428b-8796-84c4200d025a awx_periodic_scheduler([])
  
  
  • queued 074c7906-59f6-4b64-bea4-168d98d1be1a awx_periodic_scheduler([])
  
  
  • queued 06127894-454b-4fb5-8b00-4b0fe31654ba cluster_node_heartbeat([])
  
  
  • queued f3d30f57-e0b7-4dd3-b175-11e0615173fa awx_k8s_reaper([])
  
  
  • queued fe204875-2d0d-4721-987f-e6361ee4885a run_task_manager([])
  
  
  • queued f29ea8fd-a851-4c43-a898-bdbf0ab4407a run_task_manager([])
  
  
  • queued 10f6ec8a-3885-46e4-9021-f8b8dde8e23a cluster_node_heartbeat([])
  
  
  • queued 814603f6-d46e-422b-9611-f378e403c25a run_task_manager([])
  
  
  • queued 9d9a774c-a086-43e7-9a54-d5d8f5b383ea awx_periodic_scheduler([])
  
  
  • queued 4319e8dc-5e97-4e22-b3c0-0a5538ce9a1a awx_k8s_reaper([])
  
  
  • queued 6bcb79c3-646d-4d86-b2e3-c9a658b35c7a cluster_node_heartbeat([])
  
  
  • queued be6f908a-0acf-43d6-85b8-07b769ce1b3a run_task_manager([])
  
  
  • queued dec3cbfd-cb47-490b-ab68-bd4064cc72ca awx_k8s_reaper([])
  
  
  • queued 8f4e7911-656f-4cbb-abd7-df02cc26773a awx_periodic_scheduler([])
  
  
  • queued b8ce4ef0-aba1-4d29-9468-485bb720849a run_task_manager([])
  
  
  • queued a20aed1f-6357-43cb-9cf2-a20366fe73ea run_task_manager([])
  
  
  • queued ea8fae15-ca94-40ca-97ad-eda59342331a run_task_manager([])
  
  
  • queued d67d8cb5-0d10-4531-9edb-a5edd7f5495a run_task_manager([])
  
  
  • queued 3ad18faa-e22a-4ec6-947c-4598197f7dfa cluster_node_heartbeat([])
  
  
  • queued bcac3ead-9107-47e3-9e3e-e1ababab740a run_task_manager([])
  
  
  • queued 04053605-696a-4af1-a5e2-93b2ee4d338a run_task_manager([])
  
  
  • queued dc507daf-1e58-4052-be4f-6d4126ddf08a awx_periodic_scheduler([])
  
  
  • queued 165807ba-ccd9-4aa3-ac84-8ec888ed92da awx_periodic_scheduler([])
  
  
  • queued e7dda221-4378-4658-b0d4-92164164f6fa cluster_node_heartbeat([])
  
  
  • queued d0cbadf2-b352-4820-8824-88b6296a803a run_task_manager([])
  
  
  • queued 5cf50bd9-0b7c-433d-bb8a-06628e76c73a awx_periodic_scheduler([])
  
  
  • queued fcbd3eb2-e8a4-4b4e-8ea8-99b58e4ccafa run_task_manager([])
  
  
  • queued 87e0ebec-9ff1-4a17-a297-a32cfc9d2dda run_task_manager([])
  
  
  • queued 1af4c479-980c-4c7e-ae88-4497dba6ccda run_task_manager([])
  
  
  • queued df0228bd-1084-47cc-a85c-f5c20ae1f71a run_task_manager([])
  
  
  • queued 3a1b52b8-0405-4bcb-ac99-81077495cbca awx_k8s_reaper([])
  
  
  • queued 26f3bdbc-26ec-4ada-a4f7-ee050b31001a awx_periodic_scheduler([])
  
  
  • queued 07a32c5b-09f3-4118-bb9e-7cce66a1127a run_task_manager([])
  
  
  • queued 8956de72-f9dd-4ea2-bd32-c7f40838608a awx_periodic_scheduler([])
  
  
  • queued 44d2992a-0e60-4db4-9f50-c3164c939d3a awx_k8s_reaper([])
  
  
  • queued bd394557-d050-4b30-bbae-dffd990e95da awx_periodic_scheduler([])
  
  
  • queued 4edd82d0-0749-4948-b767-126919d7d73a cluster_node_heartbeat([])
  
  
  • queued f6f4a486-6d2d-4377-a4a6-2ebbfd6325ea awx_k8s_reaper([])
  
  
  • queued 5eba97e9-ea76-4cb0-ad2d-3aca61ad10ca run_task_manager([])
  
  
  • queued 62e7560a-73bb-4f1a-b134-2477f9bd654a awx_k8s_reaper([])
  
  
  • queued ba17ba69-9bcc-4cab-8ac4-4daf75e0268a run_task_manager([])
  
  
  • queued bcb0adae-4121-4ac8-80da-6de3b9e3a66a awx_k8s_reaper([])
  
  
  • queued fd34a84a-9eba-4036-844f-3d5f19ffa85a awx_periodic_scheduler([])
  
  
  • queued db4f8398-a85d-480e-8440-501e3a9fe3ba cluster_node_heartbeat([])
  
  
  • queued c8394f06-9de3-4b6e-b321-b6bc2e5370aa awx_k8s_reaper([])
  
  
  • queued 99d93d5f-2868-4d8e-8b6a-42fe5af51a9a run_task_manager([])
  
  
  • queued 7900f2ef-7aac-4427-b22f-3b2a2dc3d98a run_task_manager([])
  
  
  • queued c4688eac-becc-4881-b6ad-d59cd919617a cluster_node_heartbeat([])
  
  
  • queued ab1cce36-7853-4928-a7f5-5d28edb5880a run_task_manager([])
  
  
  • queued 48605d51-e53f-43cc-931d-8c2af445bdea run_task_manager([])
  
  
  • queued 71faf87e-1bbb-4fa5-bd52-59b5ef42f9ea run_task_manager([])
  
  
  • queued 6eec80e0-a6f9-4e1c-a024-3b85b79dc69a gather_analytics([])
  
  
  • queued 4a356bc8-952e-4dee-92a7-8987db1b47da run_task_manager([])
  
  
  • queued fe712d7e-c3d5-40a2-9cce-d169c0208b3a awx_periodic_scheduler([])
  
  
  • queued c15cec5a-0c52-428d-a586-a2bc5635e82a run_task_manager([])
  
  
  • queued bad45c4e-aff5-4e33-8567-9a819c5d5a9a awx_periodic_scheduler([])
  
  
  • queued 79a7248a-b287-441d-aa76-22203b75557a awx_k8s_reaper([])
  
  
  • queued ef3c25ac-2697-40fd-92fc-b881ef2c204a update_inventory_computed_fields([3])
  
  
  • queued 8f69abd3-1644-43d6-968d-5a89a2a3c94a run_task_manager([])
  
  
  • queued 6e810890-a406-4319-830c-2e84acc5b00a cluster_node_heartbeat([])
  
  
  • queued fb05ac95-92c2-4252-b69c-d3b4c8e5cefa awx_periodic_scheduler([])
  
  
  • queued 8f2cce3c-d293-4d32-a117-e68a5cd07b2a awx_periodic_scheduler([])
  
  
  • queued 46eadf11-9330-4d4b-a6c4-c9456c4998da awx_periodic_scheduler([])
  
  
  • queued a5b04aba-e474-4c99-8a49-a3bf50eefeda awx_periodic_scheduler([])
  
  
  • queued 201423b2-4204-4003-bdb0-b350383f289a awx_k8s_reaper([])
  
  
  • queued 8ccc62f3-a65e-4772-84d4-30cad07e097a gather_analytics([])
  
  
  • queued 4103c0be-3b1a-4428-b578-82c27638f3ea cluster_node_heartbeat([])
  
  
  • queued bd40f55d-091c-4c19-908a-3da1663c108a cluster_node_heartbeat([])
  
  
  • queued afa8bd72-58a0-4d8d-bb1d-61aca18ca51a run_task_manager([])
  
  
  • queued 79a85222-0728-4573-bf49-189828755e2a awx_periodic_scheduler([])
  
  
  • queued 5be5a0e6-27a7-4b18-91d2-3d7c1697727a run_task_manager([])
  
  
  • queued 9cd2e3f8-02c6-4db0-abfe-2e973156a49a awx_periodic_scheduler([])
  
  
  • queued 8afc929c-9dd2-4a1d-89a4-f272d94e1eaa awx_periodic_scheduler([])
  
  
  • queued 2491f0f5-a941-4208-abe7-03094f15209a awx_k8s_reaper([])
  
  
  • queued 0de1ec05-a1b8-4e72-a972-ff0a60e61bca awx_periodic_scheduler([])
  
  
  • queued 47bda24a-f514-474b-ab3a-a72e2ce1632a run_task_manager([])
  
  
  • queued eb5a93e8-976a-4968-b902-bf514b71f3ba awx_periodic_scheduler([])
  
  
  • queued d9cde1e5-f678-463e-a8a4-42c9a2ebf63a run_task_manager([])
  
  
  • queued 4e38b0d3-05bc-43e9-b41b-37d7fd7305fa run_task_manager([])
  
  
  • queued d1e10440-2d04-41cf-aa28-481299cfe98a awx_periodic_scheduler([])
  
  
  • queued 4c3eb364-23e5-4566-8b17-5ca74bd4c07a awx_periodic_scheduler([])
  
  
  • queued f8b2302e-54aa-4e40-b479-ac92c53304ca awx_periodic_scheduler([])
  
  
  • queued 8ad60d21-f32d-4abd-b7f7-5272849fd64a awx_periodic_scheduler([])
  
  
  • queued 479978de-5f01-436d-b278-1a979bac3d8a awx_periodic_scheduler([])
  
  
  • queued de6c6cdf-9e5d-41aa-b6a4-f0db4093e4da awx_periodic_scheduler([])
  
  
  • queued 3a9ac932-cdb6-4058-b1ed-19320ecc276a run_task_manager([])
  
  
  • queued bca38eed-a1b0-4a02-8700-a4293c0c162a gather_analytics([])
  
  
  • queued 144f36c8-f25b-42eb-9f1b-184ed850229a run_task_manager([])
  
  
  • queued 450e6ada-2c1e-4b96-955f-f2d2c84a3ffa cluster_node_heartbeat([])
  
  
  • queued 9e44e64b-92cc-4bc1-886f-6f201177e6ea awx_k8s_reaper([])
  
  
  • queued 0aa339cc-3def-4c60-958b-412728ebd7ba run_task_manager([])
  
  
  • queued 76b386c4-beda-4327-95c7-1069b2d45c7a awx_periodic_scheduler([])
  
  
  • queued a4f285b9-938f-472a-86d8-42c21baf5ffa awx_k8s_reaper([])
  
  
  • queued 6a132943-f343-4d7f-8e6f-08e58372afca cluster_node_heartbeat([])
  
  
  • queued a22c9b9a-39d6-4b45-a5dc-b9a2d7a443ba run_task_manager([])
  
  
  • queued 78339320-b37c-43e6-9513-a5f5534a2d4a awx_k8s_reaper([])
  
  
  • queued 12ab1a1e-1e0f-41eb-b03a-78bd37dd311a awx_periodic_scheduler([])
  
  
  • queued 02fb9e76-0a7e-4fc6-b0bb-a12d8fe3bf2a run_task_manager([])
  
  
  • queued bfd7e42e-8865-431e-8c1c-8363df640d1a run_task_manager([])
  
  
  • queued 8c0a0a4c-7025-4b4e-926b-a6bf352df99a awx_isolated_heartbeat([])
  
  
  • queued f1e2da75-bc11-41e4-b830-650a64d61a1a awx_periodic_scheduler([])
  
  
  • queued 39de1119-18da-4ac5-8c02-769bf6dd1eda run_task_manager([])
  
  
  • queued 241b6e47-6625-4dc7-b1b0-31ab784846aa awx_periodic_scheduler([])
  
  
  • queued 62f4b0d8-64af-4f9f-81e4-4c428262fb4a gather_analytics([])
  
  
  • queued c918ccb2-f0ee-48d2-bb5e-b4c71ba50dca run_task_manager([])
  
  
  • queued f9d3c8bc-55e6-4b6e-abdc-1e9582b99caa run_task_manager([])
  
  
  • queued f45f9aed-1346-4ecd-a722-38fb2656c58a awx_periodic_scheduler([])
  
  
  • queued d970650e-fdf6-43a7-b64a-495d8146e98a cluster_node_heartbeat([])
  
  
  • queued 7d721efc-f12d-45ef-846a-f731445cecea run_task_manager([])
  
  
  • queued 82deaae0-372b-4bb8-a121-114e3dd1d3ba awx_periodic_scheduler([])
  
  
  • queued 7b727506-c570-428c-84db-0ff0d67da0ea awx_k8s_reaper([])
  
  
  • queued ae64036c-604f-440f-95e8-4e2b03e4b6ba awx_periodic_scheduler([])
  
  
  • queued 648f301e-0c8b-422e-b116-c045799d6cea run_task_manager([])
  
  
  • queued 93e2cd2a-8c18-4402-b88c-d92e568afcca awx_periodic_scheduler([])
  
  
  • queued b675ca5f-a604-426e-af09-333ed1bbcd4a cluster_node_heartbeat([])
  
  
  • queued e0bd8a0b-bb4b-469a-ab34-ea47197c153a run_task_manager([])
  
  
  • queued 5dbd97d4-4187-4b1f-a55f-7524d81a323a cluster_node_heartbeat([])
  
  
  • queued a8507ee4-f979-438e-87de-583d314dccca awx_k8s_reaper([])
  
  
  • queued 5bb149dd-a73f-4d37-8864-bf2ce408535a awx_periodic_scheduler([])
  
  
  • queued df3df963-1390-4bbb-84a1-4e7205fcbf4a run_task_manager([])
  
  
  • queued de43c51a-8096-47ae-9976-a27a8f89400a run_task_manager([])
  
  
  • queued ce949833-b187-4be2-b3b7-520781f5b80a awx_periodic_scheduler([])
  
  
  • queued f755f659-b50b-4215-a788-991d2561c7aa cluster_node_heartbeat([])
  
  
  • queued 0bb22357-a124-4697-b50a-1299c79d9c6a run_task_manager([])
  
  
  • queued c25abca1-5cca-4f01-a265-504d3a94f56a cluster_node_heartbeat([])
  
  
  • queued 15cdfb23-34df-47fd-826c-93e34f26856a run_task_manager([])
  
  
  • queued 731fcebf-b0aa-4724-8209-771eb77ade3a run_task_manager([])
  
  
  • queued 98a6b7e8-a54d-4d18-bc2c-3eec24fa0caa awx_periodic_scheduler([])
  
  
  • queued c5c5bf8c-f925-4a19-9c7e-ce69c1a7ecaa awx_k8s_reaper([])
  
  
  • queued c8cf2811-84cc-4df0-937b-5b3ebbbc0fea run_task_manager([])
  
  
  • queued b1cfa36b-aeb6-48c1-bf6c-9f79a2fedb0a run_task_manager([])
  
  
  • queued 311b2db3-3e85-4a3d-bb2f-13e575d7a05a awx_isolated_heartbeat([])
  
  
  • queued 4beb2e43-4f14-4a4a-8989-c12ddd57194a cluster_node_heartbeat([])
  
  
  • queued 62d69863-eeff-494a-9e73-a2c681a596ea cluster_node_heartbeat([])
  
  
  • queued fb84dcf2-cddd-4b08-bf4e-d3365ab4e5ca awx_periodic_scheduler([])
  
  
  • queued 30028515-2aeb-47e1-ba9a-900046fe450a run_task_manager([])
  
  
  • queued 9e6145b5-1eed-4bc7-b217-601f6a60d05a run_task_manager([])
  
  
  • queued 2f35bbe9-163a-4e04-bb2e-1e335d1f441a awx_periodic_scheduler([])
  
  
  • queued c30955ce-ef25-45c9-a64c-e70d4d7e6e9a run_task_manager([])
  
  
  • queued c9d35e86-0439-4b75-9423-2c5dae20a37a cluster_node_heartbeat([])
  
  
  • queued 51455a7c-c292-4443-a2cc-1bad14824e1a awx_periodic_scheduler([])
  
  
  • queued 1a5f909c-a0b0-4413-ba25-c9a16069bd0a run_task_manager([])
  
  
  • queued bfd2654e-3783-4120-b922-d8fc20985c8a run_task_manager([])
  
  
  • queued 18401fb4-e6e3-4c9f-818b-869a4d69c3fa awx_periodic_scheduler([])
  
  
  • queued 91d4c4a1-db29-45a2-aec4-02da6175ef6a awx_k8s_reaper([])
  
  
  • queued 3815b439-65b0-43a0-9d75-8a5dbf6f4e1a run_task_manager([])
  
  
  • queued ae1f5203-c56e-480c-8218-99e74166bdea awx_k8s_reaper([])
  
  
  • queued 6f24e351-4b24-43c9-b29f-dc19387fdd4a awx_periodic_scheduler([])
  
  
  • queued 555d0a44-bb9a-48e0-9d8f-8ea0af9935da awx_periodic_scheduler([])
  
  
  • queued 5f8ca061-2acb-42e6-b38e-75196d7b70ea awx_periodic_scheduler([])
  
  
  • queued 74eafc6a-c247-4af6-99e0-1f7ce78b5f9a awx_periodic_scheduler([])
  
  
  • queued abae9770-196a-49df-a31c-6a4e54caaeea run_task_manager([])
  
  
  • queued f06b4998-49cf-4369-b7c2-ab27d5ca9f4a awx_k8s_reaper([])
  
  
  • queued e69eae9f-b6ee-4478-abaa-e013b331553a awx_periodic_scheduler([])
  
  
  • queued 76ef1205-6642-41f7-891b-9f61320d646a awx_periodic_scheduler([])
  
  
  • queued bc41684b-087b-44db-9ec4-95223a8093ea run_task_manager([])
  
  
  • queued 02ab3d4e-0685-4526-87f0-d51a0d71b69a run_task_manager([])
  
  
  • queued c5a7a660-1598-4788-8b90-11df2b756dba awx_periodic_scheduler([])
  
  
  • queued 82f7efb6-56c9-43ce-a89b-6bfcd4be593a awx_isolated_heartbeat([])
  
  
  • queued d4e19f58-af29-4de5-a945-1f01834c975a run_task_manager([])
  
  
  • queued a61e608b-bd4b-4057-90ff-b9a7145fe03a awx_periodic_scheduler([])
  
  
  • queued 8259095a-4f7e-4d66-ad90-4de6f705e66a run_task_manager([])
  
  
  • queued 3f87e139-09b7-40f7-9cf4-7135d1690aba awx_periodic_scheduler([])
  
  
  • queued da31ece7-c028-4905-be55-2d6094ea237a run_task_manager([])
  
  
  • queued a9099f61-07d8-41ec-b498-004b2562a22a run_task_manager([])
  
  
  • queued 0935ae35-5f1f-4c68-b550-d379bf26368a awx_periodic_scheduler([])
  
  
  • queued 9778b5a2-ab99-4a48-a22f-32a703549b7a awx_periodic_scheduler([])
  
  
  • queued 27060d0e-0ece-4bad-b789-a7b4f7c44f0a run_task_manager([])
  
  
  • queued ce2211fe-af8d-4a37-83fc-50383accadea run_task_manager([])
  
  
  • queued c764fe2b-be71-4922-8b45-9267977b197a awx_periodic_scheduler([])
  
  
  • queued 2cfb8bf8-8e56-47d1-9476-8e99036387ba run_task_manager([])
  
  
  • queued 29c0f3da-36e2-4aba-aca8-64560258289a awx_k8s_reaper([])
  
  
  • queued 78078f7b-b684-4a25-902d-b9116af5dbfa awx_periodic_scheduler([])
  
  
  • queued 2ca90a9b-b3b2-41ba-b40b-480915497a5a run_task_manager([])
  
  
  • queued f3fa3663-49a9-4b58-8550-a6a03721934a cluster_node_heartbeat([])
  
  
  • queued 30d81d2f-5699-482a-b77f-8db22553976a cluster_node_heartbeat([])
  
  
  • queued 1ff33e92-47af-4a53-8ac9-88a7dfbfa4ea cluster_node_heartbeat([])
  
  
  • queued ddc589ec-c2b4-4126-9e5a-3769a56da54a awx_k8s_reaper([])
  
  
  • queued 1f44d1d9-5c4a-4244-8401-d047460875ca run_task_manager([])
    
    . worker[pid:35869] sent=248 finished=0 qsize=248 rss=152.492MB
  
  
  • running 7aa25dc2-d44c-4c71-9256-b9f9f701a850 awx_k8s_reaper([])
  
  
  • queued 012511f4-4b32-4f6a-9909-5408285942bc awx_k8s_reaper([])
  
  
  • queued dfd0dae0-6c07-4ba2-bb50-63de1fa7959c run_task_manager([])
  
  
  • queued e4b43a97-577e-49fa-afe0-6c48e7f1c08c run_task_manager([])
  
  
  • queued 15301a3d-1583-47cf-a742-df2ad606640c run_task_manager([])
  
  
  • queued 1e928e14-bad1-453a-909c-d149cb7efadc cluster_node_heartbeat([])
  
  
  • queued 494c0987-c7df-4c55-8b86-76864b8ae2ec run_task_manager([])
  
  
  • queued 9ab3dc8d-feda-47ff-a35b-00f94970b3fc run_task_manager([])
  
  
  • queued 1b67fee4-2491-48c2-a18d-f5651a0c06dc cluster_node_heartbeat([])
  
  
  • queued fa7e9a82-0f1b-4e23-a588-29c9615f338c cluster_node_heartbeat([])
  
  
  • queued 8f149efd-c3eb-4437-9abd-783ffd14af8c awx_k8s_reaper([])
  
  
  • queued a1d5c3ce-4837-4ca6-a362-8d4b00f962bc awx_periodic_scheduler([])
  
  
  • queued c37add88-dd8b-4416-b0d3-03257eb14c9c run_task_manager([])
  
  
  • queued 0b63b0c7-4ef0-4fb7-a771-e2ba53d7b80c awx_periodic_scheduler([])
  
  
  • queued bab56487-0f4d-4302-962c-5da32f8cff9c cluster_node_heartbeat([])
  
  
  • queued 749882a5-550d-46c7-b22b-2754c1cbf6dc run_task_manager([])
  
  
  • queued 5cfcfb2c-73df-44f5-b0d0-e0ba8f9f4eac awx_periodic_scheduler([])
  
  
  • queued 6a050501-6efa-435e-8007-e5df20e20c4c run_task_manager([])
  
  
  • queued 67129aee-3819-40f9-bcf5-d07fe84ca47c gather_analytics([])
  
  
  • queued 8a26db29-81b3-4a49-80fb-4f068b067a2c run_task_manager([])
  
  
  • queued 60aafdfb-d561-44b6-a76b-5c3b97a3aaac run_task_manager([])
  
  
  • queued 6647922a-b467-49d2-b9aa-ca10095b996c run_task_manager([])
  
  
  • queued df1b3248-d5f6-4ea4-bc44-8012accff32c gather_analytics([])
  
  
  • queued 84d8420a-2094-4ab9-89a8-ee9f43bd9a4c awx_periodic_scheduler([])
  
  
  • queued f0763dd3-8af1-4738-973c-fb2d5c37aeac run_task_manager([])
  
  
  • queued dcbbf74e-1cbc-4bd4-b9fc-723fca4426cc run_task_manager([])
  
  
  • queued 10e6ef6d-a3fb-4d74-af00-2fc1f126f81c cluster_node_heartbeat([])
  
  
  • queued 4f444ffc-2315-46c8-a6a6-dcbe7426658c awx_periodic_scheduler([])
  
  
  • queued 6c4e5b8d-78b1-44b5-aacd-8e0e9dc020dc awx_k8s_reaper([])
  
  
  • queued 7652293b-caec-48aa-a15e-ce36d5b5730c run_task_manager([])
  
  
  • queued ac886a50-f01e-4238-9d4f-e52cacbd7abc awx_periodic_scheduler([])
  
  
  • queued 0664f27c-d266-41f1-92db-45bb23f1ab6c awx_periodic_scheduler([])
  
  
  • queued d52a5982-3712-4744-a446-f92e1473630c run_task_manager([])
  
  
  • queued d92f5728-647b-460b-b081-4e8987d054cc run_task_manager([])
  
  
  • queued f562f21d-133a-4855-81aa-99a9a58b484b run_task_manager([])
  
  
  • queued b2da29ab-4786-4e47-a3d3-2a0bb60b70fb awx_k8s_reaper([])
  
  
  • queued 31f3ee7b-0e9c-4d48-abde-595ab4c2f79b awx_periodic_scheduler([])
  
  
  • queued 091c47eb-1b3e-4fa5-87c3-9451dc04654b awx_k8s_reaper([])
  
  
  • queued d87be976-0dbe-41ee-af73-da4bd6de3e7b awx_k8s_reaper([])
  
  
  • queued 8d3faf80-ac5b-4803-af33-948413514aeb run_task_manager([])
  
  
  • queued 21701435-cb0c-4e12-92bf-2df804fd116b awx_periodic_scheduler([])
  
  
  • queued 94298915-f69e-49d2-aa36-e2f78d31e22b awx_k8s_reaper([])
  
  
  • queued afe71dce-f137-47fd-9f82-35cc2084d73b run_task_manager([])
  
  
  • queued e24485c3-2391-45b9-8d0b-da6d2f43c5db awx_k8s_reaper([])
  
  
  • queued e61c4e83-a2bf-49bd-b8e5-364047dc8d4b awx_periodic_scheduler([])
  
  
  • queued 248264b3-b0c6-4d83-9be1-e2cce2dde8db run_task_manager([])
  
  
  • queued 14e85ecc-c77b-4f86-9b70-2847a0c5207b run_task_manager([])
  
  
  • queued aca800c5-71b3-4854-9720-1dde3bc4a23b awx_k8s_reaper([])
  
  
  • queued cd1526b5-5a33-4cba-97bc-6f265078ec2b cluster_node_heartbeat([])
  
  
  • queued 03ecf77e-87dd-4237-b294-6e732106b95b awx_periodic_scheduler([])
  
  
  • queued 4af7a94d-c5a3-4773-8bff-e42209dd1c2b run_task_manager([])
  
  
  • queued c64d5ff1-5eb3-421c-856f-af881045092b awx_periodic_scheduler([])
  
  
  • queued 45ad851a-e376-4511-a155-81c1083ae40b cluster_node_heartbeat([])
  
  
  • queued b969f7ac-e05a-49b1-b471-83b77d77794b awx_periodic_scheduler([])
  
  
  • queued 0c769645-8201-484c-a12a-c24eb4222ffb awx_k8s_reaper([])
  
  
  • queued c6be6990-e280-4aa5-8ff8-ca557812585b awx_periodic_scheduler([])
  
  
  • queued 4aaf3c8b-1694-4571-bb83-82fe44a75c0b awx_k8s_reaper([])
  
  
  • queued d97df8f3-f1c6-4bd7-a9f5-16774683b91b run_task_manager([])
  
  
  • queued b3344a0f-c690-4215-b5f4-63662901e7bb run_task_manager([])
  
  
  • queued f7afcbc4-1808-403b-97c6-4c933e9ae06b gather_analytics([])
  
  
  • queued da514dd1-896f-478e-9a85-9733449201eb awx_periodic_scheduler([])
  
  
  • queued 302336b8-2640-463b-8cfa-aa159e1e09eb awx_k8s_reaper([])
  
  
  • queued 9b28d6dc-b3f6-4961-8e27-62ee524acffb cluster_node_heartbeat([])
  
  
  • queued 60d7d236-89ad-4885-af1c-c0cb8342d1fb run_task_manager([])
  
  
  • queued cf56a5cf-6221-49c3-b393-1f8c8116e21b run_task_manager([])
  
  
  • queued f8561b56-b42e-43c8-8b03-341e11c2b76b awx_k8s_reaper([])
  
  
  • queued 46cd2ee5-6350-4417-9db3-f8cdabc0a84b run_task_manager([])
  
  
  • queued 8e532a9d-67e2-4563-9251-187dd41cca5b run_task_manager([])
  
  
  • queued ebcdf4c3-5197-4a17-b825-7d3a0d80506b run_task_manager([])
  
  
  • queued a87bfdeb-b0ec-49b7-9107-297e839794db awx_k8s_reaper([])
  
  
  • queued 86ef4939-9665-4afd-8cd9-6a89a5f169cb run_task_manager([])
  
  
  • queued c8a8ed8d-8c60-423f-8583-0ac3fdbba86b cluster_node_heartbeat([])
  
  
  • queued e835feff-ec33-4d76-93d7-4f21ed7dc32b awx_periodic_scheduler([])
  
  
  • queued 586f814f-36f1-4109-8417-f7611d6c421b awx_k8s_reaper([])
  
  
  • queued 04eb6402-a3b2-4dec-a162-becb0a670d6b awx_k8s_reaper([])
  
  
  • queued 5f6a000c-f277-434c-8ead-1cd4ccb5a84b awx_periodic_scheduler([])
  
  
  • queued ce919339-699f-47b1-8905-6268e790049b cluster_node_heartbeat([])
  
  
  • queued 049454a0-1c01-43e2-aae0-7f1bb2f5438b run_task_manager([])
  
  
  • queued 06ec1a06-79df-4836-8117-384d0c99997b awx_periodic_scheduler([])
  
  
  • queued 2b86f578-05a7-4fa2-b34c-c1a5e8d3747b awx_k8s_reaper([])
  
  
  • queued 631c6456-9cd7-46b2-abe9-120c8303045b run_task_manager([])
  
  
  • queued 23e46dbb-c02c-4492-81c1-c5f37ca5cedb awx_k8s_reaper([])
  
  
  • queued d8683895-a44e-4936-9ba8-645886b5d83b cluster_node_heartbeat([])
  
  
  • queued 12832f95-038e-4c6d-bdab-3ec6a27df0cb awx_k8s_reaper([])
  
  
  • queued 424b5a82-5ada-4969-ab09-44ff3a9cd43b run_task_manager([])
  
  
  • queued 13af4d59-839c-4401-a78a-9b1cb5d6686b run_task_manager([])
  
  
  • queued 62748f0a-29d1-44c6-959e-1777a0496c6b run_task_manager([])
  
  
  • queued 8e2fbdef-4547-45c1-905b-fbde9e93930b awx_periodic_scheduler([])
  
  
  • queued 4ce61e6a-f78b-467b-90ae-cfd90d7351cb run_task_manager([])
  
  
  • queued a8a878c8-0213-4657-bc46-98d20bdc2cfb awx_periodic_scheduler([])
  
  
  • queued e7bfcd03-89bc-438b-a47d-49a112c21a7b run_task_manager([])
  
  
  • queued 5c273a4d-d83d-4e53-89f6-3c1276a49b7b run_task_manager([])
  
  
  • queued f923ce81-3e03-4cec-96dc-23c2ad55fceb awx_k8s_reaper([])
  
  
  • queued 6027b4e1-5acc-44d8-aa6a-f51228b75d6b run_task_manager([])
  
  
  • queued 845b025c-903d-4034-8cf8-1d6b9de8eefb awx_k8s_reaper([])
  
  
  • queued ccedc961-670a-45c8-b4d5-bfca65d5936b awx_periodic_scheduler([])
  
  
  • queued 95cd0fba-acf9-4877-aa0d-419e17ac718b run_task_manager([])
  
  
  • queued ee040888-de14-412f-94fa-9384e74e143b awx_periodic_scheduler([])
  
  
  • queued c065b895-669f-4150-a23b-d77f1345abfb awx_periodic_scheduler([])
  
  
  • queued 89274d66-9994-4eec-a663-1435e99c570b awx_periodic_scheduler([])
  
  
  • queued 30c922e6-8360-41b8-9a48-0511bf7a180b cluster_node_heartbeat([])
  
  
  • queued 343179ae-f805-4b4f-8568-edd1f4ad97ab run_task_manager([])
  
  
  • queued 8df0050e-ab99-4b77-ac56-82cc9b6bddbb awx_k8s_reaper([])
  
  
  • queued 9156bf19-047c-4358-8bbc-e90b409c8beb run_task_manager([])
  
  
  • queued b0321be7-94b1-46a2-a6da-23fbab1c12bb run_task_manager([])
  
  
  • queued ac12cf6f-9e0a-44be-8226-818652cf5e6b awx_periodic_scheduler([])
  
  
  • queued ed2fd2f3-2227-4e31-94ff-ee8d7cbe6d2b run_task_manager([])
  
  
  • queued decc7ab6-7909-477a-aec5-4e144bdc4cbb run_task_manager([])
  
  
  • queued c6b2712b-a5b9-40b1-98d6-b6b49ac77d5b run_task_manager([])
  
  
  • queued 1dbe0a80-d705-4984-8da7-8e56f1b67ceb run_task_manager([])
  
  
  • queued 23262768-ecbe-418a-bbc4-106747db8b8b awx_periodic_scheduler([])
  
  
  • queued 04222a96-3a00-414d-ad25-d8f82391e42b gather_analytics([])
  
  
  • queued 4e7341f2-5cc7-42eb-be9c-5d09c33bdc0b cluster_node_heartbeat([])
  
  
  • queued 497d2e3d-f7ec-4c80-a80a-0f735b42435b run_task_manager([])
  
  
  • queued 42ab4081-db32-44e9-b261-d64e8e4b4bab awx_periodic_scheduler([])
  
  
  • queued f58e3b67-f921-4eaa-ad1c-9fd15877eaab awx_periodic_scheduler([])
  
  
  • queued 7edcf889-9cf2-4912-98aa-ef07661c2c7b run_task_manager([])
  
  
  • queued 1f3f4128-c49d-43bf-89bb-589ba92dbc0b run_task_manager([])
  
  
  • queued e28f85c6-e843-4022-af15-1b5560421bfb awx_periodic_scheduler([])
  
  
  • queued 1b0a44d9-579a-4221-925b-bbc6a1dfd67b run_task_manager([])
  
  
  • queued 812d8453-69e4-4b52-81a3-080e5b5fc99b run_task_manager([])
  
  
  • queued 6e2b6db4-42b1-4dcd-bb95-82b43164836b cluster_node_heartbeat([])
  
  
  • queued 7bad9c23-4e9c-4e7b-a11b-3f29edc31b1b run_task_manager([])
  
  
  • queued 447da0d0-ca42-4ed5-bad9-033e25efb45b awx_periodic_scheduler([])
  
  
  • queued b6174efe-59fe-4b04-bcc6-d066219ad36b awx_periodic_scheduler([])
  
  
  • queued 8692d268-ead5-413e-987a-6b2ca3043e7b awx_periodic_scheduler([])
  
  
  • queued b9df2283-bce2-4bc2-bb6c-f17d2f3efe4b run_task_manager([])
  
  
  • queued 663b7e76-870e-46d9-bc14-015762264d4b run_task_manager([])
  
  
  • queued 4b9177f2-a820-4219-bf13-8d22291f159b run_task_manager([])
  
  
  • queued 320bcb9c-ccbc-4d4f-88e5-56b3d47afc0b awx_periodic_scheduler([])
  
  
  • queued 38b4f33d-2b3b-4445-97f6-bad72691a45b awx_periodic_scheduler([])
  
  
  • queued 911c5e10-0dbe-4bdd-aeb7-a7229bb0599b awx_k8s_reaper([])
  
  
  • queued 883ffaba-7693-43c8-a78c-2bb99962a3db awx_k8s_reaper([])
  
  
  • queued e4067ad1-821e-4e57-a326-290bab93170b awx_periodic_scheduler([])
  
  
  • queued e0d44cd5-dd24-4628-b45c-a62383c6523b cluster_node_heartbeat([])
  
  
  • queued 98b0ca78-65c8-4ec4-ab3f-8aa0e349dccb run_task_manager([])
  
  
  • queued 1d17ca80-3e2b-4632-8f07-634f1291790b run_task_manager([])
  
  
  • queued 2cfa95d4-694f-423a-9604-20f49195a1bb awx_periodic_scheduler([])
  
  
  • queued 54d5d164-b1fb-4d0a-80b5-70376f1ede4b awx_periodic_scheduler([])
  
  
  • queued ebb6042a-ae16-4f07-9622-1c02dbbb081b awx_periodic_scheduler([])
  
  
  • queued c369ef30-7d75-4c90-aaff-9115e7af107b awx_k8s_reaper([])
  
  
  • queued ecc59542-c2c9-4242-b29d-93f0cb23d6ab awx_k8s_reaper([])
  
  
  • queued 4b557345-1c0a-4177-8f47-77a4e9f5a7bb run_task_manager([])
  
  
  • queued b7d01ce3-26e4-4e0f-a870-43040be36b8b awx_k8s_reaper([])
  
  
  • queued 0305d905-69e2-444a-93a4-b8c149e6cbcb awx_periodic_scheduler([])
  
  
  • queued 3267c87e-dcdc-4365-ab1e-bfd158476e7b awx_periodic_scheduler([])
  
  
  • queued 6c0c32df-f6f5-461f-bc23-1503464d901b awx_k8s_reaper([])
  
  
  • queued 4dad1ef7-c938-4f24-ba92-c3a3de2882db run_task_manager([])
  
  
  • queued 08a87adc-1828-4cc2-9c5d-baa97bf7909b run_task_manager([])
  
  
  • queued 526cae99-e079-43d6-b5a4-0cd1cefe2dfb awx_periodic_scheduler([])
  
  
  • queued 87049194-85c1-408d-952f-769bfa8cd72b cluster_node_heartbeat([])
  
  
  • queued c385c256-0ce4-4177-bd67-e981b56ea25b run_task_manager([])
  
  
  • queued 4f2d5422-e71b-45fb-af1f-66fea7b504fb awx_periodic_scheduler([])
  
  
  • queued 2d2b4e78-10c2-4e1e-ad98-52eb1aa11b4b cluster_node_heartbeat([])
  
  
  • queued 100a8013-1a4f-4246-a7a0-99d923baa2fb awx_periodic_scheduler([])
  
  
  • queued d6a52928-65da-46bd-bebe-7148306cc62b run_task_manager([])
  
  
  • queued 6b7f7ed2-6d4e-45d9-b61f-39988ab5442b cluster_node_heartbeat([])
  
  
  • queued f87baebe-12d5-413d-943e-c9e2d752c6ab run_task_manager([])
  
  
  • queued 5f21167f-1d22-4f0f-9a19-6be6e1d802bb awx_periodic_scheduler([])
  
  
  • queued 4b2d1f8c-9595-4d27-a936-508c20ddff3b cluster_node_heartbeat([])
  
  
  • queued c4f5ea4e-7046-457a-86a6-cfca75c17a2b awx_k8s_reaper([])
  
  
  • queued 97b2b03d-a251-49b5-9ead-666b3ad444db cluster_node_heartbeat([])
  
  
  • queued b3923bd9-6b8d-4440-9ee1-469286186beb awx_periodic_scheduler([])
  
  
  • queued d990b133-d810-4b69-aa72-8eae3405e9db awx_periodic_scheduler([])
  
  
  • queued c5536b7b-0ded-4c54-8c1e-1ab6888821bb run_task_manager([])
  
  
  • queued 92838399-0abd-4a5d-9982-217e2cd4991b awx_k8s_reaper([])
  
  
  • queued 4dcd8981-47c5-4e18-bffa-b1d7546c989b run_task_manager([])
  
  
  • queued 66e729bc-e939-40c8-b130-79349ce3732b run_task_manager([])
  
  
  • queued 22e04e1b-c494-4abd-a0b2-23453fba43bb run_task_manager([])
  
  
  • queued d60232d7-6331-49f5-b8cb-39db2f73d04b awx_periodic_scheduler([])
  
  
  • queued 781ae78f-fb46-465f-842d-9ca081d0f53b run_task_manager([])
  
  
  • queued b5f6baf5-7c48-4d76-9cef-52f1c45c388b run_task_manager([])
  
  
  • queued 890f2fe3-01b1-4824-8417-8118cdfe852b cluster_node_heartbeat([])
  
  
  • queued a13efc30-65cb-4db6-99f6-e92ecd9317ab awx_periodic_scheduler([])
  
  
  • queued 7976cdbe-4128-4308-bffc-ff665e99bb1b run_task_manager([])
  
  
  • queued b4e85116-866f-4078-9aac-76918ca9944b run_task_manager([])
  
  
  • queued 08ae25c0-e839-4844-9074-22af6e172beb cluster_node_heartbeat(*[])
  
  

strace: attach: ptrace(PTRACE_ATTACH, 35212): Operation not permitted

[ryanpetrello]

@blaisemichel,

I've got a PR which I suspect will resolve this issue:

https://github.com/ansible/awx/pull/6093

@xTrekStorex,

Any updates on your end after a weekend of running periodic jobs?

[xTrekStorex]

[...]
@xTrekStorex,

Any updates on your end after a weekend of running periodic jobs?

All good. Not a single hung job.

[fischerdr]

All, is there a way to monitor the job stuck issue and restart the stuck task container?

[wbieniek]

You can monitor with API
https://"tower host URL"/api/v2/unified_jobs/?status=pending
not sure how to find out which pod has stucked container

[xTrekStorex]

All, is there a way to monitor the job stuck issue and restart the stuck task container?

@fischerdr Instead of an alert handler based workaround you could either apply the above mentioned patch manually like I did or downgrade to AWX 9.1.1 if that's an option for you until the fixed release becomes available

[ryanpetrello]

My recommendation would be what @xTrekStorex said - either apply the patch directly (it's just merged into devel a short while ago), or wait for the next release of AWX (I'm planning to cut a new minor release at some point in the next week with this fix).

I don't suspect that detecting and working around this bug is worth the effort, personally.

[fischerdr]

So 9.1.1 is ok to deploy, till new version is out?

[ryanpetrello]

@fischerdr to the best of my knowledge, this was a regression introduced in 9.2.0, and I've heard several people report that 9.1.1 doesn't suffer from this bug.

[kakawait]

I have the same exact issue.

What I did to patch my container:

1. Create folder ./patches/9.2.0 and add files from https://github.com/ansible/awx/pull/6093/files
2. Mount volume inside container using docker-compose volumes syntax
3. Override command to hook my how script before launching service

# Patch awx 9.2.0
awx_version="$(awx-manage --version)"
echo "current awx version is: ${awx_version}"
if [[ "${awx_version}" == "9.2.0" ]]; then
  echo "Patching awx because current version == 9.2.0"
  # patch
  cp -f /patches/9.2.0/awx/main/dispatch/periodic.py /var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/dispatch/
  cp -f /patches/9.2.0/awx/main/management/commands/run_dispatcher.py /var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/management/commands/
fi

I just patched today, I'll see if it works with time

---

Edit Since I've patched I don't have problem anymore

[xTrekStorex]

Just a little more feedback:
My awx_task container was recreated last night and the issue returned again shortly after. Patching the (new) container again fixes it - again.
I could build a fixed image or modify the image command like @kakawait did to patch it persistently but since a fixed release is coming I just do the following (in case someone just wants to try the patch without any extra volumes or command overrides):

docker exec -it awx_task bash
yum install -y wget
wget https://raw.githubusercontent.com/ryanpetrello/awx/5364e7839724026ad4ae49120c4b38dd8ab24326/awx/main/management/commands/run_dispatcher.py -O /var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/management/commands/run_dispatcher.py
wget https://raw.githubusercontent.com/ryanpetrello/awx/5364e7839724026ad4ae49120c4b38dd8ab24326/awx/main/dispatch/periodic.py -O /var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/dispatch/periodic.py
exit
docker kill awx_task
docker start awx_task

[tgates81]

Any plans yet on a date for the next release that will include this fix?

[ryanpetrello]

Hey @tgates81 (and others),

I'm planning to release a new version of AWX tomorrow that will address this issue:

https://github.com/ansible/awx/pull/6263/

[kdelee]

As we were reproducing this on a regular basis on internal jobs and many upstream users as well, I am going to call this verified by the fact that it has been solved for all parties. Closing