Awx: [help] How to delete thousands of pending jobs?

[rromanchuk]

I have a situation where something happened with my deploy and thousands of ec2 inventory jobs were scheduled, i'm not going to be able to manually cancel each one. Is there any solution to cancel all these pending jobs, or am i going to have to do this manually through postgres?

Just looking for advice, I will close shortly.

[ghjm]

You could query the API for the job IDs and then use the API to cancel them all. This could be done in Python or any language that supports REST APIs, or even using bash and wget/curl.

[rromanchuk]

actually turns out it's something a little more nefarious, i think it's related to https://github.com/ansible/awx/issues/318

I'm actually not able to cancel a job that has been running forever. tower-cli says the job doesn't even exist. The backlog is from the scheduled ec2 inventory update job.

[rromanchuk]

you can close, most likely related https://github.com/ansible/awx/issues/955

[mageru]

This worked for me

sudo -u postgres LD_LIBRARY_PATH=/opt/rh/rh-postgresql10/root/usr/lib64/:$LD_LIBRARY_PATH /opt/rh/rh-python36/root/usr/bin/awx-manage shell_plus

from awx.main.models import UnifiedJob
for i in UnifiedJob.objects.filter():
   if(i.status == 'pending'):
     i.delete()

[wbieniek]

in longer working under 13.0

[velzend]

For a local development setup.

Find the container id of the task container:

docker ps

CONTAINER ID        IMAGE                COMMAND                  CREATED             STATUS              PORTS                                         NAMES
e3f545e8bd97        ansible/awx:15.0.1   "/usr/bin/tini -- /u…"   5 days ago          Up 23 hours         8052/tcp                                      awx_task
d1dae2d6a975        ansible/awx:15.0.1   "/usr/bin/tini -- /b…"   5 days ago          Up 12 hours         0.0.0.0:80->8052/tcp, 0.0.0.0:443->8053/tcp   awx_web
da04a995ddc9        redis:6.0.9-alpine   "docker-entrypoint.s…"   6 days ago          Up 23 hours         6379/tcp                                      awx_redis
4b33a04e2d45        postgres:10          "docker-entrypoint.s…"   6 days ago          Up 23 hours         5432/tcp                                      awx_postgres

The awx_task container has container ID e3f545e8bd97.
Next exec into the container:

docker exec -ti e3f545e8bd97 bash

bash-4.4# 

Change directory:
cd /var/lib/awx/venv/awx/

Activate the AWX Python virtual environment:
source bin/activate

Start AWX manage with the shell_plus argument:

awx-manage shell_plus

# Shell Plus Model Imports
from awx.conf.models import Setting
from awx.main.models.activity_stream import ActivityStream
from awx.main.models.ad_hoc_commands import AdHocCommand
from awx.main.models.credential import Credential, CredentialInputSource, CredentialType
from awx.main.models.events import AdHocCommandEvent, InventoryUpdateEvent, JobEvent, ProjectUpdateEvent, SystemJobEvent
from awx.main.models.ha import Instance, InstanceGroup, InventoryInstanceGroupMembership, OrganizationInstanceGroupMembership, TowerScheduleState, UnifiedJobTemplateInstanceGroupMembership
from awx.main.models.inventory import CustomInventoryScript, Host, Inventory, InventorySource, InventoryUpdate, SmartInventoryMembership
from django.contrib.auth.models import Group, Permission, User
from awx.main.models.jobs import Job, JobHostSummary, JobLaunchConfig, JobTemplate, SystemJob, SystemJobTemplate
from awx.main.models.label import Label
from awx.main.models.notifications import Notification, NotificationTemplate
from awx.main.models.oauth import OAuth2AccessToken, OAuth2Application
from awx.main.models.organization import Organization, OrganizationGalaxyCredentialMembership, Profile, Team, UserSessionMembership
from awx.main.models.projects import Project, ProjectUpdate
from awx.main.models.rbac import Role, RoleAncestorEntry
from awx.main.models.schedules import Schedule
from awx.main.models.unified_jobs import UnifiedJob, UnifiedJobDeprecatedStdout, UnifiedJobTemplate
from awx.main.models.workflow import WorkflowApproval, WorkflowApprovalTemplate, WorkflowJob, WorkflowJobNode, WorkflowJobTemplate, WorkflowJobTemplateNode
from awx.sso.models import UserEnterpriseAuth
from django.contrib.contenttypes.models import ContentType
from django.contrib.sessions.models import Session
from django.contrib.sites.models import Site
from oauth2_provider.models import Grant, RefreshToken
from social_django.models import Association, Code, Nonce, Partial, UserSocialAuth
from taggit.models import Tag, TaggedItem
# Shell Plus Django Imports
from django.core.cache import cache
from django.conf import settings
from django.contrib.auth import get_user_model
from django.db import transaction
from django.db.models import Avg, Case, Count, F, Max, Min, Prefetch, Q, Sum, When, Exists, OuterRef, Subquery
from django.utils import timezone
from django.urls import reverse
Python 3.6.8 (default, Apr 16 2020, 01:36:27)
[GCC 8.3.1 20191121 (Red Hat 8.3.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)

Now you can perform the actions...

example:

>>> Organization.objects.filter(Q(member_role__members=User(username='bob')))
<QuerySet [<Organization: Sales>, <Organization: IT>]>