Nomad: Nomad CLI should allow multiple jobs in stop command

[bengaywins]

Currently one is only able to feed a single job ID into the nomad stop <job> command. It would be great if this could allow for many as opposed to just a single one. That way one could stop thousands of jobs at once if necessary as opposed to needing a for loop or similar single execution per job.

[nugend]

I see the utility of this, but also question what the implementation should look like. Mainly, if one is stopping thousands of jobs, then it implies that there is some scripting involved (to get the job names in the first place). At that point, what practical difference is there between issuing nomad stop <job> thousands of times in a loop?

My thoughts are that some sort of name or tag filtering would be the most appropriate way to implement it, but maybe I am overlooking something?

[bengaywins]

I will use my exact scenario that happened. I had nearly 5k jobs that needed to be stopped and it took a little over 2hrs to do in a for loop. If one can stop many jobs in a single go, without the need to submit a single command each time for every job and wait for that command to give a status, this could be dropped down to minutes.

[dadgar]

@gehzumteufel Did you use nomad stop -detach <job-id>.

[nugend]

@gehzumteufel Ah, I see. So not just a nomad stop, but a stop that works in parallel.

I think @dadgar's idea is workable, but solutions that involve external scripting (since you might not want to move on until everything is down) can be failure prone.

[Miserlou]

We are also finding that stopping a large number of jobs in sequence is very time consuming. We would like to be able to nomad stop --all.

Also why do all hashicorp products only use single - rather than -- for named arguments. So annoying.

[schmichael]

Also why do all hashicorp products only use single - rather than -- for named arguments. So annoying.
-- @Miserlou

A Go-ism (from Plan 9 before that?) we chose to keep I'm afraid: https://golang.org/pkg/flag/

[Miserlou]

In case anybody is here looking for this basic functionality that Nomad should provide but doesn't, and you have likely made a mistake by choosing this stack, you can at least try this:

echo "Killing dispatch jobs... (This may take a while.)"
if [[ $(nomad status) != "No running jobs" ]]; then
    for job in $(nomad status | awk {'print $1'} || grep /)
    do  
        # Skip the header row for jobs.
        if [ $job != "ID" ]; then
            nomad stop -purge -detach $job > /dev/null
        fi  
    done
fi

[Fuco1]

With use of GNU parallel you can speed this up significantly

nomad job status YOURJOB | grep pending | awk '{print $1}' > jobs
cat jobs | parallel -j32 nomad job stop -detach -purge 

Adjust the number of cores as you see fit

[analytically]

Or just kill all pending jobs. It's pretty bad for operators that this behaviour is not built-in.