Nomad: Accept JSON via nomad job run

[ccakes]

Nomad version

Nomad v0.9.6

Operating system and Environment details

Mac + Linux

Issue

From the Nomad CLI tool I want to be able to output a job spec and then "run" that output.

For example I have a script which stops a running tasks, applies changes to an underlying service, and then restarts it. nomad job inspect outputs JSON which I can submit back via the Job HTTP API but then I'm forced to pass Nomad and Vault tokens via CURL and it's a bit messy. I have an environment setup where the nomad CLI works and it makes sense to be able to ingest the same format as is output

Reproduction steps

$ nomad job inspect my-task > job.json
$ nomad job stop my-task
==> Monitoring evaluation "f5ac2705"
    Evaluation triggered by job "my-task"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "f5ac2705" finished with status "complete"
$ nomad job run job.json
Error getting job struct: Error parsing job file from job.json: 1 error(s) occurred:

* invalid key: Job

I'm open to any alternatives suggested, but basically I want to be able to interact with a job (ie stop + start) from an environment with access to Nomad, but without access to the original jobspec files.

[tgross]

Hi @ccakes! You're right that the command line doesn't accept the JSON version of the jobspec directly, but the HTTP API does. In fact, the command line client converts the HCL to JSON before POSTing it to the HTTP API itself.

So you're right that the best way to do this is to use curl to send the JSON body directly like so:

▶ nomad job init -short
▶ nomad job run ./example.nomad
==> Monitoring evaluation "6d962c2e"
    Evaluation triggered by job "example"
    Allocation "ea878e56" created: node "707a29ca", group "cache"
    Evaluation within deployment: "8b08b848"
    Allocation "ea878e56" status changed: "pending" -> "running" (Tasks are running)
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "6d962c2e" finished with status "complete"

▶ nomad job inspect example > example.json

▶ nomad job stop example
==> Monitoring evaluation "17721706"
    Evaluation triggered by job "example"
    Evaluation within deployment: "8b08b848"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "17721706" finished with status "complete"

▶ curl -XPOST -H "Content-Type: application/json" \
    -d @example.json http://localhost:4646/v1/jobs
{"EvalCreateIndex":27,"EvalID":"0320367b-cdd8-dd4f-bedc-77aafcc5ec02","Index":27,"JobModifyIndex":26,"KnownLeader":false,"LastContact":0,"Warnings":""}%

▶ nomad job status example
ID            = example
Name          = example
Submit Date   = 2019-11-22T08:25:19-05:00
Type          = service
Priority      = 50
Datacenters   = dc1
Status        = running
Periodic      = false
Parameterized = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost
cache       0       0         1        0       1         0

Latest Deployment
ID          = 114691ea
Status      = running
Description = Deployment is running

Deployed
Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
cache       1        1       0        0          2019-11-22T13:35:19Z

Allocations
ID        Node ID   Task Group  Version  Desired  Status    Created    Modified
157bdc9a  707a29ca  cache       2        run      running   8s ago     7s ago
ea878e56  707a29ca  cache       0        stop     complete  1m19s ago  53s ago

I hope this helps!

[ccakes]

Thanks for the reply! The intent is to run this from within a Nomad job where the environment is already setup nicely to call the nomad CLI with all the appropriate tokens set. If I use curl then I need to pass in NOMAD_TOKEN and VAULT_TOKEN via the request and make sure they're not logged.

Definitely do-able, but not ideal 😉

Was intending this to be a bit of a feature request to create a way for the CLI to output a format which can be read back in.

[tgross]

Was intending this to be a bit of a feature request to create a way for the CLI to output a format which can be read back in.

It'd be a nice quality-of-life improvement for sure to allow for some kind of round-tripping of the jobspec with the CLI. Marking this as an enhancement.

[briansorahan]

This behavior is very surprising and disappointing.
nomad run needs to accept what nomad job inspect writes.

[briansorahan]

Also, since nomad job inspect outputs JSON and not HCL, what is the point of the -json flag?
Wouldn't it be better to have an -hcl flag that would write something to stdout that I could save and send back to nomad with nomad run?

[shishir-a412ed]

@tgross What would be a good way to approach this?

Introduce a boolean CLI flag -json in the nomad job run command.
$$ nomad job run -json example.json

If this flag is set, you can either do (1) or (2):

1) Since nomad job JSON is pretty complex, with multiple nested structs. It cannot be directly unmarshalled into the job struct. One way would be to unmarshal the JSON into a generic interface.

var f interface{} json.NewDecoder(r).Decode(&f)
where r is your io.Reader for the input job JSON.

and use this generic interface f.(map[string]interface{}) to traverse into the job structure and unmarshal nested elements again into generic interfaces, until you are able to decode to atomic types like int, bool, string. Not sure if there is an easier way to unmarshal it directly?

c.JobGetter.ApiJob(args[0]) could take that boolean flag json (as an extra arguement) and unmarshal the JSON into job struct and return.

Then you can just use that job struct to submit the job

OR

2) Take the JSON from CLI and post to nomad server endpoint /v1/jobs, get the response
and post back to the CLI.

[silasb]

I'm new to Nomad and this lack of consistency is frustrating. Ideally you could nomad job inspect --output hcl|json and nomad run could take in any version of hcl or json in.