Jq: runs from command line not in shell script

[rojomisin]

Sorry if this is an easy one, banging my head on the wall at the moment... I can run this command on the command line, but once this is executed from sh/bash in OS X it fails

This works fine from command line
jq --sort-keys '.TemplateBody | .Parameters.AMIpacker.Default = "ami-1234addd" '

shell script is

aws cloudformation get-template --stack-name $stack_name --profile $aws_profile | $JQPATH --sort-keys '.TemplateBody' >"$stack_name-current_$MYDATESTAMP.json"

### doesn't work in double quotes either
#JQSUBCMD="jq -r --sort-keys \".TemplateBody | .Parameters.$cfparam_name.Default = \\\"$cfparam_value\\\"\""

### single quote escaping 
JQSUBCMD='jq -r --sort-keys '"'"' .TemplateBody | .Parameters.'"$cfparam_name"'.Default = "'"$cfparam_value"'" '"'"' '

echo $JQSUBCMD
aws cloudformation get-template --stack-name $stack_name --profile $aws_profile >"$stack_name-target_$MYDATESTAMP.json"
$JQSUBCMD "$stack_name-target_$MYDATESTAMP.json"

Resulting output which echos the command and trys to run it

sh cloudformation_update_stack.sh stack pro AMIpacker ami-1234addd 
jq --sort-keys '.TemplateBody | .Parameters.AMIpacker.Default = "ami-1234addd" '
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
'.TemplateBody
jq: 1 compile error

Why doesn't this work in a shell script?

My use-case is to substitute or merge values into cloudformation templates using shell scripts (and jenkins jobs).

[glennj]

You can't store the command in a variable like this:

JQSUBCMD='jq -r --sort-keys '"'"' .TemplateBody | .Parameters.'"$cfparam_name"'.Default = "'"$cfparam_value"'" '"'"' '
....
$JQSUBCMD filename.

The jq body that you so carefully quoted will not remain as a single argument due to shell word splitting. You are actually trying to execute this command:

jq -r --sort-keys "'.TemplateBody" "|" ".Parameters.AMIpacker.Default" "=" "\"ami-1234addd\"" "'"

See http://mywiki.wooledge.org/BashFAQ/050 for the gory details.

If you absolutely need to store the command in a variable, use a bash array:

jqCmd=( jq -r --sort-keys ".TemplateBody | .Parameters.$cfparam_name.Default = \"$cfparam_value\"" )
....
"${jqCmd[@]}" "$filename"

[rojomisin]

Fair enough, and I would not normally do that, however I cannot get a shell variable to parse properly in jq. I guess my problem is that I cannot get the simplest use case of shell variable substitution to work in a script.

I think the problem lies in the fact that both shell and jq use the $ for variables.

why doesn't this work?

jq --sort-keys \'".TemplateBody | .Parameters.$cfparam_name.Default = \"ami-1234addd\""\' file.json

jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
'.TemplateBody | .Parameters.MyCentOSAMIpacker.Default = "ami-1234addd"'
jq: 1 compile error

Would you have an example for the simplest way to do this?

[wtlangford]

Assuming $cfparam_name is a shell variable, something like this should work
for you:

jq --arg cfparam_name "$cfparam_name" --sort-keys '.TemplateBody |
.Parameters.[$cfparam_name].Default = "ami-1234addd"' file.json

The trick, here, is to rebind your shell variable to a jq variable.
We always recommend the use of single quotes on your jq program, to
prevent your shell from attempting to expand jq variables. I put
quotes around "$cfparam_name" when rebinding it to prevent spaces from
breaking things.

On Wed, Apr 6, 2016 at 4:21 PM rojomisin [email protected] wrote:

Fair enough, and I would not normally do that, however I cannot get a
shell variable to parse properly in jq. I guess my problem is that I cannot
get the simplest use case of shell variable substitution to work in a
script.

I think the problem lies in the fact that both shell and jq use the $ for
variables.

why doesn't this work?

jq --sort-keys '".TemplateBody | .Parameters.$cfparam_name.Default = "ami-1234addd""' file.json

jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at , line 1:
'.TemplateBody | .Parameters.MyCentOSAMIpacker.Default = "ami-1234addd"'
jq: 1 compile error

Would you have an example for the simplest way to do this?

—
You are receiving this because you are subscribed to this thread.

Reply to this email directly or view it on GitHub
https://github.com/stedolan/jq/issues/1124#issuecomment-206542868

[rojomisin]

hi @wtlangford I cant see the example, I only see ... elipsis

[wtlangford]

Well, that's what I get for using my phone. Let's try again.
Assuming $cfparam_name is a shell variable, something like this should work for you:

jq --arg cfparam_name "$cfparam_name" --sort-keys '.TemplateBody | .Parameters.[$cfparam_name].Default = "ami-1234addd"' file.json

The trick, here, is to rebind your shell variable to a jq variable. We always recommend the use of single quotes on your jq program, to prevent your shell from attempting to expand jq variables. I put quotes around "$cfparam_name" when rebinding it to prevent spaces from breaking things.

[rojomisin]

Yes, and this I have also tried, which is why it's so puzzling that it does not work (OS X el capitan jq-1.5)

I am switching the jq arg to be cfp to keep it straight, but I did try your example verbatim and it also failed the same way

jq --arg cfparam_name "$cfparam_name" --sort-keys '.TemplateBody | .Parameters.[$cfparam_name].Default = "ami-1234addd"' FusionNSqa-target_040612232016.211459970601.json
jq: error: syntax error, unexpected '[', expecting FORMAT or QQSTRING_START (Unix shell quoting issues?) at <top-level>, line 1:
.TemplateBody | .Parameters.[$cfparam_name].Default = "ami-1234addd"                            
jq: 1 compile error

shell substitution happens first, and then I would expect jq to be able to parse the arg variable cfp but this does not work, does it work for you?

I also tried double quotes to get it to interpolate but it also does not seem to work

enclosing the whole thing in "" just echoes a well-formed string, not execute jq on the string

jq --arg cfparam_name "$cfparam_name" --sort-keys "\".TemplateBody | .Parameters.$cfparam_name.Default = "ami-1234addd"\"" FusionNSqa-target_040612232016.211459970601.json
".TemplateBody | .Parameters.MyCentOSAMIpacker.Default = ami-1234addd"

[wtlangford]

Can I get the output of jq --version, please. I see you said 1.5, I just
want to confirm.

On Wed, Apr 6, 2016 at 5:34 PM rojomisin [email protected] wrote:

Yes, and this I have also tried, which is why it's so puzzling that it
does not work (OS X el capitan jq-1.5)

I am switching the jq arg to be cfp to keep it straight, but I did try
your example verbatim and it also failed the same way

jq --arg cfparam_name "$cfparam_name" --sort-keys '.TemplateBody | .Parameters.[$cfparam_name].Default = "ami-1234addd"' FusionNSqa-target_040612232016.211459970601.json
jq: error: syntax error, unexpected '[', expecting FORMAT or QQSTRING_START (Unix shell quoting issues?) at , line 1:
.TemplateBody | .Parameters.[$cfparam_name].Default = "ami-1234addd"
jq: 1 compile error

shell substitution happens first, and then I would expect jq to be able to
parse the arg variable cfp but this does not work, does it work for you?

I also tried double quotes to get it to interpolate but it also does not
seem to work

enclosing the whole thing in "" just echoes a well-formed string, not
execute jq on the string

jq --arg cfparam_name "$cfparam_name" --sort-keys "".TemplateBody | .Parameters.$cfparam_name.Default = "ami-1234addd""" FusionNSqa-target_040612232016.211459970601.json
".TemplateBody | .Parameters.MyCentOSAMIpacker.Default = ami-1234addd"

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/stedolan/jq/issues/1124#issuecomment-206580832

[rojomisin]

 jq --version
jq-1.5

[jontrainor]

I think I'm having a similar issue where this line in my script:
echo $ASSESSMENT_RUN_STATUS | jq --arg arn "$ASSESSMENT_RUN_ARN" '.failedItems.$arn.failureCode'

results in the following error:
jq: error: syntax error, unexpected '$', expecting FORMAT or QQSTRING_START (Unix shell quoting issues?) at <top-level>, line 1: .failedItems.$arn.failureCode

EDIT:
I think I may have found a workaround for this issue. changing my command like this did not error:

echo $ASSESSMENT_RUN_STATUS | jq --arg arn "$ASSESSMENT_RUN_ARN" '.failedItems | .[$arn] | .failureCode'

[ruchikaguptaa]

I have similar issue in makefile target build.

BUILD_DATE := $(shell date -u +"%Y%m%d")
BUILD_TIME := $(shell date -u +"%H%M%S")
CHART_VERSION_OVERRIDE := "1.${BUILD_DATE}.${BUILD_TIME}"
$(BUILD_DIR)/abc:
    chart_meta=$(shell cat ../../ccp/charts/kube-control-plane/Chart.yaml | jq --arg version "$(CHART_VERSION_OVERRIDE)" '.version = $(version)')
    echo $(chart_meta) > ../../ccp/charts/kube-control-plane/Chart.yaml

But I get the following error. Any recommendations on how I can fix this?

jq: error: syntax error, unexpected $end (Unix shell quoting issues?) at <top-level>, line 1:
.version =           
jq: 1 compile error