Jq: Raw arguments from files

[phs]

I have a use-case that needs to construct json (a kubernetes secret) from multiple sensitive "raw" inputs. This _almost_ fits the purpose of --arg, except that I want to avoid placing my secret values on the process ARGV:

jq \
  --null-input \
  --arg FIRST_SECRET "$(first-secret-on-stdout)" \        # ps aux -> leak
  --arg SECOND_SECRET "$(second-secret-on-stdout)" \
  '{
    "apiVersion": "v1",
    "kind": "Secret",
    "metadata": {
      "name": "my-compound-secret",
    },
    "type": "Opaque",
    "data": {
      "first-secret": $FIRST_SECRET | @base64,
      "second-secret": $SECOND_SECRET | @base64
    }
  }' | \
kubectl create -f /dev/stdin --save-config

Since the secrets are being streamed in from a pipe, I (faintly) hoped I could somehow combine --raw-input with --slurpfile, similar to how --raw-input interacts with --slurp:

jq \
  --null-input \
  --raw-input \
  --slurpfile FIRST_SECRET <(first-secret-on-stdout) \        # Now it's just /dev/fd/XX
  --slurpfile SECOND_SECRET <(second-secret-on-stdout) \
  '{
    /* ... */
    "data": {
      "first-secret": $FIRST_SECRET | @base64,
      "second-secret": $SECOND_SECRET | @base64
    }
  }' | \
kubectl create -f /dev/stdin --save-config

Unfortunately --slurpfile does not take the hint, and insists its input stream be legal json texts.

Is there a way to bind whole, raw files to arguments? I'm not convinced there _isn't_ already a way given the richness of the language, but for the life of me I don't see it.

$ jq --version
jq-1.5

[phs]

@pkoppstein The encoding issue is an important point, but if it becomes a nuisance I can apply base64 myself beforehand instead of relying on (the delightful) @base64:

--arg FIRST_SECRET "$(first-secret-on-stdout | base64)" 

That doesn't resolve the central problem, though, of the sensitive information appearing on jq's ARGV (and so visible in ps or /proc/$pid/cmdline):

$ function secret-on-stdout {
> echo 'super secret!'
> }
$ cat pretend-this-is-jq
#! /bin/bash
sleep 5
$ ./pretend-this-is-jq --arg SECRET "$(secret-on-stdout)" &
[1] 2834
$ ps -o args | grep pretend-this-is-jq
/bin/bash ./pretend-this-is-jq --arg SECRET super secret!

[pkoppstein]

@phs - Sorry, I meant to ask whether you couldn't use --slurpfile. If first-secret-on-stdout produces a suitable JSON string, then the fragment would be like so:

--slurpfile FIRST_SECRET <(first-secret-on-stdout)

and you'd use $FIRST_SECRET[0]

[phs]

@pkoppstein I suppose if we're already base64ing then wrapping the whole result in double quotes is not far off.

jq \
  --null-input \
  --slurpfile SECRET <(echo -n '"'; echo 'secret!' | base64 | tr '\n' '"') '$SECRET[0]'

[pkoppstein]

... or let jq do the walking:

--slurpfile SECRET <(base64 <<< 'secret!' |  jq -R . )

[berlincount]

I stumbled into something similar.

How about making --slurpfile behave like --slurp, and honour -R? :)

[jsoref]

I'm hitting this where I'm trying to use jq to wrap markdown in a json wrapper to post to github. It works most of the time, except when I hit:
/app/unknown-words.sh: line 294: /usr/bin/jq: Argument list too long
https://github.com/jsoref/piskel/runs/515430735#step:4:14431
Having to consider how to call jq 14,000 times in order to build up a file which I can then send to jq just to wrap the content in json seems a bit much.

[pkoppstein]

Having to consider how to call jq 14,000 times in order to build up a file which I can then send to jq just to wrap the content in json seems a bit much.

Indeed, but it's difficult to imagine a scenario that would require a large number of invocations being necessary "just to wrap the content". In fact I'd venture to say that it's likely you have either not understood or not considered the relevant options.

If you would like further assistance, please give further details about the problem at hand, preferably on stackoverflow.com using the jq tag and following the guidelines at http://stackoverflow.com/help/mcve