Jq: Edit files in place

[ghost]

With GNU Sed for example you can edit files in place:

~~~
-i[SUFFIX], --in-place[=SUFFIX]

   edit files in place (makes backup if SUFFIX supplied)

~~~

However I see no option for this with JQ, currently I must do this:

~sh
br=$(mktemp)
jq .charlie delta.json > "$br"
mv "$br" delta.json
~

Here is a workaround, Awk slurp:

~sh
jq .firstName alfa.json | awk 'BEGIN{RS="";getline<"-";print>ARGV[1]}' alfa.json
~

[karelv]

+1

[ipmcc]

+1

[nicowilliams]

Scripting around this is not an option? In shell:

tmpf=$(mktemp --tmpdir=$(dirname $f) -t)
jq ... "$f" > "$tmpf"
mv -f "$tmpf" "$f"

[nicowilliams]

I'm asking why it's not acceptable.

[ipmcc]

By analogy: Would it be acceptable if <insert your favorite word processor here> made you save to a different file every time you wanted to save your document?

It's not that the problem can't be worked around, it's that doing so is cumbersome. The fact that something as fundamentally stream oriented as sed has an in-place option is a pretty good indicator that this is common expectation/desire.

[nicowilliams]

I see. I've generally not bothered with sed's -i option. For me the
problem has been two-fold: a) it is (well, was) not portable enough, b) I
always have to check whether such a tool's in-place update functionality is
documented as working this way (rename into place), because I generally
need atomicity. And then there's Windows (where renames generally don't
work when there's open file handles for the file in question, unless all
the processes that opened it used an option to allow it).

On Mon, Jun 9, 2014 at 10:23 AM, ipmcc [email protected] wrote:

By analogy: Would it be acceptable if made you save to a different file
every time you wanted to save your document?

It's not that the problem can't be worked around, it's that doing so is
cumbersome. The fact that something as stream oriented as sed has an
in-place option is a pretty good indicator that this is common
expectation/desire.

—
Reply to this email directly or view it on GitHub
https://github.com/stedolan/jq/issues/105#issuecomment-45502984.

[nicowilliams]

Anyways, I can add this easily. My only concern is to limit the number of
new command-line options. Since this option couldn't be implemented in a
jq program, I'll probably do it. Though perhaps a generic utility to wrap
filters with might be better:

$ inplace some-file command args

which would run the given command with some-file as the standard input
and which would redirect the command's output to a temporary file in the
same directory then rename that into place. That's the Unix philosophy,
after all.

On Mon, Jun 9, 2014 at 10:58 AM, Nico Williams [email protected]
wrote:

I see. I've generally not bothered with sed's -i option. For me the
problem has been two-fold: a) it is (well, was) not portable enough, b) I
always have to check whether such a tool's in-place update functionality is
documented as working this way (rename into place), because I generally
need atomicity. And then there's Windows (where renames generally don't
work when there's open file handles for the file in question, unless all
the processes that opened it used an option to allow it).

On Mon, Jun 9, 2014 at 10:23 AM, ipmcc [email protected] wrote:

By analogy: Would it be acceptable if made you save to a different file
every time you wanted to save your document?

It's not that the problem can't be worked around, it's that doing so is
cumbersome. The fact that something as stream oriented as sed has an
in-place option is a pretty good indicator that this is common
expectation/desire.

—
Reply to this email directly or view it on GitHub
https://github.com/stedolan/jq/issues/105#issuecomment-45502984.

[nicowilliams]

Ah, there's a sponge(1) that almost works this way:

http://joeyh.name/code/moreutils/
http://backreference.org/2011/01/29/in-place-editing-of-files/

I think both, sponge(1) and my idea of an inplace(1) would be very useful
generally and very much in keeping with the Unix philosophy.

Would that work for you?

[nicowilliams]

@svnpenn @ipmcc @karelv

Regardless of how you feel about this issue, what think ye of this:

https://github.com/nicowilliams/inplace

?

[drorata]

Is the -i option available or not? In the linked commit it says that this feature was removed.

[pkoppstein]

@drorata - It was (as best I can tell) never available in any official jq release, and is certainly not available in jq 1.4 or 1.5. It was added as an experimental feature on Jul 20, 2014, and removed on Mar 5, 2015.

[pkoppstein]

@svnpenn - Although there are obviously circumstances when -i would be convenient, jq is more like awk than sed. The earlier attempt to endow jq with -i confirms that such attempts will be fraught with difficulties. awks do not have an -i option, and I think it is asking too much of the jq developers/maintainers to devise and implement such an option, especially given there are easy workarounds.

[nicowilliams]

Some of the refactoring to move I/O from main.c was done with in-place
editing in mind, and is necessary to implement sed-style in-place editing.
But it's not a high priority for me at this time, meaning: send us a PR if
you want this done soon :)

[pkoppstein]

@svnpenn - My apologies if I over-generalized, but the GNU awk documentation characterizes "inplace editing" as "an extension" (https://www.gnu.org/software/gawk/manual/html_node/Extension-Sample-Inplace.html). Also, your example does not work with the GNU awk installed on my machine (4.0.2), and the awk that comes with OS X 10.9.5 does not support a "-i" option at all.

[dtolnay]

I agree that jq can and should do in-place. The earlier problems were due to a poor original implementation.

[nicowilliams]

The original implementation was poor and, in petticoat, had bad semantics.
I refactored the way I/O is done in main.c in order to make it easier to
later add sed-like -i -- it should be easy enough now. Send a PR or be
patient :)

[danielbayley]

Is this feature in the next release? I found it in 'release candidate 1', or is that the version that was retracted?

[nicowilliams]

@danielbayley No, it got taken out. 1.5rc1 implemented the wrong -i semantics., sorry.

[nicowilliams]

There's no PR for it at the moment. Feel free to send one :)

[ghost]

+1

[BrendanThompson]

any action on this one?

[ChristianTacke]

Until there is a "-i" builtin option, could you add something like

    tfn="$(mktemp file.XXXXXX)" && jq '.' file >"$tfn" && mv "$tfn" file

to the Examples section of the manpage?
And possibly add "sponge(1)" to the SEE ALSO section?
When the inplace tool becomes more popular (included in moreutils?), it should be linked too.

[ChristianTacke]

@svnpenn you mean the ex(1)-Examples?. ex uses a tempfile internally.
But anyway:

My suggestion is basicly to include _anything_ appropiate in the examples to guide people in the right direction. Having seen the above http://backreference.org/2011/01/29/in-place-editing-of-files/ I was surpsied, on how many stupid ideas are out there to solve this problem.
If that _anything_ is the ex(1) example, also fine.

With the intention of "Hey user, you're looking for -i? Well it's not there yet for various good reasons, and well, you already have plenty of options to do this correctly with already existing tools".

[pkoppstein]

There is now a Q&A about "in-place editing" in the FAQ:
https://github.com/stedolan/jq/wiki/FAQ#general-questions

[mfilenko]

I think tee can be used as a workaround:

jq '.foo' bar.json | tee bar.json > /dev/null

(Output redirection to the black hole is optional if quiet mode is not strictly necessary.)

[mfilenko]

Ah, good call, @svnpenn!

If jq fails the whole bar.json will be truncated, so must be used with care ;-)

[pkoppstein]

Some recommended and some not-recommended workarounds are covered on the FAQ page at "General Questions": https://github.com/stedolan/jq/wiki/FAQ#general-questions

[pkoppstein]

@svnpenn - Done.

[mfilenko]

The workaround with a temporary file can be hardened the following way for use _in scripting:_

jq '.foo' bar.json > baz.json && mv baz.json bar.json

Otherwise, it will also lead to an empty file in case of jq failure.

[p-groarke]

+1

[naugtur]

If this thread is not a proof the feature would save people a lot of time, I don't know what is.

[rjfellman]

Looking for this feature as well!

[pkoppstein]

@svnpenn - Neat. If that works on Windows, then it would be worthwhile having your slurp() packaged as sponge.bat since neither moreutils in general nor sponge in particular seems to be available via choco.

[jakub-bochenski]

contrary too @mfilenko suggestion using tee is not a workaround: https://askubuntu.com/a/752451

[moll]

You could use sponge from https://joeyh.name/code/moreutils/.

[nicowilliams]

There is https://github.com/nicowilliams/inplace.

[jakub-bochenski]

I know, those are both referenced in the FAQ.
Just don't use tee (or direct > in general)

EDIT: just noticed there is a warning against tee in the FAQ too

[BrendanThompson]

Would love to see this in #1352

[ssbarnea]

Here is a more pragmatic question: What alternatives are for in-place editing of yaml or json files?

I am asking this because based on the history of this bug one thing is clear: the author doesn't want this in and it will not happen. If it didn't happen in 4 years, I doubt it will happen too soon so it makes sense to look for alternatives.

I am saying this because the last of this feature breaks the usability completely because it makes almost impossible to write reliable and easy to maintain scripts without it.

@pkoppstein OS X also has -i on awk, is just that its implementation is a little bit different than GNU awk, but is still possible to write code that works for both blends.

[k0pernikus]

@ssbarnea You can use sponge from moreutils together with jq:

 jq . your.json | sponge your.json

[lfarkas]

+1

[kislyuk]

+1

[mchalek]

fwiw +1

[hachi]

4.5 years later, still waiting for inline editing mode

[ssbarnea]

For me the lack of this feature was a reason enough for not using jq at all. I need to change a config file as part of our build process and this involves changing it multiple times during the process. Mainly we successfully used crudini in the past for doing the same to ini files.

The proposed workaround of using wrappers or other extra tools takes the dependency chain too far and we all know that we need and want to avoid having a kitchen-sink dependency chain. At the same time we want to keep out shell scripts clean and easy to read and update.

[joelittlejohn]

I think the options have been spelled out pretty clearly here. Either:

1. Send a PR
2. Add a :+1: to the description above and wait
3. Fork this repo and do whatever you like

It's not the author's responsibility to implement this feature. I've added my vote above as I think this would be a useful feature, but there's no amount of elapsed time that will make this the responsibility of the maintainers to implement. Remember, they have already contributed untold hours of work to this project, to your benefit, for free, to provide all the other features you find so useful.

I'm not trying to cause offence here, just let's remember that we should be grateful for the time that others have spent on this project already, not demanding more. If you need something, implement it and please (if you are so inclined) share your implementation with the rest of us.

[arikkfir]

In case this helps someone, if/until this is implemented - here's another shortcut to achieve this using bash but without temporary files and other tools (bash & jq only):

TJQ=$(cat ./my.json | jq ".myProp = \"newValue\"") && echo ${TJQ} > ./my.json

Sorry if this has already been suggested - couldn't find it in the comments above, but I might have missed it.

[fadado]

I don't like to put the full output in memory, but in this case the code can be clarified a bit.
Subtle changes, not tested:

TJQ=$(jq '.myProp = "newValue"' < ./my.json) 
[[ $? == 0 ]] && echo "${TJQ}" >| ./my.json

The main changes are disable the field spliting in the echo argument and
remove one cat subprocess.

[huan]

+1

[kellytgold]

+1 :( Once again I want to reiterate that this as a native feature of JQ would be very useful. I frequently finding myself needing to merge two or more json data sets. This means there is going to be a source and a destination.

If the source data set has 1000 objects and the destination data set has 10,000 objects we will need to maintain the extraneous objects that are in the destination data set. Unless there is something I am missing I don't actually see a way to do this without producing a new object and thus a new file.

Creating a tmp file and moving it back to the original does work but seems like a sad workaround

Bump

[samliddicott]

+1

[galindro]

+1

[geirha]

I vote against this. jq should remain a stream editor, and not have to deal with the filesystem other than opening files for reading.

There are many edge cases to think about.

1. A temporary file must be created in the same directory (to be renamed over the original at the end). How much effort should be made to make sure it's a unique name that won't accidentally overwrite an existing file?
2. How much effort should jq do in copying the metadata? should it copy ownership? it can only do that if it's running as root. Should it copy permission modes or just rely on umask? should it try to detect if the original file had any OS-specific extended attributes and attempt to duplicate those? what about ACLs?
3. What if the file is a symlink? should it follow the symlink and edit the regular file it targets, or replace the symlink with a regular file, effectively breaking the link? What if it's not a regular file? abort, or replace it with a regular file?

Whatever choices it would end up taking from the above, would be the correct ones in some context, but not in others. In my opinion it's better to let a third-party deal with these choices rather than burdening jq with them.

[metashock]

What if the distribution would come with a separate executable jq-inplace? This could be a shell script.

[mnpenner]

The reason I want in-place editing is because I want to mass update many files. jq already lets me do that, but > will only write back to a single file. Correct me if I'm wrong, but sponge, inplace and the mv hack only work on one file?

I can probably cobble together some kind of for-loop with my pathetic bash skillz but it'd be nice if someone posted an elegant solution for that.

[pkoppstein]

I want to mass update many files.

Since you have a particular scenario in mind, you would probably be best off writing a script to do what you want, maybe using sponge (moreutils) or inplace (https://github.com/nicowilliams/inplace). The general problem is quite hard to state, let alone solve, because there are so many ways to invoke jq with multiple input sources. E.g. consider: jq . <(command) file1.json file2.json.

[geirha]

@mnpenner With a for-loop you'd have your jq script write to a new file, then move it over the original file if jq succeeded:

for file in ./*.json; do
    jq '.[]|the|jq|script' "$file" > "$file.tmp" && mv "$file.tmp" "$file"
done

This is essentially what GNU's sed -i does (and what jq -i would do if it were implemented), it just does it transparently, and also tries to copy some of the file's metadata.

[mnpenner]

I worked out another solution:

find /some/dir -name "*.json" -exec inplace {} jq 'filter_here' \;

I couldn't get sponge to work because find/-exec doesn't like the pipe. Loop has fewer dependencies though, so thanks for that @geirha

[pkoppstein]

@mnpenner - To use sponge with find, you could write s.t. like:

find /some/dir -name "*.json" | while read -r f ; do jq 'filter_here' "$f" | sponge "$f" ; done

[metashock]

@pkoppstein It should be

find /some/dir -name '*.json' -print0 | while read -r -d '' f ;  do jq 'filter_here' "$f" | sponge "$f" ; done

because file names can contain newlines.

Wouldn't

jq -i 'filter_here' *.json

be much better? And more safe to use?

Isn't this the world we all want to live in? Power editing of json files right at our fingertips?

[pkoppstein]

@metashock - The proposed -i option would probably not work that way (*), which is in a sense the main point - previous attempts to implement the feature have run into one problem or another. (One is tempted to say that the feature sounds great until confronted with someone else’s implementation of it.) See the most recent posts by joelittlejohn and geirha above.

As for my suggestion, please note that I did say “s.t. like”, precisely because of the “edge case” you mention.

(*) And if it did, then you’d still need sponge in other cases.

[fadado]

I reject to add -i to JQ. Is a bad idea.

This is a very old theme, solved by Pike and Kernigan since 1984, in section 5.5 (_Replacing a file: overwrite_), page 155 in the book:

https://cs.uwec.edu/~buipj/teaching/cs.388.s14/static/pdf/upe.pdf

This book is a MUST!

Also a must reading is _UNIX Style, or cat -v Considered Harmful_:

http://harmful.cat-v.org/cat-v/

[samliddicott]

Joan;

do you know any unixes that have the "overwrite" command that you have so
kindly referred us to in this 30 year old work?

Furthermore, even use of "overwrite" (or sponge or inplace) would have to
work in conjunction with the various find/for loops that users are having
trouble constructing safely and reliably.

It's no answer.

Generally:

jq is a stream editor, perl is (among other things) a stream editor, awk is
a stream editor, sed is a stream editor (THE Stream EDitor)

jq is the only one without -i, though GNU awk was late, getting -i just
over 5 years ago.

Consider that sed -i is supported in BSDish and GNUish implementations, but
not part of POSIX or Single Unix Specification. Users will have their -i

If you have a [files...] section (and jq does) then this is where -i
applies. Each time the file is opened, refresh stdout.

Of course there may be invocations where -i won't make sense, Unix is full
of such contradictions.

Be assured, jq will get -i, the only question is how much gnashing of teeth
and user time and expense will be wasted playing code golf and fighting
over it before it happens.

Sam

On 6 August 2018 at 10:59, Joan Josep Ordinas Rosa <[email protected]

wrote:

I reject to add -i to JQ. Is a bad idea.

This is a very old theme, solved by Pike and Kernigan since 1984, in
section 5.5 (Replacing a file: overwrite), page 155 in the book:

https://cs.uwec.edu/~buipj/teaching/cs.388.s14/static/pdf/upe.pdf

This book is a MUST!

Also a must reading is UNIX Style, or cat -v Considered Harmful:

http://harmful.cat-v.org/cat-v/

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/stedolan/jq/issues/105#issuecomment-410655342, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFf3zHq9iTRn4Bkv1WOwKyyiShfKti_gks5uOBOdgaJpZM4AmJxc
.

[untoreh]

herestrings use temp files

cat <<< "$(jq ".stuff += [ \"thatsthestuff\" ]" < config.json)" > config.json

also jj does have -i -o flags to pass files

[AtibaBryan]

[gentunian]

herestrings use temp files

cat <<< "$(jq ".stuff += [ \"thatsthestuff\" ]" < config.json)" > config.json

also jj does have -i -o flags to pass files

this is not portable and it works on bash and maybe a few others shells.

[geirha]

herestrings use temp files

cat <<< "$(jq ".stuff += [ \"thatsthestuff\" ]" < config.json)" > config.json

also jj does have -i -o flags to pass files

this is not portable and it works on bash and maybe a few others shells.

In addition, if you make a typo in the jq filter, or jq otherwise fails, config.json will be truncated to 0 bytes.

[k0pernikus]

jq . your.json | sponge your.json

Today, I had the use-case of having to format multiple json files in a folder while also taking into account subfolders.

I wrote a small script to achieve that:

#!/bin/bash
set -euxo pipefail
shopt -s globstar
for file in **/*.json; do
    jq . $file | sponge $file
done

(This script has to be in the folder where your json files are located. I may update this one day with the path being parameter.)

[whyvez]

jqi() {
  cat <<< "$(jq "$1" < "$2")" > "$2"
}

jqi ".stuff += [ \"thatsthestuff\" ]" config.json