Fish-shell: Cannot save multi-line output in a variable (support $() command substitution syntax)

[lilyball]

There seems to be no way to capture multi-line output in a shell variable. Any attempt to do so splits each line into a separate array element. In bash I'd simply put double-quotes around my $() invocation, but in fish you cannot quote a command substitution. This is rather unfortunate because it means I cannot capture multi-line output in a variable and send it back to a separate command without really weird contrivances, such as replacing all newlines with NUL when saving and reverting the process when emitting.

[lilyball]

And for the record, doing such a replacement makes it impossible to look at the status of the original command* (e.g. in issue #158), because Fish provides no replacement for bash's PIPESTATUS.

*without really stupid things like piping the output to a file temporarily

[lilyball]

As a separate idea, if there was some flag to echo that made it emit each argument on a separate line instead of space-separation, that would at least provide a workaround. And yes, this is something that can be shellscripted. But it really shouldn't be necessary.

[lilyball]

The lack of a way to quote a command substitution has just reared its head in another location: the inability to use command substitution with test -n. If the command substitution results in no output, it also results in no arguments at all, and test -n with no subsequent arguments actually returns true.

[maxfl]

For me it looks quite natural: no output - no arguments. One empty line of output - one empty argument.
What is strange for me is why 'test -n' returns 0 at all? Naively I would suppose it to return 1.
I would suggest to change the returning value for this case, but the system 'test' and the bash internal 'test' also return 0 in this case.

Concerning the multi-line output I completely agree with you. It would be very nice to have possibility to do command substitution inside double quotes, something like "$()", because substituting "()" is dangerous.

[lilyball]

As an alternative to $() (which, btw, I still want just so I can more easily combine command substitution with surrounding variables, etc), it might be nice to just have a trivial way of taking a multi-element variable and creating one argument with all elements joined by a newline. This would effectively reverse the process of storing a command substitution into a variable, with the one exception that this will force a trailing newline when the command substitution may not have had one. Right now I can write (for elem in $var; echo $elem; end), but that's rather awkward to do everywhere.

[maxfl]

Shorter ways to do the same:
(printf '%sn' $var)
or
(echo $varn)?
But fish again splits the substitution into arguments by newlines. How do you use it?

[lilyball]

I assume you mean printf, and that would work but it has to spawn a separate process which is slow.

As for (echo $var\n), that's going to put a space after every newline, which is wrong, as well as include two newlines at the end.

The use of this is for piping to another process. Obviously storing it back in a variable would just split it again, but I can say (for line in $var; echo $line; end | grep foo)

[maxfl]

You are right.
echo you can use when you do not care about extraspace (like in grep example). And printf when you do not care about preformance.

[ridiculousfish]

This outputs 1:

count (printf "%s %s %s" first second third)

This outputs 3:

count (printf "%s\n%s\n%s" first second third)

I don't understand why newlines from a substitution result in arrays, while spaces from a substitution do not. I wonder how bad it would be to treat whitespace uniformly here.

[maxfl]

I think that it's perfectly consistent behaviour (considering that fish distinguishes whitespaces). From my experience automatic expanding space-separated strings to array makes it difficult to cope with space containing strings and lead to rather extensive subquoting.

From the other hand: additional splitting can be implemented by additional command. But if fish will split all whitespaces it will be difficult to glue them back if needed.

[ridiculousfish]

My suggestion was actually in the other direction: treat newlines like spaces (so they do not split). So

count (printf "%s\n%s\n%s" first second third)

would output 1. On the other hand, then we would need some way to split a string.

I also think kballard's suggestion of quoted substitutions via e.g. "$(echo hello)" is very good and would address this neatly.

[maxfl]

I completely agree about "$(echo hello)".
I also sometimes think, that replacing () by $() in normal mode is a good
way, because in this case there will be no need to escape ().

[Soares]

Personally I'm very -1 to $() syntax.

One of the things that drew me to fish was that it simplifies all of the crazy different ways to do things that bash/zsh/etc have.

See the design doc, the law of orthogonality. To quote,

The shell language should have a small set of orthogonal features. Any situation where two features are related but not identical, one of them should be removed
...

• The many Posix quoting styles are silly, especially $''.

I think it would be a serious degradation of fish's principles to add $() syntax.

From what I understand, adding a third quoting syntax ("" vs '' and unquoted) was a large and difficult decision. If we're going to do a change like this we should at least strongly consider all alternatives first.

Understanding the whitespace rules right now is a big pain point. Currently:

> count 'one two three'
1
> count one two three
3

That I get. Quotes prevent splitting. Cool.

> count (echo one two three)
1

Ok and it looks like subcommands are automatically considered quoted...

> count one\ntwo\nthree
1

And newlines don't count as whitespace when you're splitting things, that's cool I guess

> count (echo one\ntwo\nthree)
3

Wait, what?! Something's seriously smelly here. If (echo one two three) acts like 'one two three' then (echo one\ntwo\nthree) should act like 'one\ntwo\nthree'.

And, of course, the big scripting problem is that the data isn't preserved by set:

> set foo (echo one\ntwo)
> echo $foo
one two

We had newlines and now we have spaces.

I understand why it all works like this but it definitely doesn't seem very fish-like. User focus is violated here, orthogonality certainly isn't helped. I'm not sure what the answer is, but any or all of these are options I like better than $():

1. Substitution output should be treated exactly like a quoted string. If count "one\ntwo\nthree" is 1 then count (echo one\ntwo\nthree) should be 1
2. Alternatively, make substitution output be _unquoted_ and allow it inside double quotes. This is a pretty big semantic change.
3. Don't split output on newlines by default. Add a dice command which splits output-in-lines into quoted-args-separated-by-spaces. This is analogous to how we have psub instead of process substitution: fish should prefer new commands to new syntax.
4. Give set a new flag which tell it how to dice variables.
   (Pros: you could split things on colons if that's your thing. Con: doesn't help count or () usage.)
5. Give echo a new flag which tells it how to separate args (fixes the symptoms not the problem.)

Concerning option 2, it would mean that unquoted command substitution would act like this:

> count one two three
3
> count "one two three"
1
> count (echo one two three)
3
> count "(echo one two three)"
1

Which is a big change, but is at least consistent.

Many of these are more work-arounds than solutions. And even if we fix set and echo by adding more flags we still have the count elephant in the room, which can't take any flags of any sort: if we add flags to set that tell it how to split arrays then it seems pretty silly that count can never have the same functionality.

I'm most in favor of 1. or 2. above, but either way I'm against adding $() syntax. One more design doc quote:

Most tradeoffs between power and ease of use can be avoided with careful design.

Most of my ideas above are still half-baked, but I think more carefully designing the text-splitting rules is a far superior option to adding more syntax and flags on top of the existing ones in an attempt to patch all the holes. That sort of activity is what got bash where it is today.

[ridiculousfish]

Thank you for the thoughtful comments Soares.

I think there might be a point of confusion. The $() proposal is to add a way to do command substitutions within double quotes. It need not work outside of them:

> count (echo 1\n2\n3)
3
> count "$(echo 1\n2\n3\n)"
1
> count $(echo 1\n2\n3\n)
syntax error

In that way I think it's similar to your suggestion 2 above. The dollar sign has the advantages of not requiring escaping parens within double quotes (which would be irritating), and because it doesn't conflict with any existing valid syntax.

This was an element of kballard's "Fish Scripting Wishlist" from June 21. To quote Kevin:

1. Fish needs a way to do command substitution within a double-quoted string. The simplest solution is probably to support $() within double-quotes. The reasoning is twofold: first, a command substitution that evaluates to no output ends up being stripped entirely from the argument list of its surrounding command. This has extremely bad implications when using it with, e.g. test -n (some command). Second, this would allow for storing the output with its newlines in a variable, instead of having the output be split into multiple elements. And as a bonus it would make it a lot easier to combine command substitution output with other text in the same argument, e.g. "$somevar"(some command)"$anothervar".

So "$()" would solve several problems, which is why it's interesting.

I think we'd also like to avoid splitting on newlines as you suggest in 3 above. The main blocker there is the sheer quantity of work required to vet all existing fish code, and in adding the new 'dice' command.

[Soares]

Cool. I'm much less opposed to that syntax if it's only in double quotes.

(Note: In fish 1.x variables in double quotes expanded to the _first_ arg in that variable array and there was no way to slice/get the other args. This was a concession to Axel Liljencrantz IIRC, who didn't want double quotes in the first place and insisted on having something that set double quotes apart, which I thought was stupid. I was right on the brink of a rant about the magical things that double quotes do before I realized that double quotes are pretty sane in fish2.0. Nice work on that!)

It's still weird to me to use $() instead of (). I understand how much of a pain in the ass it would be to escape all parens in double quotes, but isn't that sort of the point of single quotes instead of double quotes?

Also I'm wary of conflating $() in double quotes with POSIX $(); it could be confusing to newcomers. I'd prefer #() or {} syntax to distance ourselves a bit from the dollar sign, which so far _only_ means variable expansion. (The fact that it's not used in () sets something of a precedent for not re-using the dollar sign.)

Also I think that expanding an empty variable should yield '', because even if we do have expansion in double quotes you still have the old

> set var
> test -n $var

problem, which is a serious gotcha for newcomers.

[lilyball]

What is POSIX $()? Bash has been supporting $() in double-quotes to do exactly what we're suggesting here for quite a long time, and nobody has a problem with that. Adding yet another syntax for process expansion seems like a really really bad idea. You say dollar sign only means variable expansion so far. That's fine, but I see no problem with expanding that to just meaning expansion in general, with the two supported expansion types being either variables or subprocesses.

Expanding an empty variable should never yield '' unless that variable is enclosed in double-quotes. Changing that would basically introduce a requirement to use eval whenever you want to conditionally add an argument, which is a really bad idea. I wouldn't worry too much about test -n $var; newcomers who don't understand what they're doing have worse issues than eliding an argument due to empty variable expansion.

[Soares]

Bash's $() is what I meant when I said posix (is that posix?), as I didn't want to single out bash. Bash supports $() syntax everywhere, not just in double quotes: if fish supports it but only in special circumstances that seems a bit weird.

Especially since fish keeps command substitution but changes the syntax from $() to (). Switching from $() to () for command substitution implies that fish is trying to distance itself from the ``$() ${} <() hellhole that is bash/zsh substitution; it seems weird to turn around and put $() syntax back in double quotes.

Which is why I'm tentatively in favor of biting the bullet and allowing () in double quotes, though that's pretty backwards incompatible.

[lilyball]

I would consider interpreting () inside double-quotes as substitution to be extremely surprising and absolutely terrible. Currently the _only_ substitution that occurs inside of double-quotes (note: escaping is not substitution) requires a $ character. There is no good reason to introduce any separate ways to invoke substitution inside of double-quoted strings.

[Soares]

Yeah, I concede that that might be the best course of action from where we're standing. However, the converse to your statement is this:

Remember that the _only_ substitution that occurs outside of quotes is $ and () characters. There is no good reason to add new syntax for the sole purpose of restricting one of the existing substitution methods.

Having $ and () substitute outside of quotes while having $ and $() substitute inside of quotes is messy and dichotomous and exactly the sort of thing that fish (as upposed to bash/zsh/etc) is supposed to avoid. It's in direct opposition to a few parts of the design doc and it raises the question of "if I use $() in a string why don't I use it outside of a string?".

It's a shit situation. () was chosen for command substitution back when fish didn't have double quoted strings. $var is already supported in double quoted strings, so we can't do it 'ruby-style' and say that #{} un-double-quotes you, so that you have to do variables like #{$var} and command substitutions like #{(command args)}.

[lilyball]

If you really want to be pure, you could introduce $() outside of double-quoted strings as well, and deprecate the bare () style. This way "the only substitution that occurs requires $" will be true regardless of quoting.

Of course, at this point, you may then say "what about {}?" While not strictly speaking substitution, you could then argue that we should change that to ${} as well. And actually, I'd have no objection to that. The fact that fish interprets {} specially even when there's only one branch inside is very annoying when trying to work with git. Plus you could then allow ${} inside of double-quoted strings without a problem.

That said, from a practical standpoint, I think making these changes here is unnecessary. I'd rather just introduce $() inside of double-quoted strings and be done with it.

[Soares]

Yeah, if the language were being designed from the get-go I'd recommend axing the bare version. If $() is added to double quoted strings then I definitely think there should be some sort of long-term plan to restore consistency, which I think is important to fish in its role as a bastion against shell scripting insanity.

Just a note: If you have long double quoted strings, you can already

> "have command substitution "(in double-quoted strings)" like this."

It's one character more and it doesn't require extra syntax.

It's really only the edge-case of "$(some expression)" where the new syntax is useful, and I'd prefer to see that fixed by sane newline handling and a nice dice function. If we get those I'm not sure we even need the $() syntax.

[lilyball]

No you can't.

> echo "test"(echo one\ntwo)"ing"
testoneing testtwoing

The desired behavior is

>echo "test$(echo one\ntwo)ing"
testone
twoing

[Soares]

Right. Sorry. I meant to say that you can do that if we make variables not be split on newlines by default and add a dice command (as discussed above), then the existing syntax works.

[ridiculousfish]

Soares, out of curiosity, where did you get name for the dice command? We're interested in adding some more sophisticated string manipulation, and I'd rather adopt an existing syntax than invent a new one.

[Soares]

For some reason I thought the existing split command was named "slice". Given that split splits things by length and not content I thought a command that split things by content and not length could be called dice (as in 'slice and dice'.)

Seeing now that I was thinking of split, slice is probably a much better name for the command we're discussing.

[maxfl]

Soares, thank you for that long argumented answer. I now agree with you (:

[gustafj]

@Soares, very well written and good arguments, I agree with consistency here, I don't see the point of having one syntax in double quotes and one outside of double quotes.
I also wouldn't want to type $() for every command substitution (outside of "") I do daily, but I see the problem of changing the meaning of () inside "".
@kballard, the issue with {} is solved by #354 (remove it).

To summarize, either only $var & $() or $var & (), I would prefer the second option, although it breaks horribly.

[pgan002]

How about an option 2a: Substitution output should be treated like an unquoted string, and instead of dice or slice and list, splitting and joining can be done using the IFS variable (as discussed for Fish 1.0). Something like:

> count a b
2
> count (echo a b)
2  # As we usually want
> set -l IFS ''; count (echo a b)
1
> set -l IFS '\n'; count (echo a b)
1

[lilyball]

I hate IFS. IMO it's one of the worst parts of bash scripting.

[Soares]

Agreed on the IFS hatred.

I also can't quell the sneaking suspicion that we're going to re-implement
lisp.

But seriously, the array syntax promises to be pretty powerful if we can
get it right.

On Wed, Dec 12, 2012 at 11:08 PM, Kevin Ballard [email protected]:

I hate IFS. IMO it's one of the worst parts of bash scripting.

—
Reply to this email directly or view it on GitHubhttps://github.com/fish-shell/fish-shell/issues/159#issuecomment-11324611.

[JanKanis]

Just now reading up on this due to the list discussion.

One thing missing in this discussion is why splitting behavior currently is as it is:

me@mypc ~/test> ls -1
another file
file 1
third file
me@mypc ~/test> count (ls)
3

If proposal 1 were implemented:

me@mypc ~/test> count (ls)
1

if proposal 2 or 3 were implemented:

me@mypc ~/test> count (ls)
6

Newline and space/tab in current unix behave as a level 1 and level 2 separator. That comes (I guess) from human text where newline and space in a sense do the same. In command output, neither of these already mean something and sometimes you need spaces, so the logical choice was to use the level 1 separator. When entering commands, the newline is already taken as separating different commands, so the level 2 separator was used. This is how all command line unix software got written, and fish just follows by converting the level 1 separator used in command output to the level 2 separator that count or any command line argument uses. There are zillions of programs that give output in one line per item format.

This allows e.g. using of spaces in filenames. Of course it breaks when there's a newline in a filename or anywhere where you don't want it to split output, and the general design turned out to suck, but _you can't change it without changing all command line unix software ever written!_ Doing so would make fish suck for interacting with non-fish-aware commands -- and therefore as a shell.

So -1000 on changing the default whitespace splitting. That said, I'm in favor of something like a list command, and I also hate $IFS.

[dag]

I really think consistency is king and fish should do one of:

1. Allow all substitutions _and_ escapes _and_ expansions inside double quotes, with the exact same syntax. Quotes then simply delimit arguments containing whitespace, without changing any semantics. If you want a literal string either escape the special characters or use single quotes.
2. Remove support for double quotes or make them behave like single quotes. Quotes then are "string literals".

For option 1 I'm strongly against changing the syntax for command substitution, and for option 2 we need some other way to say "treat this command substitution as a single string regardless of newlines". I favor the first option.

[dag]

Or 3. have an "unquote" syntax like Ruby's #{} but I'd prefer it to be just {}. I'm not a fan of this option as "{(ls)}" is a bit awkward and it means double quotes are almost the same thing as single quotes.

[ghost]

The first option for that sounds perfectly valid and it'd probably solve a lot of problems. It seems more orthogonal.

[kevna]

I don't know about others here but I rarely use literal () characters as part of a string, if other people feel this way then perhaps it would not be so significant to have to escape them to prevent substitution.

Alternatively, are there any side issues to the following - use ( to begin a substitution ( to use brackets normally (or vice versa) where one is followed by a space.

[xfix]

@kevna: I like that one. I wasn't sure about suggesting it, but now I think it's a good idea. When I type ( anyway, it's part of code (for example Perl oneliner) or regular expression, and usually I write code in single quotes to avoid variables being written into result.

Then again, some code (mostly related to completions uses ( in double quotes). But I think that completions are relatively easy to fix.

[ghost]

I dunno how much of this has actually been worked on, but I just thought of a better alternative: why not just make (( and )) literally paste the output of the command as one argument? It looks similar to the bash syntax for a conditional, but in cases where bash conditionals would be used, it would throw an error for invalid syntax anyway. (just having a subshell as the command doesn't work)

[xixixao]

That isn't multiline.

[xfix]

I made this just for fun. The real solution would be nice, obviously, as currently there is no real way to do so. This is a really needed feature, the issue is that nobody can decide on its syntax. Obviously, pipeset is not a proposed syntax, it's just a function because I cannot create new syntax from the fish shell itself.

function pipeset --no-scope-shadowing
    set -l _options
    set -l _variables
    for _item in $argv
        switch $_item
            case '-*'
                set _options $_options $_item
            case '*'
                set _variables $_variables  $_item
        end
    end
    for _variable in $_variables
        set $_variable ""
    end
    while read _line
        for _variable in $_variables
            set $_options $_variable $$_variable$_line\n
        end
    end
    return 0
end

Example:

~ $ perl --version | pipeset perl_version
~ $ echo $perl_version

This is perl 5, version 14, subversion 4 (v5.14.4) built for cygwin-thread-multi
(with 7 registered patches, see perl -V for more detail)

Copyright 1987-2013, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl".  If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.


~ $

[kaleb]

@xfix That's a nice solution. It would be nice if one could just pipe to set.

> perl --version | set perl_version
> echo $perl_version

[anordal]

PHP's explode/implode functions look easy to adopt to a shell syntax:
http://php.net/manual/en/function.explode.php
http://php.net/manual/en/function.implode.php
(IMO, more straightforward & recognizable than python's split/join)

I am very much in favor of ridiculousfish's "no treatment of whitespace" in variables (should be compatible with dag's suggestion 1, if I get him right, as he's talking about quoting whitespace itself). Having to quote variables to keep them together (keep them from "exploding", so to say) is the number one thing I hate about posix shells.** Explosion should be explicit!

In other words, no implicit array interpretation, so

count (echo a\nb)

would output 1, whereas if I actually want to do that kind of string processing,

count (explode \n (echo a\nb))     #Look ma, no IFS

would output 2.
As can be guessed, there is a reverse command, implode. This would output 1:

count (implode \n (explode \n (echo a\nb)))

As a bonus, we then get some string processing primitives!
Search & replace:

implode 'replace' (explode 'search' $mystring)

Basename:

(explode / $mypath)[-1]

** What posix shells really really want to do, is to explode your variables, eat your data as commands, and kill you! How many times, in writing sizable shell scripts, have you stopped to think how many bugs you're just creating, and contemplated a safer programming language like C?

[xixixao]

@xfix Thanks for your code, works like a charm!

[derekstavis]

Another great syntax that runs away from $ is what Swift proposes for string interpolation:

set CPUINFO "\(cat /proc/cpuinfo)"

[jakwings]

I'm opposed to ((...)), #{...}, {...}, "\(...)". And "$(...)" is the best since $ is already a special symbol for variables. If we also change the syntax of unquoted command substitution to $(...), that would be better. Then we can use () in test (...) -a (...) without escaping.

[imgx64]

Maybe I'm missing something, but isn't this already fixed?

From the documentation:

If the output is more than one line long, each line will be expanded to a new parameter. Setting IFS to the empty string will disable line splitting.

So the solution is:

> count (echo one\ntwo\nthree)
3
> begin; set -l IFS; count (echo one\ntwo\nthree); end
1

[faho]

@imgx64: That has been mentioned here before.

While mucking with $IFS does solve the issue, it's supremely awkward.

[dxlr8r]

# echo 1 2 3 | read -z flatvar; count $flatvar
1

[anordal]

LGTM, dxlr8r!

Needless to say, while read -z reads up to NUL, an option to always read to EOF would be a slightly more proper solution for this use case.

[andrewrk]

What's preventing this issue from being resolved? It seems like several reasonable paths forward have been proposed. Can we pick one and move on with it?

I'll throw in a vote for deprecating () in favor of $().

@dxlr8r thanks for the tip about read. Useful workaround.

[faho]

What's preventing this issue from being resolved? It seems like several reasonable paths forward have been proposed. Can we pick one and move on with it?

Well, we'd have to actually pick one and then someone has to write the code. Or someone writes code that we can agree on.

---

Personally, I now rather strongly believe that switching to "$()" is the right solution.

• Not splitting on newlines would be an awfully breaking change. Just see our scripts. Plenty of our command substitutions use this feature, and finding out which exactly would require auditing everything. Additionally, users would need to do all of that too, and it would have to be a flag-day transition (i.e. fish 3.0 gets released, everyone needs to rewrite everything for it, some people are still on 2.X). See what happened with python.
• Splitting on newlines is a good default. Unix usually operates on a one-thing-per-line basis. E.g. grep will print one match per line, git log has a "--oneline" switch to print one commit per line, which is quite useful in scripts. Configuration files are usually line-based. Our own commandline will print one token per line if told to split by tokens, ........ ("all whitespace" is unworkable because that leads to bogosities like bash's issues with spaces)
• $() would be usable inside and outside of double-quotes, without changing the list of characters that start an expansion (and hence need to be escaped) in them. This also means it could be the one-and-only syntax to denote command substitutions.
• $() would be the same syntax as POSIX, and inside of double-quotes would have the same semantics. While we don't _need_ to adhere to POSIX, if the POSIX syntax works why invent something else? We don't do if $COMMAND; then $COMMAND; fi and for $COMMAND; do $COMMAND; done because it's nicer to have just one block separator (that being the begin and end pair), but we do do $COMMAND $argument1 $argument2 because that's a perfectly cromulent way to write it.

I believe all this together means that $() is better than all other proposals.

---

So, what's my concrete proposal?

First, introduce $() as an alternate command substitution syntax. Unquoted, this would work like () does currently - it would split the output on newlines, and pass each line as a separate argument. In double-quotes, it would pass the entire output as-is, including newlines.

This would mean you could use it as a drop-in replacement for (), and if you wanted to save multi-line output verbatim, you'd just do set var "$(COMMAND)".

Also, start deprecating (). With the announcement of fish X.Y.Z, include a notice with a deadline. With a later release, start warning when () is used directly on the commandline. With an even later release, start warning always. Then, when the deadline has passed, remove it. We don't want a ```` (backtick) situation.

---

Addendum:

A related thing is splitting on something else. If my proposal is used, this could at least be done with string split $CHAR "$(COMMAND)". Additionally it would be possible to make string split special in a command substitution, so $(COMMAND | string split $CHAR) splits not on newline but on $CHAR. I don't know how that would interact with quotes.

[dxlr8r]

My 2 cents. I don't like the $() approach personally, to confusing for new people coming from Bash etc. and not needed the way I see it. Add a flag to _set_ and _read_, that should be adequate.

[faho]

I don't like the $() approach personally, to confusing for new people coming from Bash

Why? With what I propose, the _only_ difference between fish's $() and bash's would be that when unquoted it splits only on newline and not "any" whitespace (I think it's actually $IFS or newline, tab and space - not non-breaking space and such). That's easily explained (and AFAIK also what zsh does by default).

[andrewrk]

My 2 cents. I don't like the $() approach personally, to confusing for new people coming from Bash

Hmm I have the other conclusion, that $() would be more familiar to people coming from Bash. I mean, it _is_ the same syntax as Bash.

[dxlr8r]

That is the problem, that could lead to people only using $(). And to be frank I don't see the point, () is fine the way it is. And this is about storing data in a different way, not to be confused by sub shelling etc. I think this belongs to _set_ and _read_ but I would use $() as well because of the functionality, but I don't think that is so clean.

[faho]

That is the problem, that could lead to people only using $().

And that is the idea. That people switch to using $(), and we can in future dump (), leaving only one syntax to do this.

Or do you mean people could come from bash, use $() expecting it to split on space? Well, in my experience 90% of splitting on space is actually a bug. IIRC some projects demand all variables in a shellscript be quoted. Zsh has the SH_WORDSPLIT option (that only applies to parameter expansion - command substitution it also splits on spaces, unlike what I mentioned previously) to reenable splitting on spaces, and it defaults to off (and is strongly discouraged).

Or do you mean that people use $() when they mean "$()"? Well, they need to decide how they want to split the argument. I can't see a way around that. If we added piping to set or an option to read (that actually took the entire input as one element, not just up to 0), then you'd need to decide if you wanted to split on newlines or not, i.e. whether you'd want COMMAND | set var or set var (COMMAND) (or read's "--the-entire-thing-as-one-please" option).

Really, you have some command that produces some kind of output. We can offer a default for how to handle it, but if you want it differently, you need to express that.

And for this there's a nice symmetry between comsubs and variable expansion - if you want it split (on newlines/elements) you don't use quotes. If you want it as one, you quote.

[dxlr8r]

I just don't see the point as I said. Why should $() take over for ()? () is a clean break from Bash syntax, which is confusing. Fish "few" options of doing things are it's strength, it does what it does well and you don't have 5 ways of writing an if statement ( test, [], (), (()), etc.).

I don't see how/why prefixing $ before () would make it better, what does the $ add to the syntax? () is cleaner, shorter and less like bash. If people want a lot of features but still be compatible with bash syntax there is zsh for that.

[faho]

I don't see how/why a prefixing $ before () would make it better, what does the $ add to the syntax? () is cleaner, shorter and less like bash

What the "$" allows is for this issue to be resolved. "$"s are already special inside double-quotes, so we can reuse that fact to allow for a syntax to command substitutions that also works in double-quotes _without increasing the amount of special characters_. This is then a natural notation for command substitutions that aren't split.

It would also allow for the future removal of () would also reduce the amount of special characters in general. That would be nice because any special character needs to handled specially - specifically they need to be escaped, via (single) quotes or \.

It also would not add any expressions that work _differently_ since $() is currently an error both unquoted and in double-quotes.

Fish "few" options of doing things are it's strength, it does what it does well and you don't have 5 ways

The idea is that providing two options is very much a temporary thing so that scripts don't need to be rewritten instantly (and with two versions, one for fish before and one for fish after this change). We'd have to communicate this properly.

[andrewrk]

() is a clean break from Bash syntax, which is confusing

$() isn't inherently confusing. IMO it's one of the cleaner parts of bash syntax.

what does the $ add to the syntax?

It allows you to execute a sub command inside a quoted string, similar to how $FOO lets you expand a variable inside a quoted string.

[dxlr8r]

I think quoting only should be for strings. And I quote the design docs:

The many Posix quoting styles are silly, especially $''.

I agree with @Soares to that we should add new commands/arguments instead of redesign. This subject can easily be solved without changing fish's core.

[andrewrk]

I agree with @Soares to that we should add new commands/arguments instead of redesign. This subject can easily be solved without changing fish's core.

I'm content with that solution as well. But just to play devil's advocate for the other solution, I'll say that I filed #3329 because fish's behavior was surprising. I expected set -x foo (cat foo.txt) to not mangle the contents of foo.txt by default. I tried putting the command in quotes but realized that without $ or another escape character it wouldn't be possible. Then I tried bash, got the expected behavior, and concluded that fish had a bug. So this would be an argument from intuitiveness of how users would expect things to work.

[alphapapa]

I like the idea of having flags for set and read to change the splitting behavior between splitting on newlines or not, but that seems like only half the solution, since command substitution wouldn't be affected by it (and we don't want to force people to capture output into a variable before passing it to a command as arguments).

I think using the $() syntax makes sense. Not only would it allow for fixing this issue by deprecating or maintaining in parallel the () syntax, but I think there's also something to be said for using the $() syntax in general: it means that substitutions always begin with a $, which helps clarity. While the () syntax is cleaner (and even lispy, if you like that), it's very different from other shells, and it's inconsistent with variable substitution. I think saving one character when typing and visual cleanliness are not as valuable as clarity and consistency.

[andrewrk]

I think this issue is important. Do we foresee "$()" syntax being available anytime soon?

[s-ol]

this is also very frustrating for me, to work with meteor sometimes I need to read a rather long settings.json into an environment variable like so:

env METEOR_SETTINGS=(cat settings.json) meteor run ....

currently I can't really do this and have to drop into bash for a bit.

[anordal]

With what I propose, the only difference between fish's $() and bash's would be that when unquoted, it splits on newline

Let's be thankful that quoting variables is not necessary in fish the same way it is in bash. As Wikipedia sums it up:

Quoting not necessary to suppress word splitting and glob interpretation.
Instead, quoting signifies serialization.

The only reason we can confidently tell bash programmers «Come to fish, here you can forget everything you know about quoting» is that quoting means something else in fish. But if we have to add «Oh, except for command substitutions», then it's less convincing.

Which is a reason for a clean break, and not making things look too familiar to bash programmers.

[andrewrk]

Which is a reason for a clean break, and not making things look too familiar to bash programmers.

I'd rather have the attitude be, let's forget bash and design a good syntax, rather than trying to be so different from bash limit our options.

[anordal]

… but isn't a good syntax one that is explicit about splitting?

[faho]

… but isn't a good syntax one that is explicit about splitting?

A good syntax is one where, if you need to write it, you know what to write, and if you need to read it, you know what it does.

Being explicit works for that, but isn't necessary, and may also be _too_ explicit.

So, why don't we want required explicit splitting syntax? Well, we _can't_ actually change the default (if no splitting is specified) because that would break the world (of fish, at least). We also wouldn't want to, because the default (splitting on newline) is the right thing to do for 90% of cases - unix is traditionally line-oriented, so handling lines separately by default is nice. Requiring an explicit "split this on newlines please" is annoying explicit syntax.

So what's "good syntax" about $()? It's consistent with $var. echo $var splits, while echo "$var" doesn't. That means if you know the latter, you can guess the former. If you need to write it, there's a good chance you'll come up with it, and if you read it, there's a good chance you'll understand it.

The only reason we can confidently tell bash programmers «Come to fish, here you can forget everything you know about quoting» is that quoting means something else in fish. But if we have to add «Oh, except for command substitutions», then it's less convincing.

No, that'd still work. Without quoting, stuff is split on a good default. If you need something else, add quotes and do what you want.

The problem with quoting in bash isn't that it exists, it's that it's a "do the right thing, please" syntax. For variables, you want quotes almost always. The default is the thing you want only in special cases, and the special syntax is what you usually want. If normal conversation were like that, you'd have to add "and don't scream the answer at me" after every question.

[s-ol]

similar to @xfix's pipeset above, here's what I came up with to solve my problem:

function withsub
  if [ (count $argv) -lt 2 ]
    echo need at least two arguments!
    return 1
  end
  set -l cmd
  set -l val (eval $argv[1])
  for arg in $argv[2..-1]
    if [ "$arg" = "[]" ]
      set cmd $cmd "$val"
      else
        set cmd $cmd "$arg"
      end
    end
  eval (string escape -- $cmd)
end

the OP problem doesn't really concern me, setting it in a variable and using the variable like "$var" solves that problem, but command substitution can't be quoted.

$ withsub "seperated a b c" seperated 1 [] 2
1
--
a -- b -- c
--
2

if you have something like

function seperated
  echo $argv[1]
  for arg in $argv[2..-1]
    echo --    
    echo $arg
  end
end

[Pysis868]

Meanwhile, I wanted a newline stored into a variable so I could throw it more cleanly into a screen -X "...$newLine" command call.

Since () wasn't cooperating, I used this, in case anyone else needs it too: printf "\n" | read -z newLine;

Thanks @dxlr8r!

[faho]

@Pysis868: In that case, set newLine \n would just work - or later screen -X "..."\n.

[andrewrk]

I just ran into this again and even though I participated in this issue 4 months ago, I was completely surprised and it took an hour to figure out what was happening. And look above this comment at these other commits referencing this issue, because as it stands fish is not intuitive.

let's acknowledge the design flaw, fix, and move on. milestone fish-future does not acknowledge the severity of this design flaw.

[krader1961]

let's acknowledge the design flaw, fix, and move on

We would love to review a pull-request from you, @andrewrk, or anyone else that implements this feature. More contributors are always welcome as there are too few contributing on a regular basis.

[spinningarrow]

@krader1961 I would be happy to contribute but I don't know where to start. Is there something about how the code is organised that I can refer to?

[krader1961]

@spinningarrow: Not really. We do have a how to contribute document. But most of that is for people who intend to contribute changes on an on going basis. If you are interested in making infrequent contributions you don't really need to worry about those guidelines since the core developers will take care of style, lint, and similar issues. The only way to learn about how the code is organized is to git clone the repo and explore it.

[GReagle]

Why would we change fish when, as @imgx64 has pointed out, it is already solved and documented as "Setting IFS to the empty string will disable line splitting."

Also, similar result can be achieved with read: ls | read -z test.

I don't agree with @faho that

While mucking with $IFS does solve the issue, it's supremely awkward

I think it just slightly awkward. But, surely, modifying IFS or using read -z can't be as awkward as changing the syntax and semantics of fish.

[krader1961]

Thanks for the feedback, @GReagle, but you're definitely in the minority. The only reason this hasn't already been done is that it requires more work than most changes and is technically not backward compatible even if we continue to support the current bare (cmd) syntax. If we do a fish 3.0 (i.e., major) release this is definitely going to be included. Even if it means not doing some other things we'd like to see in a major release.

[krader1961]

Also, it should never be necessary to manipulate IFS in fish script. That var should die and removing it is something else we should consider for a major release. Whenever I have needed to, or seen it used to, change how strings are split into words in other shells (ksh, bash, zsh) it always made me want to vomit. It is the poster child for what is wrong with POSIX 1003 compatible shells.

[spinningarrow]

I don't know much about IFS and its manipulation. Why is it bad?

[faho]

I don't know much about IFS and its manipulation. Why is it bad?

A few things:

It can only be set for an entire command, meaning e.g.

set -l var (cat file1) (cat file2)

means you can only split or not both files (or command substitutions in general) - there's no way to only split file1 other than to do it over two commands.

To use it, you either need to reset it (set -l oldifs $IFS; set -l IFS; command; set IFS $oldifs) or mess with scoping (e.g. introduce a begin; end block just for it).

It's also pulling double-duty - if IFS is unset or set to empty (and just that, the actual value doesn't matter) command substitutions won't split, and it will be used to split for read - e.g. set -l IFS =; read -l key value < file.ini to read an ini-file).

It's also kind of inconsistent with how fish does things - we don't use variables to change behavior all that much.

All that means it's easy to mess up and requires more care than usual.

It's not that it's unfit for purpose - yes, it can be used to not split command substitutions - it's that it's an awkward interface.

Or ask yourself what you like more:

set -l oldifs $IFS
set -l IFS =
somecommand | read -l key value
set IFS oldifs

# or

somecommand | read -l -f = key value # the "-f" option is hypothetical at this point

and

set -l oldifs $IFS
set -l IFS
set -l var (somecommand)
set IFS $oldifs
set var $var (somecommand2)

# or
set -l var (somecommand2)
begin
    set -l IFS
    set var (somecommand1) $var
end

# or

set -l var "$(somecommand)" $(somecommand)

Note: There are of course other ways to denote that a command's output should not be split - some things have been mentioned in this thread. But many of them would be better than awkward ifs-trickery. Also in many cases resetting IFS can be skipped - e.g. because that's the end of the function anyway.

[GReagle]

@faho thanks for that instructive explanation.

[spinningarrow]

@faho That was a great explanation - thanks!

[nesl247]

Any update to this? Not having this is extremely annoying at the moment.

[hauleth]

I would like IFS to be removed, and default to empty string, if someone wants to split command output as separate params then should use string split <pattern> without exceptions.

[lilyball]

string split only works because of IFS.

> begin
    set -l IFS
    set -l wat (string split x fooxbarxbaz)
    set -S wat
end
$wat: set in local scope, unexported, with 1 elements
$wat[1]: length=11 value=|foo\nbar\nbaz|
$wat: not set in global scope
$wat: not set in universal scope

[lilyball]

Or to put it another way, for this to work we'd have to introduce a special class of builtins that can produce multiple arguments regardless of IFS (and then figure out some way to expose this feature to fishscript as well). And removing IFS would end up breaking a truly massive amount of fishscript.

[joelmaxuel]

Maybe there is another way around this for my use case; I am not seeing one without having multi-line output support for a variable.

I use screenfetch for a (rather) quick display of system status upon opening a terminal. I originally set it up the lazy way at the bottom of config.fish:
set fish_greeting ""
echo
screenfetch

However doing it this way has ill effects when pushing files via scp to that account with it active (first couple lines of screen appears instead of progress notification; file does not copy). So I attempted to set screenfetch as the fish_greeting - scp was happy as a result, but due to lack of line breaks, my eyes are not.
set fish_greeting (screenfetch)

I tried piping string split '\n' as mentioned in issue 2490 to no avail. Also, to ensure this is not a problem unique to output of screenfetch, I tried the same methods with cat /proc/cpuinfo instead of screenfetch. The newlines get stripped every time.

[krader1961]

@joelmaxuel You should always predicate anything that should only be done for interactive shells on if status is-interactive. You can also capture multiline output without splitting it on newlines easy enough: screenfetch | read -z fish_greeting.

[mqudsi]

@faho @zanchey I think this can be closed with the deprecation of IFS, the decision not support $(...) syntax, the improvements to read -z and others?

[mhubig]

Lot of stuff to read ... what I am missing is a way to do this:

export GOOGLE_CLOUD_SQL_CREDENTIALS=(cat google-cloud-sql-credentials.json)

and beeing able to do this, would ease up the use of googled examples:

export GOOGLE_CLOUD_SQL_CREDENTIALS=$(cat google-cloud-sql-credentials.json)

[ridiculousfish]

@mhubig simplest way as of fish 3.0:

export GOOGLE_CLOUD_SQL_CREDENTIALS=(cat google-cloud-sql-credentials.json | string split0)

split0 means to split on zero bytes instead of newlines. You can make a little non-splitting helper:

function cat1 ; cat $argv | string split0 ; end
export GOOGLE_CLOUD_SQL_CREDENTIALS=(cat1 google-cloud-sql-credentials.json)

though I don't think we can honestly close this yet.

[lilyball]

Maybe we should add a string nosplit command for this? foo | string split0 is a cute hack, but won't quite do what we want if the string actually contains NULs. I realize we can't store NULs in a variable (or, apparently, pass them as an argument to another command), but it might be nice to fix that someday.

Of course, it's still kind of kludgy to say that the official way to disable newline splitting is to pipe to a magical command. I'd prefer some kind of dedicated syntax for this.

[faho]

I realize we can't store NULs in a variable (or, apparently, pass them as an argument to another command), but it might be nice to fix that someday.

Sooo.... we can't pass them as an argument because in unix arguments are passed as... NUL-terminated strings. And we also can't pass them as exported variables because in unix the export array is... NUL-terminated.

Which immediately reduces the use of NULs in a shell quite a bit. The only possiblity would be to store them internally, allow passing them to builtins (though that alone is a bit of a rearchitecturing), and then allow doing echo $var | thing. That's not massively useful because you don't often deal with "binary" data in a shell, but there's some merit there. IIRC, that's what zsh does.

But... what would nosplit do right now? It would have to either do the same as string split0 (which means lying), or keep the NULs (which means they'd truncate "automatically"), or truncate manually.

I'd probably go for keeping the NULs and just letting unix take its course.

[lilyball]

Good point about unix not letting us pass them as arguments, but in the past I have wanted to stuff a string with NULs in it into a variable in order to later pipe to a command, like

set -l foo
if condition
    set foo (cmd-emitting-NULs)
else
    set foo (other-cmd-emitting-NULs)
end
echo "$foo" | xargs -0 …

In any case, even if we never support this scenario, it's still just nonobvious that string split0 is the right thing to use, especially as that won't provide the guarantee that the resulting variable has only one entry. So I'd expect string nosplit to literally do no splitting, which would then invoke automatic NUL truncation behavior (and therefore in the future if we ever start preserving NULs then this would automatically preserve them too).

Incidentally, after having thought about it, I'm inclined to call this string collect now, because it "collects" all the lines into a single variable. This makes a bit more conceptual sense when considering something like foo=(echo $arrayVar | string collect).

[paragbaxi]

Just wanted to throw in that this impacts Splunk:

(clear; '/Applications/Splunk/bin/splunk' start || touch "/tmp/splunk_start_failed_14856"); rm "/tmp/splunk_start_running_32525" 
fish: Command substitutions not allowed

[lilyball]

@paragbaxi Your listed command is unrelated to this issue, and is simply a case of you're trying to use a command substitution as the whole command instead of as an argument, and Fish only allows it in argument position. It's not clear why you're using a command substitution at all, you could just delete the parens in the command you listed, but if you have some particular need to group the commands together then this is what begin; commands_go_here; end is for.

[lilyball]

I've gone ahead and submitted a PR for string collect as https://github.com/fish-shell/fish-shell/pull/5943. As I state in the PR, I don't think this new command should be sufficient to close this issue, as there's value in having dedicated discoverable syntax for this (a string command is not where I'd expect users to look first when trying to figure out how to capture multi-line output).

Also see #5942 wherein I argue that string join should disable implicit splitting of its output, as doing something like set contents (cat filename | string join \n) is something users might guess, and yet the string join \n ends up doing nothing at all.

[lilyball]

With the new string collect command, maybe we should just tweak the parser to detect the attempted use of $() inside double-quotes and suggest | string collect instead?

[ngortheone]

I was using Amazon's ECR the other day, and in order to log in they tell me to do

bash/zsh

$(aws ecr get-login --no-include-email --region us-east-1)

I tried this in fish:

(aws ecr get-login --no-include-email --region us-east-1)
fish: Command substitutions not allowed

But it says that command substitutions not allowed. Is there something I can do to make this work?

[lordlycastle]

@ngortheone what is the output of that function. Can you save it to file and show us?

[faho]

But it says that command substitutions not allowed. Is there something I can do to make this work?

@ngortheone: This is the entirely wrong issue for this, but here's what you do:

You're trying to run the output of a command substitution as a command. Fish doesn't allow that, but it does allow running variable contents.

So:

set -l var (aws ecr get-login --no-include-email --region us-east-1)
$var

or just

env (aws ecr get-login --no-include-email --region us-east-1)

[ngortheone]

@faho
sorry, the issue looked relevant. Is there a relevant issue open?

I tried the suggested above and it didn't work.

the command itself emits another command. Example output (redacted as it has credentials)

~ aws ecr get-login --no-include-email --region us-east-1
docker login -u AWS -p VERYVERYLONGBASE64 https://1234567890123.dkr.ecr.us-east-1.amazonaws.com

The error messages are

 env (aws ecr get-login --no-include-email --region us-east-1)
env: output_of_command_here: File name too long

Or sometimes base64 credentials end with '== ' then I get

fish: Unsupported use of '='. In fish, please use ...

[faho]

sorry, the issue looked relevant.

It is not. This is about introducing something similar to something "$(somethingelse)".

Is there a relevant issue open?

This is one of those things that would work better on gitter.

But anyway:

Unlike bash et al, fish does not split command substitutions on spaces, only newlines. You'll see that it's passing something like 'docker login -u AWS -p...' as one argument, which is why it's complaining that the filename is too long (if it wasn't, it would most likely complain that it doesn't exist).

So what you want is to split them on space, so you need to use string split

 env (aws ecr get-login --no-include-email --region us-east-1 | string split " ")

[ngortheone]

@faho It worked, thanks!

[lordlycastle]

Last related comment was over a year ago. This is still a big missing feature with no good/standard way of doing it. Are we not happy with "$(echo $var)"? Are we waiting for something that can be helped?

[lilyball]

We have string collect now, but as I stated before I think we need to add a discovery mechanism for users before we can close this issue.

[ridiculousfish]

I agree with keeping this open. We'll probably end up supporting "$(cmd)" syntax, but it's a lot of work.

First step I guess is to design it. In bash "$(cmd)" establishes a new quoting context:

echo "$(echo "nested quotes!")"

do we want to allow that, to arbitrary depth? Or can we start with something useful but more restricted, which would be easier to implement?

[krobelus]

I think we can hack the parser to close double quotes on $( and reopen them after the matching).
So echo "b$(...)c" is virtually tokenized as echo "b"$(...)"c".
This way we get nesting, syntax highlighting and completion for free - see my implementation.

The cost here is the more complex quoting syntax - a single regex will no longer be enough to correctly highlight such strings.
Probably there are ready solutions to support this kind of syntax in most 3rd party highlighters.

If we don't support quotes inside "$()", then the highlighting syntax would remain unchanged.
The nesting is less important than in bash but it feels natural to include it eventually.

Not sure what to do about unquoted $(), I feel like it should mean the same as "$()" but previous suggestions use the () meaning. We can also leave it unsupported for now.

---

Alternatively, if we want to avoid making the quoting syntax more complex, we could support unquoted $() only :

echo "no interpolation "$(echo "(but nested quotes)")" for you"

That may be less elegant but does not require to understand the concept of command substitutions inside quotes.
In situations where no quotes are required for the surrounding text (like the OP), it's also more concise.

We could try to make this as convenient to type as the bash version:
when the user types "foo$(, fish could just insert "foo"$( instead...

[jos128]

As someone rather inexperienced with shell scripting, I think it's not intuitive to see why a variable can't "simply catch the output of a command".

It seems like string collect still tampers with the input, by removing the trailing new line.

# 1
> printf "A\nB\nC\n" | od -a
0000000   A  nl   B  nl   C  nl
0000006

> set var (printf "A\nB\nC\n")
# 2 Expected (1)
> printf "$var" | od -a
0000000   A  sp   B  sp   C
0000005

> set var (printf "A\nB\nC\n" | string collect)
# 3 Expected (1)
> printf "$var" | od -a
0000000   A  nl   B  nl   C
0000005

So it seems, string collect replicates $() problems. This gets really messy, if you try to catch something with CRLF, like an HTTP header from curl. I really, really like fish, but this issue makes it very impractical for scripting.

[faho]

It seems like string collect still tampers with the input, by removing the trailing new line.

@jos128: string collect -N, if you really must have the trailing newline.

In many many many cases you don't.

[jos128]

In many many many cases you don't.

@faho: That may be true, but it is not what's naively expected. Personally, I think stripping the last newline should be the option. As I said, it gets really unintuitive with CRLF, as that prints fine itself, but is very obfuscated, if processed in a terminal. E.g.

> set var (curl "https://example.com" -D - -s -o /dev/null)
> printf "$var"
> printf "$var" | grep server
 content-length: 1256g)

Of course, in this case string collect does the trick, but I hope you see how this is not expected. Especially since it works just fine in bash. Of course, with HTTP headers, stripping the last newline may cause even more significant, but hidden problems downstream.

Sorry, for being OT. I thought the perspective of someone who doesn't take any internals for granted might be of value for design decisions. To me this behaviour of fish seems arcane, so as an outsider it appears to not match fish's design pattern.