PowerShell should support passing verbatim args

[bitcrazed]

Following a conversation with @SteveL-MSFT where he confirmed I am not (entirely) insane 😜

PowerShell currently makes if VERY difficult to

1. Specify that an arbitrary set of command-line args gets passed to an exe, verbatim
2. That the same command-line also supports PowerShell command chaining, etc.

For example, trying to pass a Linux command-line cd / && ls . | cowsay to wsl in PowerShell 7+:

wsl cd / && ls .

Expected: Directory listing of the root of the users' default Linux distro
Actual: Error reporting that cowsay is not a recognized command:

Note: to perform the above command, you'll need to enable WSL and a Linux distro. Within your default distro, you'll also need to install cowsay via your distro's package manager. For Ubuntu and Debian, you can do this via sudo apt install cowsay

As can be seen from the syntax color highlighting, PowerShell is parsing && as a PowerShell command continuation, not as a part of the command to be passed to wsl.

So, how about wrapping the args in quotes?

Doesn't work because that would make arg[1] 'cd / && ls . | cowsay' which wouldn't make sense to bash.

@SteveL-MSFT recommended prefixing the command-line args with --% to indicate that PowerShell should pass the remaining command-line verbatim. Alas, no joy here - notice the syntax highlighting which indicates that the parser thinks the && is a PowerShell command which, in this case, it is not!

Boo! 😟

Okay, so how about escaping the && with a ````` (backtick)?

Oh! PowerShell thinks that I meant to spawn the remaining command as a job. NOPE!

How about escaping each &?

Closer!

Now, how about also escaping the pipe operator:

HUZZAH! 🙌

Escaping many symbols in lengthy command-lines could get tedious pretty quickly, especially when pasting more lengthy & complex bash command-lines into WSL, or complex page splits into wt (Windows Terminal):

While --% may be useful (when fixed) in some cases to pass the remainder of __this__ command-line to the tool being executed, commands couldn't be chained.

So, ideally, PowerShell should (also) provide a way of escaping a specific portion of a command-line to the preceding tool, but then continue treating the remaining command-line as PowerShell script.

Perhaps enclosing the command-line segment in some of the following delimiters would work?

• Pointy-braces: < ... >, e.g. wsl <cd / && ls . | cowsay>
• Square-braces: [ ... ], e.g. wsl [cd / && ls . | cowsay]
• Curly-braces: { ... }, e.g. wsl {cd / && ls . | cowsay}
• Double-backticks
• Arrow: <-- ... --<<< , e.g. wsl <-- cd / && ls . | cowsay --<<< 😜

... but will leave the choice of the enclosing delimiters to the language designers 😁

[mklement0]

PowerShell currently makes if VERY difficult

PowerShell is a _shell_ - just like cmd.exe and bash are.

This means: you need to understand and accommodate the running shell's syntax rules, and you need to understand that if you submit something like:

wsl cd / && ls . | cat -n

that any _unquoted_ token is subject to _PowerShell_'s syntax rules; specifically, && is interpreted as _PowerShell_'s pipeline-chaining operator, and | is interpreted as PowerShell's pipeline operator.

---

Additionally, you need to understand the external wls.exe program's syntax requirements: it can execute a _WSL_ shell command passed _as individual arguments_, which means that you must _quote_ arguments selectively in order to _pass them through_ verbatim to wsl.exe:

wsl cd / '&&' ls . '|' cat -n

---

The alternative is to explicitly pass an _entire command line_ _as a single string_ to the default WSL shell, sh, via its -c option:

wsl sh -c 'cd / && ls . | cat -n'

---

There is nothing that PowerShell can reasonably do to ease this perceived pain; that the ill-fated and Windows-focused --% doesn't help in this case is no surprise and serves to underscore that point.

Given that an understanding of the syntax requirements allows you to solve the problem, I don't think any new syntax is called for.

[rjmholt]

Yes, after discussing with @SteveL-MSFT, this behaviour seems to be quite explicitly coded for:

https://github.com/PowerShell/PowerShell/blob/a3319d7580dabbae6a9bb06a16771fc6f6f1d4ff/src/System.Management.Automation/engine/parser/tokenizer.cs#L2311-L2315

Alas, no joy here - notice the syntax highlighting which indicates that the parser thinks the && is a PowerShell command which, in this case, it is not!

To be clear, the verbatim argument parameter is terminated by |, || and &&. Perhaps the && behaviour is derived from the earlier idea that && and || are statement separators, but, | still terminates the arguments in Windows PowerShell, meaning the current behaviour is totally consistent with that.

As @mklement0, the solution is to quote the &&.

One thing I've noticed though is that in the sparse documentation of --%, it only ever mentions terminating at newlines. @mklement0 are you aware of any other documentation (including 3rd party) that states otherwise?

It's not really clear to me why this choice was made, but even constrained in scope as it is, I think we'd need to have very strong motivation to break it.

Additionally, you need to understand the external wls.exe program's syntax requirements

I imagine you're making a general statement, but I suspect @bitcrazed is reasonably familiar with those 🙂

[mklement0]

are you aware of any other documentation (including 3rd party) that states otherwise?

I don't think it's covered in the official documentation, but I've summarized the behavior and limitations in this Stack Overflow answer; there's also this one, which focuses on Unix-like platforms.

I suspect @bitcrazed is reasonably familiar with those 🙂

Well, I wasn't, until I took a closer look: wsl.exe truly requires _individual_ arguments, and doesn't (also) support passing the pass-through shell command _as a single string_ (command line - except if you explicitly use sh -c / bash -c), which differs from pwsh -c, which accepts _either_ form (though I do consider that problematic - see https://github.com/PowerShell/PowerShell/issues/4024#issuecomment-311541803 and https://github.com/PowerShell/PowerShell/pull/3963#issuecomment-307392273).

[bitcrazed]

@mklement0 Yes, I am very familiar with shells and shell syntax.

The problem is that the lack of an ability to delimit a portion of a command-line to be passed varbatim to the receiving command/script is something that trips users up all the time.

The stop parsing symbol (which, by the way is practically impossible to search for) is only documented here (as far as I could find) and states (emphasis mine):

When calling an executable program in PowerShell, place the stop-parsing symbol before the program arguments. This technique is much easier than using escape characters to prevent misinterpretation.

When it encounters a stop-parsing symbol, PowerShell treats the remaining characters in the line as a literal. The only interpretation it performs is to substitute values for environment variables that use standard Windows notation, such as %USERPROFILE%.

The stop-parsing symbol is effective only until the next newline or pipeline character. You cannot use a continuation character (`) to extend its effect or use a command delimiter (;) to terminate its effect.

This is a perfect example of why I believe we need some kind of enclosure, rather than expecting users to escape their command-line actions correctly: Stop parsing up until some incorrectly/poorly documented arbitrary char/sequence seems to defeat the point of "stop parsing".

PowerShell is awesome, but has some major impediments and barriers to adoption that need attention. Heck, this issue tripped-up me (owner of Cmd, PM for WSL & Terminal) and @SteveL-MSFT (owner of PowerShell) in an interactive WTH session yesterday afternoon.

And this isn't just an issue that affects WSL: It also affects Windows Terminal's wt command-line invocations, and many other tools. Expecting users to jump through hoops to figure out that they have to manually escape complex commands is only going to perpetuate the belief of many that PowerShell is hostile to interactive use and will drive people away from using it for this purpose (see this thread for examples). Which would be a dreadful shame.

[mklement0]

@bitcrazed

Yes, I am very familiar with shells and shell syntax.

Then you are undoubtedly familiar with the need to escape or quote the calling shell's metacharacters if you want to _pass them through_ (use them _verbatim_).

Doing so gives us the aforementioned solution:

PS> wsl cd / '&&' ls . '|' cat -n
# Alternative, with escape char.
PS> wsl cd / `&`& ls . `| cat -n

Incidentally, you would in essence _normally_ have to do the same if you called from cmd.exe (the only difference being the need for _double_ quoting) - but, as it turns out - that _doesn't even work_ in this case, because of how wsl.exe parses the command line:

:: BREAKS! because `wsl` apparently doesn't strip the double quotes and `bash` sees them
:: verbatim - yet you need the double quotes in order to pass && and | through.
C:\>wsl cd / "&&" ls . "|" cat -n

However, using the escape character (^ in cmd's case) works: wsl cd / ^&^& ls . ^| cat -n

--% was specifically introduced for verbatim reuse of command lines written for cmd.exe, i.e. for accommodating the syntax _of a different shell_.

This invariably entailed problematic compromises, and, in my opinion, --% should never have been implemented.
Now that PowerShell is cross-platform, --% is even more problematic, because you cannot (meaningfully) use it with command lines written for bash / sh (POSIX-like shells) - no support for single-quoted strings, no support for globbing and other shell expansions, no support for interpreting something such as $HOME as an (environment) variable.

Focusing just on the cmd.exe-related compromises:

• no ability to reference _PowerShell_ variables and expressions on the command line
• no ability to use _redirections_
• focused on a _single_ command - which is why parsing stops at | (and therefore ||) and && (even though the latter wasn't even implemented at the time) - though _not_ at ;
  
  
  
  • Yes, the documentation is lacking; I've created https://github.com/MicrosoftDocs/PowerShell-Docs/issues/6149 to suggest fixing this.
  
  

---

The underlying challenge is that PowerShell has _additional_ metacharacters, and that one of them - the escape character - differs from that of legacy shells.

• With the exception of \ vs. `, which is an unfortunate consequence of the historical decision to use \ in paths on Windows), these additional metacharacters are the unavoidable price to pay for the added power and flexibility of the PowerShell command line (splatting, expressions in (...), array literals, hash table literals, ...)

  • If you want to harness this power, there is simply no way around understanding PowerShell's unique syntax requirements.

• This means that command lines _written for other shells_ cannot (generally) work - which is definitely unfortunate, but unavoidable.

I've previously summarized the issue in https://github.com/PowerShell/PowerShell/issues/1995#issuecomment-640711192, where I suggest an approach other than --%:

The introduction of a Invoke-NativeShell (ins) cmdlet that calls the platform-native shell with a single command string; use of a here-string, if needed, can be used to pass a command line verbatim.

As an aside: the linked issue is primarily about a truly broken aspect of PowerShell, which has to date not been fixed, because the fix would be a severe breaking change: how arguments with _embedded_ double quotes are passed to external programs.

---

If wsl.exe (also) accepted a _single_ command string to effectively pass to the -c option of the user's WSL default shell (bash, by default) - which I think it should - a single-quoted [here-]string would be the simple solution to the problem; if you know what the default shell is, you can therefore simply do:

# OK - command string is passed through as a literal
PS> bash.exe -c 'cd / && ls . | cat -n'

Note that you may optionally combine this approach with up-front interpolation by PowerShell, so as to embed PowerShell variables / expressions, via "..." - something that --% doesn't support.

In a manner of speaking, the single-quoted [here]-strings then provide the enclosure you're looking for.

This approach also allows you to call from cmd.exe (with double quotes):

:: OK
C:\>bash.exe -c "cd / && ls . | cat -n"

---

In cases where you do need to pass arguments verbatim _individually_, PowerShell offers another method, albeit not a very convenient one: _array-based splatting_:

# OK
PS> wsl ('cd', '/', '&&', 'ls', '.', '|', 'cat', '-n')

In this _simple_ case you can ease the pain a bit, but it's not exactly obvious (and won't work with arguments with embedded spaces):

# OK
PS> wsl (-split 'cd / && ls . | cat -n')

However, I would expect that the far more typical case will be where calling the platform-native shell via a single command string is feasible, in which case Invoke-NativeShell or explicit invocation (e.g., cmd /c '...' or sh -c '...') would work.

---

To summarize:

• PowerShell having additional metacharacters and a different escape character than other shells is unavoidable; the additional metacharacters give PowerShell its power and flexibility.

• --% has always been problematic, and its lack of (meaningful) support on Unix-like platforms makes it even more so.

• A cleaner alternative is to make calls to other shells explicit, via single-quoted [here-]strings, using cmd /c '...', sh -c '...', or - once implemented - Invoke-NativeShell / ins - or via an array constructed from single-quoted strings.

• Separately, PowerShell's broken handling of embedded double quotes must be fixed - see #1995.

---

I definitely understand the pain of the issue, but I also think that --% isn't the answer - although fixing the docs should help.

As PowerShell grows in popularity, the differing syntax needs will become more widely known, and the pain of not knowing how to modify command lines written for other shells / not understanding why there is even a need to will lessen.

To me, the best we can do is:

• Be clear in the docs (@sdwheeler just helpfully added two other links to https://github.com/MicrosoftDocs/PowerShell-Docs/issues/6149):

  • Fix about_Parsing: https://github.com/MicrosoftDocs/PowerShell-Docs/issues/6149
  • Introduce a conceptual help topic about calling external programs: https://github.com/MicrosoftDocs/PowerShell-Docs/issues/5152
  • At least for now _document_ the broken handling of embedded " chars: https://github.com/MicrosoftDocs/PowerShell-Docs/issues/2361 (the subject of #1995)

• Implement the aforementioned Invoke-NativeShell (ins) cmdlet.

[TSlivede]

@bitcrazed

The problem is that the lack of an ability to delimit a portion of a command-line to be passed varbatim to the receiving command/script is something that trips users up all the time.

What do you mean by "portion of a command-line to be passed varbatim"?

Do you mean

1. pass some sequence of characters verbatim to the called executable, such that the called executable has this sequence of characters in one element of its argument array (e.g. those characters are then available in argv[1] in main)

Or do you mean

1. insert some sequence of characters verbatim into the lpCommandLine parameter of CreateProcess

---

If you mean (1.), then it essentially already works:

PS /home/User> /bin/echo 'cd / && ls . | cowsay'
cd / && ls . | cowsay
PS /home/User>

(Except for the problem of embedded quotes as discussed in https://github.com/PowerShell/PowerShell/issues/1995)

One could argue, that adding a one-line-here-string would improve some usecases, but I think, that's not really the point of this issue.

As this does already work more or less, I assume, you meant (2.)

---

If you mean (2.), then let me state my opinion on that in a somewhat dramatic way:

Please please please don't add special syntax for this. This is basically what --% tried to do, which also should have never ever been implemented.

Why am I so strongly against this?

1. It is a Windows only problem, so adding syntax would mean that powershell on Windows has different syntax than on Linux. (Or the syntax would be supported but is totally meaningless, as it is currently the case for --%)

2. If the main commandline shell on Windows published by Microsoft adds a first-class feature (via special syntax opposed to via a cmdlet) to call executables that don't follow the typical commandline parsing rules (if the tool follows the typical rules, you don't need (2.), you can usually better use (1.)), then that encourages authors of command line tool, to not follow these rules, which only worsens the "Windows command line anarchy". The less people follow the typical rules, the harder it is to programmaticly call external executables or generally write cross platform code, so I definitely think, that program authors should be encouraged to follow those typical rules.

3. I strongly doubt, that this is a common use-case

   And this isn't just an issue that affects WSL: It also affects Windows Terminal's wt command-line invocations, and many other tools.

   Could you add some examples, where such problems occur? Because in case of WSL, I'd say that WSL's parsing of the commandline is simply broken (issue was about bash.exe but situation is by default not better with wsl.exe) in the default case - I'd consider every tool, that doesn't follow the typical commandline parsing rules broken, but WSL's default behavior is IMHO not even properly documented...

   I said the "default" behavior of wsl.exe is broken - while writing this response, I noticed, that wsl.exe actually seems to behave as expected, when using -e:

PS C:\> wsl -e bash -c 'cd / && ls . | cowsay'
 _______________________________________
/ acct bin boot cache cygdrive data dev \
| etc home init lib lib64 lost+found    |
| media mnt opt proc root run sbin snap |
\ srv sys tmp usr var                   /
 ---------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||
PS C:\>

So the only thing missing for this explicit usecase is IMO a parameter to wsl.exe to call the default shell with the commandline arguments parsed as in any other normal exe.

[TSlivede]

So, how about wrapping the args in quotes?

Doesn't work because that would make arg[1] 'cd / && ls . | cowsay' which wouldn't make sense to bash.

To bash it actually makes perfect sense, to pass cd / && ls . | cowsay as arg[2]:

$ pwsh
PowerShell 7.0.0
Copyright (c) Microsoft Corporation. All rights reserved.

https://aka.ms/powershell
Type 'help' to get help.

PS /home/User> bash -c 'cd / && ls . | cowsay'
 _______________________________________
/ acct bin boot cache cygdrive data dev \
| etc home init lib lib64 lost+found    |
| media mnt opt proc root run sbin snap |
\ srv sys tmp usr var                   /
 ---------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||
PS /home/User>

It's not bash that can't cope with cd / && ls . | cowsay as an argument but wsl.exe.

[TSlivede]

If https://github.com/PowerShell/PowerShell/issues/13068 was implemented (and I hope it will not - at least not as yet another spedial syntax/operator), I guess this issue would be resolved with that.

[yecril71pl]

It's not bash that can't cope with cd / && ls . | cowsay as an argument but wsl.exe.

wsl.exe does the right thing here, although it should be noted that in order to achieve that result, it uses nonstandard command line processing. We would not expect single quotes to work this way for a Windows program.

[mklement0]

@TSlivede's point was that you currently cannot pass the meant-for-bash command line _as a single argument_ to wsl.exe, which would make the most sense.

Aside from that, wsl.exe does _not_ itself recognize '...' quoting - it is PowerShell that translates _its_ '...' quoting into "..." quoting _behind the scenes_ (You'll see that true '...' quoting doesn't work if you call wsl 'cd / && ls . | cat -n' _from cmd.exe_).

(Recognizing '...'-quoting on Windows is definitely _unusual_ (ruby supports it, for instance), but you cannot call it _nonstandard_, because _there is no standard_, only widely adhered-to conventions. It is ultimately up to _each executable_ to define the rules - that's the anarchy that is argument-passing on Windows).

[yecril71pl]

@TSlivede's point was that you currently cannot pass the meant-for-bash command line _as a single argument_ to wsl.exe, which would make the most sense.

wsl.exe does not support passing the command line for bash as a single argument. This is by design.

Arguments to run Linux binaries:

If no command line is provided, wsl.exe launches the default shell.

--exec, -e %CommandLine%
    Execute the specified command without using the default Linux shell.

--
    Pass the remaining command line as is.

My understanding is that -- means "Here be dragons" and the spell WSL -- -c "cd / && ls . | cowsay" should work. It does not—but it is not our problem.

[mklement0]

This is by design.

By _limiting_ design, because not only would passing the command line as a single string ease the escaping woes that prompted this issue, it would allow full access to bash's CLI. For instance:

# Note: The following does NOT work with `bash.exe` on Windows, but works *from inside WSL*
# (and from POSIX-like shells on Unix-like platforms)
$ bash -c 'echo $# args; printf "%s!" $@' - one two
2 args
one!two!

That is, bash expects the 1st argument following -c to be the _ad-hoc script_ (command line) to execute, with all subsequent arguments to be passed _to that ad-hoc script_ (via $*/ $@), as _verbatim_ arguments, starting with $0(!).

Neither wsl.exe nor bash.exe support such invocations, albeit for different reasons: wsl.exe doesn't permit passing the ad-hoc script as a distinct argument - _all_ arguments become the ad-hoc script - whereas while bash.exe with -c does accept the first argument as a command line / ad-hoc script, it seemingly ignores all remaining arguments (unlike true Unix bash executables).

---

Note that, by contrast, with -e it _is_ appropriate to use _individual_ arguments, because then by definition _no shell_ is involved, and it is the _calling_ shell's responsibility (PowerShell, in our case) to parse the arguments.

---

-- is again designed for _individual_ arguments, to unambiguously mark the start of the arguments to pass through to bash. Given that all of wsl.exe's _own_ options must be specified _before_ the pass-through command parts anyway, this is probably rarely, if ever needed:

Hypothetically, if you had a command named, say, literally -u available in your bash session, you'd need to invoke it with wsl.exe -- -u, but _that doesn't actually work_, because wsl.exe seemingly _ignores_ such command names _except if they are quoted_ (wsl.exe -- -u fails with /bin/bash: -c: option requires an argument)

_Update_: It is bash itself that doesn't recognize -u as a command in this case, and the proper solution is to use
bash -c -- -u, which, however, can _not_ be called as the seemingly equivalent wsl.exe -- -- -u - see below.

[yecril71pl]

wsl.exe -- -u does not work because the spell bash -c '-u' is invalid according to bash itself. wsl.exe does not eat up anything. wsl.exe -- command -- -u is what you want to cast. See, ma’am, no quotes! 😍
Your other use case is covered by wsl.exe -- set one two ^&^& echo $# args ^&^& printf %s! $@.
I agree that what happens after wsl.exe -- defies common sense. But it is workable—and it is not our problem.

[mklement0]

wsl.exe -- -u does not work because the spell bash -c '-u' is invalid according to bash itself.

Good point, I've also corrected the comment above. The reason is that bash cannot tell what was _originally_ quoted, in the context of the _calling_ shell; both bash -c -u and bash -c '-u' (on Unix) result in bash seeing the _same_ two verbatim arguments: -c and -u.

However, bash itself also supports -- as a disambiguation mechanism (as all POSIX-compatible CLIs should): what follows it is an _operand_, even if it _looks_ like an option; therefore, it is the following form that works and is simplest (no quoting needed (also works with bash.exe):

# The proper way to make bash treat -u as an operand (the script), not as an option.
bash -c -- -u

Alas, this again _doesn't_ work via wsl.exe (_update_ - see below for the reason):

PS> wsl.exe -- -- -u
/bin/bash: --: invalid option

So, yes, your _workaround_ via the command Bash builtin is required.

Similarly:

Your other use case is covered by wsl.exe -- set one two ^&^& echo $# args ^&^& printf %s! $@

Kudos for figuring out a _workaround_.

---

I'm pretty sure we're on the same page, but just to spell it out:

These workarounds are (a) far from obvious and (b), more importantly, _shouldn't be necessary_.

The problems discussed are the problems of wsl.exe and bash.exe _themselves_, not PowerShell.

The proper solution is to allowing passing of an ad-hoc script / command line _as a single string_, while also supporting additional arguments to pass to the former.

In concrete terms, without breaking backward compatibility, this could mean:

• Add a -c option to wsl.exe that behaves as bash -c does (and other major POSIX-compatible shells do).

• Make bash.exe with the -c option properly pass _additional_ arguments to the ad-hoc script (first argument).

[yecril71pl]

Alas, this again _doesn't_ work via wsl.exe (not sure why)

wsl.exe -- -- -u invokes bash -c '-- -u', which is equivalent to bash -c -- '- ' -- -u.