Powershell: Arguments for external executables aren't correctly escaped

[be5invis]

Steps to reproduce

1. write a C program native.exe which acquires ARGV
2. Run native.exe "`"a`""
   

   Expected behavior

ARGV[1] == "a"

Actual behavior

ARGV[1] == a

Environment data

Windows 10 x64

Name                           Value
----                           -----
PSVersion                      5.1.14393.0
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.14393.0
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

[andschwa]

Why do you think that "\"a\"" the expected behavior? My understanding of PowerShell escapes says that the actual behavior is the correct and expected behavior. ""a"" is a pair of quotes surrounding an escaped pair of quotes surrounding an a, so PowerShell interprets the outer unescaped pair as "this is a string argument" and so drops them, then interprets the escaped pair as escaped quotes and so keeps them, leaving you with "a". At no point was a \ added to the string.

The fact that Bash uses \ as an escape character is irrelevant. In PowerShell, the escape character is a backtick. See PowerShell escape characters.

If you want to pass literally "\"a\"", I believe you would use:

> echo `"\`"a\`"`"
"\"a\""

[be5invis]

@andschwa
Yes, escapes works fine for internal cmdlets, but things get weird when communicate with native binaries, especially on Windows.
When running native.exe ""a"", the ARGV[1] should be

"a"

(three characters)

instead of

a

(one character).

[be5invis]

Currently to make native.exe correctly receive an ARGV with two quotes and an a character, you have to use this weird call:

native.exe "\`"a\`""

[andschwa]

Ah, I see. Re-opening.

[andschwa]

Out of a strong curiosity, what happens if you try a build using #1639?

[be5invis]

@andschwa The same. You HAVE to double-esacpe to satisify both PowerShell and CommandLineToArgvW. This line:

native.exe "`"a`""

results a StartProcess equalivent to cmd

native.exe ""a""

[andschwa]

@be5invis @douglaswth is this resolved via https://github.com/PowerShell/PowerShell/pull/2182?

[be5invis]

No, We still need to add a backslash before a backtick-escaped double quote? This does not solve the double-escaping problem. (That is, we have to escape a double quote for both PowerShell and CommandLineToArgvW.)

[vors]

Since ""a"" is equal to '"a"', do you suggest that native.exe '"a"' should result in "\"a\""?

[douglaswth]

This seems like a feature request that if implemented could break a large number of already existing PowerShell scripts that use the required double escaping, so extreme care would be required with any solution.

[be5invis]

@vors Yes.
@douglaswth The double-escaping is really silly: why do we need the “inner” escapes made in the DOS era?

[be5invis]

@vors @douglaswth
This is a the C code used to show GetCommandLineW and CommandLineToArgvW results:

#include <stdio.h>
#include <wchar.h>
#include <Windows.h>

int main() {
  LPWSTR cmdline = GetCommandLineW();
  wprintf(L"Command Line : %s\n", cmdline);

  int nArgs;
  LPWSTR *szArglist = CommandLineToArgvW(cmdline, &nArgs);
  if (NULL == szArglist) {
    wprintf(L"CommandLineToArgvW failed\n");
    return 0;
  } else {
    for (int i = 0; i < nArgs; i++) {
      wprintf(L"argv[%d]: %s\n", i, szArglist[i]);
    }
  }
  LocalFree(szArglist);
}

[be5invis]

Here is the result

$ ./a "a b"
Command Line : "Z:\playground\ps-cmdline\a.exe" "a b"
argv[0]: Z:\playground\ps-cmdline\a.exe
argv[1]: a b

$ ./a 'a b'
Command Line : "Z:\playground\ps-cmdline\a.exe" "a b"
argv[0]: Z:\playground\ps-cmdline\a.exe
argv[1]: a b

$ ./a 'a"b'
Command Line : "Z:\playground\ps-cmdline\a.exe" a"b
argv[0]: Z:\playground\ps-cmdline\a.exe
argv[1]: ab

$ ./a 'a"b"c'
Command Line : "Z:\playground\ps-cmdline\a.exe" a"b"c
argv[0]: Z:\playground\ps-cmdline\a.exe
argv[1]: abc

$ ./a 'a\"b\"c'
Command Line : "Z:\playground\ps-cmdline\a.exe" a\"b\"c
argv[0]: Z:\playground\ps-cmdline\a.exe
argv[1]: a"b"c

[douglaswth]

@be5invis I do not disagree with you about the double escaping being annoying, but I am merely saying that a change to this would need to be backward compatible with what existing PowerShell scripts use.

[be5invis]

How many are them? I do not think there are script writers know about such double-quoting. It is a bug, not feature, and it is not documented.

???? iPhone

? 2016?9?21??01:58?Douglas Thrift <[email protected][e-mail]> ???

@be5invishttps://github.com/be5invis I do not disagree with you about the double escaping being annoying, but I am merely saying that a change to this would need to be backward compatible with what existing PowerShell scripts use.

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/PowerShell/PowerShell/issues/1995#issuecomment-248381045, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAOp20f_W0mTl2YiJKi_flQBJUKaeAnLks5qsB7ZgaJpZM4JpVin.

[douglaswth]

PowerShell has been around for 9 years so there are very likely a good number of scripts out there. I found plenty of information about the need for double escaping from StackOverflow and other sources when I ran into the need for it so I don't know if I agree with your claims about nobody knowing about the need for it or that it is not documented.

[vors]

For the additional context, I'd like to talk a little bit about the implementation.
PowerShell calls .NET API to spawn a new process, which calls a Win32 API (on windows).

Here, PS creates StartProcessInfo that is uses
https://github.com/PowerShell/PowerShell/blob/master/src/System.Management.Automation/engine/NativeCommandProcessor.cs#L1063

The provided API takes a single string for arguments and then it's re-parsed into an array of arguments to do the execution.
The rules of this re-parsing are not controlled by PowerShell. It's a Win32 API (and fortunately, it consistent in dotnet core and unix rules).
Particularly, this contract describes the \ and " behavior.

Although, PowerShell may try to be smarter and provide a nicer experience, the current behavior is consistent with cmd and bash: you can copy native executable line from them and use it in powershell and it works the same.

@be5invis If you know a way to enhance the expirience in non-breaking way, please line up the details. For the breaking changes, we would need to use RFC process, as described in https://github.com/PowerShell/PowerShell/blob/master/docs/dev-process/breaking-change-contract.md

[TSlivede]

This applies to Windows, but when running commands on Linux or Unix, its strange that one needs to double escape quotes.

On Linux processes don't have a single commandline but instead an array of arguments.
Therefore arguments in powershell should be the same as those, that are passed to the executable, instead of merging all arguments and then resplitting.

Even on windows, the current behavior is inconsistent:
If an argument contains no spaces, it is passed unchanged.
If an argument contains spaces, if it will be surrounded by quotes, to keep it together through CommandLineToArgvW call. => Argument is changed to meet CommandLineToArgvW requirement.
But if argument contains quotes, those are not escaped. => Argument is not changed, although CommandLineToArgvW requires this.

I think arguments should either never be changed, or always be changed to meet CommandLineToArgvW requirements, but not in half of the cases.

Regarding breaking-the-contract:
As I couldn't find any official documentation about double escaping, I'd consider this as category "Bucket 2: Reasonable Grey Area", so there are chances to change this, or am I wrong?

[be5invis]

@vors This is extremely annoying if your argument is an variable or something else: you have to manually escape it before sending it into a native app.
An "auto-escaping" operator may help. like ^"a"" -> "a\""`

[vors]

I think @TSlivede put it right with the inconsistency in the behavior.

I think arguments should either never be changed, or always be changed to meet CommandLineToArgvW requirements, but not in half of the cases.

I'm not sure about the bucket, but even the "clearly breaking change" bucket could potentially be changed. We want to make PowerShell better, but backward compatibility is one of our highest priorities. That's why it's not so easy.
We have a great community and I'm confident that we can find consensus.

Would anybody want to start an RFC process?

[lzybkr]

It would be worth investigating the use of P/Invoke instead of .Net to start a process if that avoids the need for PowerShell to add quotes to arguments.

[vors]

@lzybkr as far as I can tell, PInvoke would not help.
And this is where unix and windows APIs are different:

https://msdn.microsoft.com/en-us/library/20y988d2.aspx (treats spaces as separators)
https://linux.die.net/man/3/execvp (doesn't treat spaces as separators)

[lzybkr]

I wasn't suggesting changing the Windows implementation.

[vors]

I'd try to avoid having platform-specific behavior here. It will hurt scripts portability.
I think we can consider changing windows behavior in a non-breaking way. I.e. with preference variable. And then we can have different defaults or something like that.

[lzybkr]

We're talking about calling external commands - somewhat platform dependent anyway.

[TSlivede]

Well, i think it can't be really platform independent, as Windows and Linux just have different ways to call executables. In Linux a process gets an argument array while on Windows a process just gets a single commandline (one string).
(compare the more basic
CreateProcess -> commandline (https://msdn.microsoft.com/library/windows/desktop/ms682425)
and
execve -> command array (https://linux.die.net/man/2/execve)
)

As Powershell adds those quotes when arguments have spaces in them, it seems to me, that powershell tries** to pass the arguments in a way, that CommandLineToArgvW splits the commandline to the arguments that were originally given in powershell. (This way a typical c-program gets the same arguments in its argv array as a powershell function gets as $args.)
This would perfectly match to just passing the arguments to the linux systemcall (as suggested via p/invoke).

** (and fails, as it doesn't escape quotes)

PS: What is necessary to start an RFC process?

[lzybkr]

Exactly - PowerShell tries to make sure CommandLineToArgvW produces the correct command and _after_ reparsing what PowerShell has already parsed.

This has been a longstanding pain point on Windows, I see on reason to bring that difficulty over to *nix.

To me, this feels like an implementation detail, not really needing an RFC. If we changed behavior in Windows PowerShell, it might warrant an RFC, but even then, the right change might be considered a (possibly risky) bug fix.

[TSlivede]

Yes, I also think, that changing it on Linux to use a direct system call would make everyone feel more happy.

I still think it should also be changed on windows,
(Maybe by adding a preference variable for those who don't want to change their scripts)
because it's just wrong now - it is a bug. If this was corrected, a direct syscall on linux wouldn't even be necessary, because any argument would reach the next process unchanged.

But as there are executables, that split the commandline in a way, incompatible to CommandLineToArgvW, I like @be5invis's idea of an operator for arguments - but I wouldn't create an auto-escape operator (should be default for all arguments), but instead add an operator to not escape an argument (add no quotes, don't escape anything).

[rkeithhill]

This issue just came up for us today when someone tried the following command in PowerShell and was dissing PowerShell when it didn't work but CMD did:

wmic useraccount where name='username' get sid

From PSCX echoargs, wmic.exe sees this:

94> echoargs wmic useraccount where name='tso_bldadm' get sid
Arg 0 is <wmic>
Arg 1 is <useraccount>
Arg 2 is <where>
Arg 3 is <name=tso_bldadm>
Arg 4 is <get>
Arg 5 is <sid>

Command line:
"C:\Users\hillr\Documents\WindowsPowerShell\Modules\Pscx\3.2.2\Apps\EchoArgs.exe" wmic useraccount where name=tso_bldadm get sid

So what API does CMD.exe use to invoke the process / form the command line? For that matter, what does --% do to make this command work?

[be5invis]

@rkeithhill CreateProcessW. direct call. really.

[cspotcode]

Why is Powershell behaving differently in these two situations? Specifically, it is inconsistently wrapping args containing spaces in double-quotes.

# Desired argv[1] is 4 characters: A, space, double-quote, B
$ .\echoargs.exe 'A \"B'
<"C:\test\echoargs.exe" "A \"B">
<A "B>
# Correct!

# Desired argv value is 4 characters: A, double-quote, space, B
$ .\echoargs.exe 'A\" B'
<"C:\test\echoargs.exe" A\" B>
<A"> <B>
# Wrong...

There seems to be no rhyme or reason. In the first situation, it wraps my arg with double-quotes, but in the second situation it doesn't. I need to know exactly when it will and won't wrap in double-quotes so that I can manually wrap (or not) in my script.

.echoargs.exe is created by compiling the following with cl echoargs.c

// echoargs.c
#include <windows.h>
#include <stdio.h>
int wmain(int argc, WCHAR** argv) {
    wprintf(L"<%ls>\n", GetCommandLineW());
    for(int i = 1; i < argc; i++) {
        wprintf(L">%s< ", argv[i]);
    }
    wprintf(L"\n");
}

EDIT: Here's my $PSVersionTable:

Name                           Value
----                           -----
PSVersion                      5.1.15063.296
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.15063.296
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

[TSlivede]

The behaviour regarding quotes changed multiple times, therefore I'd suggest to use something like this:

Edit: Updated form below

Old version:

# call helper

function Run-Native($command) {
    $env:commandlineargumentstring=($args | %{'"'+ ($_ -replace '(\\*)"','$1$1\"' -replace '(\\*)$','$1$1') + '"'}) -join ' ';
    & $command --% %commandlineargumentstring%
}

# some test cases

Run-Native .\echoargs.exe 'A "B' 'A" B'
Run-Native .\echoargs.exe 'A "B'
Run-Native .\echoargs.exe 'A" B'
Run-Native .\echoargs.exe 'A\" B\\" \'

Output:

<"C:\test\echoargs.exe"  "A \"B" "A\" B">
<A "B> <A" B>

<"C:\test\echoargs.exe"  "A \"B">
<A "B>

<"C:\test\echoargs.exe"  "A\" B">
<A" B>

<"C:\test\echoargs.exe"  "A\\\" B\\\\\" \\">
<A\" B\\" \>

The first -replace doubles backslashes in front of quotes and adds one additional backslash, to escape the qoute.
The second -replace doubles backslashes at the end of the argument, such that the closing quote is not escaped.

This uses --% (PS v3 and above), which is AFAIK the only reliable way to pass quotes to native executables.

---

Edit:

Updated version of Run-Native, now called Invoke-NativeCommand (as suggested)

function Invoke-NativeCommand() {
    $command, [string[]] $argsForExe = $args
    if($argsForExe.Length -eq 0){
        & $command
    } else {
        $env:commandlineargumentstring=($argsForExe | %{
            if($_ -match '^[\w\d\-:/\\=]+$'){
                $_ #don't quote nonempty arguments consisting of only letters, numbers, or one of -:/\=
            } else {
                $_ <# double backslashes in front of quotes and escape quotes with backslash #> `
                    -replace '(\\*)"','$1$1\"' `
                   <# opening quote after xxx= or after /xxx: or at beginning otherwise #> `
                    -replace '^([\w\d]+=(?=.)|[/-][\w\d]+[:=](?=.)|^)','$1"' `
                   <# double backslashes in front of closing quote #> `
                    -replace '(\\*)$','$1$1' `
                   <# add closing quote #> `
                    -replace '$','"'
            }
        }) -join ' ';
        & $command --% %commandlineargumentstring%
    }
}

(with some inspiration from iep)

• doesn't quote simple switches
• still works with empty args
• works if no args present
• should mostly work with msiexec, cmdkey, etc...
• still always works for programs, that follow the common rules
• does not use nonstandard "" as escaped " - will therefore still not work for embedded quotes in .bat or msiexec arguments

[cspotcode]

Thanks, I didn't know about --%. is there any way to do that without leaking the environment variable to the native process? (and to any processes it might invoke)

Is there a PowerShell module that implements a Run-Native Cmdlet for everyone to use? This sounds like something that should be on the Powershell Gallery. If it were good enough, it could be the basis for an RFC.

[TSlivede]

"leaking" sounds like you are concerned about security. Notice however that the commandline is visible to child processes anyway. (For example: gwmi win32_process |select name,handle,commandline|Format-Table on Windows and ps -f on Linux)

If you still want to avoid an environment variable, you may be able to construct something using invoke-expression.

Regarding the RFC:
I don't think such a commandlet should be necessary, instead this should be the default behavior:

https://github.com/PowerShell/PowerShell-RFC/issues/90

[cspotcode]

I agree that PowerShell's default behavior should be fixed. I had been pessimistically assuming that it would never change for backwards compatibility reasons, which is why I suggested writing a module. However, I really like the way your RFC allows the old escaping behavior to be re-enabled via a preference variable.

[mklement0]

Let me summarize the discussion, with just the right dose of opinion:

• It's clear that we have a backward-compatibility issue, so the old behavior must continue to remain available.

• @TSlivede's RFC proposal accounts for that while commendably pointing _the way to the future_.
  Unfortunately, his proposal languishes as a _PR_ as of this writing, and it hasn't even been accepted as a an RFC _draft_ yet.

---

By _the future_, I mean:

• PowerShell is a shell in its own right that will hopefully soon shed its cmd.exe-related baggage, so the only considerations that matter when it comes to calling external utilities (executables that are (typically) console/terminal applications) are:

  • Arguments to pass should be specified by the rules of _PowerShell_'s argument-mode parsing _only_.

  • Whatever _literals_ result from that process must be passed _as-is_ to the target executable, as _individual_ arguments.

  • In other words: As a user, all you should ever need to focus on is what the result of _PowerShell_'s parsing will be, and to be able to rely on that result getting passed as-is, with PowerShell taking care of any behind-the-scenes encoding - if necessary.

---

_Implementing_ the future:

• On _Windows_:

  • For _historical_ reasons, Windows does _not_ permit passing arguments _as an array of literals_ to the target executable; instead, a _single_ string encoding _all_ arguments using _pseudo shell syntax_ is needed. What's worse, it is _ultimately up to the individual target executable to interpret that single string_ and split it into arguments.

  • The best PowerShell can do is to form that single string - behind the scenes, after having performed its _own_ splitting of the command line into individual arguments - in a _predictable , standardized manner_.

  • @TSlivede's RFC proposal proposes just that, by suggesting that PowerShell synthesize the pseudo shell command line in a manner that will cause the Windows C/C++ runtime to recover the input arguments as-is when performing its parsing:

    • Given that it's ultimately up to each target executable to interpret the command line, there is no _guarantee_ that this will work in all cases, but said rules are the most sensible choice, because _most_ existing utilities use these conventions.

    • The only notable exceptions are _batch files_, which could receive special treatment, as the RFC proposal suggests.

• On _Unix_ platforms:

  • Strictly speaking, the issues that plague Windows argument parsing _need never arise_, because the platform-native calls for creating new processes _accept arguments as arrays of literals_ - whatever arguments PowerShell ends up with after performing its _own_ parsing should just be passed on _as-is_.
    To quote @lzybkr: " I see no reason to bring that difficulty over to *nix."

  • Sadly, due to the current limitations of .NET Core (CoreFX), these issues _do_ come into play, because the CoreFX API needlessly forces the anarchy of _Windows_ argument passing onto the Unix world too, by requiring use of a pseudo command line even on Unix.

  • I've created this CoreFX issue to ask for that problem to be remedied.

  • In the meantime, given that CoreFX splits the pseudo command line back into arguments based on the C/C++ rules cited above, @TSlivede's proposal should work on Unix platforms too.

[TSlivede]

As https://github.com/PowerShell/PowerShell/issues/4358 was closed as duplicate of this, here a short summary of that problem:

If an argument of an external executable with a trailing backslash contains a space, it is currently naively quoted (add quote before and after the argument). Any executable, that follows the usual rules interprets that like this:
_{From @mklement0's comment:}

The 2nd " in ".\test 2\", due to being preceded by \ is interpreted as an escaped ", causing the remainder of the string - despite a then-missing closing " to be interpreted as part of the same argument.

Example:
_{(from @akervinen's comment)}

PS X:\scratch> .\ps-args-test.exe '.\test 2\'
Received argument: .\test 2"

The Problem occurs very often, because PSReadLine adds a trailing backslash on auto-completion for directories.

[SteveL-MSFT]

Since corefx seems open to producing the api we need, I'm deferring this to 6.1.0. For 6.0.0, I'll see if we can fix #4358

[SteveL-MSFT]

@TSlivede I took your function, renamed it to Invoke-NativeCommand (as Run isn't a valid verb) and added an alias ^ and published it as a module on PowerShellGallery:

install-module NativeCommand -scope currentuser
^ ./echoargs 'A "B' 'A" B'

[mklement0]

@SteveL-MSFT:

It's nice to have a stopgap, but a less cumbersome one would be - while we wait for a CoreFX solution - to implement the well-defined official quoting / argument-parsing rules as detailed in @TSlivede's RFC proposal ourselves preliminarily - which doesn't sound too hard to do.

If we only fix the \" problem, argument passing is still fundamentally broken, even in simple scenarios such as the following:

PS> bash -c 'echo "hi there"'
hi    # !! Bash sees the following tokens:  '-c', 'echo hi', 'there'

I think at this point there's sufficient agreement on what the behavior should be so we don't need a full RFC process, do we?

The only outstanding decision is how to deal with backward-compatibility issues in _Windows_.

[be5invis]

@mklement0 @SteveL-MSFT
Are we already broke compatibility?

[joeyaiello]

The only outstanding decision is how to deal with backward-compatibility issues in Windows.

Yeah, but that's the hard part, right?

@be5invis what do you mean by "already broke compatibility"?

Plus, if CoreFX is on the verge of a fix in their layer, I'd rather not create a stopgap in our layer before they do.

And as someone said above in the thread, this is annoying, but it's also pretty well-documented in the community. I'm not sure we should break it twice in the next two releases.

[mklement0]

@joeyaiello:

Isn't the fix for #4358 already a breaking change for those who've worked around the issue by doubling the final \; e.g., "c:\tmp 1\\"? In other words: if you limit the changes to this fix, _two_ breaking changes are guaranteed: this one now, and another later after switching to the future CoreFx API; and while that _could_ also happen if a complete stopgap were to be implemented now, it is unlikely, given what we know about this coming change.

Conversely, it may hamper adoption on Unix if common quoting scenarios such as
bash -c 'echo "hi there"' don't work properly.

I do realize that fixing this is a much larger breaking change, however.

[SteveL-MSFT]

@PowerShell/powershell-committee discussed this and agreed that minimally, using --% should have the same behavior as bash in that the quotes are escaped so that the native command receives them. What is still open for debate is if this should be the default behavior w/o using --%

[mklement0]

Note:

• I'm assuming that a call to an actual shell executable is necessary when using --% on _Unix_, as opposed to trying to _emulate_ the shell behavior, which is what happens on _Windows_. Emulating is not hard on Windows, but would be much harder on Unix, given the many more features that would need emulating.

• Using an actual shell then raises the question what shell to use: while bash is ubiquitous, its default behavior is not POSIX-compliant nor is it required by POSIX to be present, so for portability other scripting languages call out to /bin/sh, the shell executable decreed by POSIX (which _can_ be Bash running in compatibility mode (e.g., on macOS), but certainly does't have to (e.g., Dash on Ubuntu)).

Arguably, we should target /bin/sh as well - which, however, means that some Bash features - notably brace expansion, certain automatic variables, ... - won't be available

---

Use of --%

I'll use command echoargs --% 'echo "hi there"' as an example below.

the same behavior as bash in that the quotes are escaped so that the native command receives them.

The way to do _in the future, once the CoreFX API has been extended_ would be to perform no escaping at all, and instead do the following:

• Create a process as follows:
  
  
  
  • /bin/sh as the executable, (effectively) assigned to ProcessStartInfo.FileName.
  
  
  • The following array of _literal_, _individual_ argument tokens as ProcessStartInfo.ArgumentList:
  
  
  • -c as the 1st argument
  
  
  • echoargs 'echo "hi there"' as the 2nd argument - i.e., the original command line used _literally_, exactly as specified, except that --% was removed.
  
  

In effect, the command line is passed through _as-is_ to the shell executable, which can then perform _its_ parsing.

I understand that, in the current absence of an array-based way to pass literal arguments, we need to combine -c and echoargs 'echo "hi there"' into a _single_ string _with escaping_, regrettably _solely for the benefit of the CoreFX API_, which, when it comes time to create the actual process, then _reverses_ this step and splits the single string back into literal tokens - and ensuring that this reversal always results in the original list of literal tokens is the challenging part.

Again: The only reason to involve escaping here at all is due to the current CoreFX limitation.
To work with this limitation, the following single, escaped string must therefore be assigned to the .Arguments property of a ProcessStartInfo instance, with the escaping performed as specified by Parsing C++ Command-Line Arguments:

• /bin/sh as the executable, (effectively) assigned to ProcessStartInfo.FileName.
• The following single, escaped string as the value of ProcessStartInfo.Arguments:
  -c "echoargs 'echo \"hi there\"'"

## Default behavior

What is still open for debate is if this should be the default behavior w/o using --%

The default behavior on Unix should be very different:

• No escaping considerations _other than PowerShell's own_ should ever come into play (except on _Windows_, where that cannot be avoided, sadly; but there the MS C++ rules are the way to go, _to be applied behind the scenes_; failing that, --% provides an escape hatch).

• Whatever arguments _PowerShell_ ends up with, after its own parsing, must be passed as an _array of literals_, via the upcoming ProcessStartInfo.ArgumentList property.

Applied to the example without --%: echoargs 'echo "hi there"':

• PowerShell performs its usual parsing and ends up with the following 2 arguments:

  • echoargs
  • echo "hi there" (single quotes - which only had syntactical function to _PowerShell_, removed)

• ProcessStartInfo is then populated as follows, with the upcoming CoreFX extension in place:

  • echoargs as the (effective) .FileName property value
  • _Literal_ echo "hi there" as the only element to add to the Collection<string> instance exposed by .ArgumentList.

Again, in the absence of .ArgumentList that is not an option _yet_, but _in the interim_ the same MS C++-compliant auxiliary escaping as described above could be employed.

[TSlivede]

@SteveL-MSFT
As I already mentioned at Make the stop-parsing symbol (--%) work on Unix (#3733) I'd strongly advise against changing the behavior of --%.

If some special functionality for /bin/sh -c is needed please use a different symbol and leave --% the way it is!

[mklement0]

@TSlivede:

_If_ something --%-_like_ is implemented on Unix - and with native globbing and a generally more command-line-savvy crowd on Unix I perceive less of a need for it - then choosing a _different symbol_ - such as --$ - probably makes sense (sorry, I'd lost track of all aspects of this lengthy, multi-issue debate).

Different symbols would also serve as visually conspicuous reminders that non-portable _platform-specific_ behavior is being invoked.

That leaves the question what PowerShell should do when it comes across --% on Unix and --$ on Windows.

[SteveL-MSFT]

I'm fine leaving --% as-is. Introducing something like --$ which calls out to /bin/sh and I guess cmd.exe on Windows may be a good way to solve this.

[iSazonov]

No chance of creating a cmdlet for these behaviors?

[SteveL-MSFT]

@iSazonov are you suggesting something like Invoke-Native? Not sure I'm a fan of that.

[iSazonov]

Yes, like Start-Native.
As a joke :-), do you not like cmdlets in PowerShell?

In Build.psm1 we have Start-NativeExecution with link to https://mnaoumov.wordpress.com/2015/01/11/execution-of-external-commands-in-powershell-done-right/

[mklement0]

@SteveL-MSFT

I'm fine leaving --% as-is.

I think we all agree that --% must continue to behave the way it does on _Windows_.

On _Unix_, by contrast, this behavior makes no sense, as I've tried to demonstrate here - in short:

• single quotes aren't being handled correctly
• the only way to reference environment variable is as _cmd.exe-style_ (%var%)
• important native features such as globbing and word-splitting don't work.

The primary motivation for introducing --% was, if I understand correctly, to enable the _reuse of existing cmd.exe command lines_ as-is.

• As such, --% is useless on Unix with its current behavior.
• _If_ we wanted an _analogous_ feature on Unix, it would have to be /bin/sh -c-based, as proposed above, probably using a different symbol.

I don't think there is a need for a cmd /c-based feature on Windows, as --% has that _mostly_ covered, arguably in a manner that is _good enough_.
@TSlivede has pointed out that not all shell features are being emulated, but in practice that appears not to be a concern (e.g., variable-value substitutions such as %envVar:old=new% aren't supported, ^ isn't an escape char., and use of --% is limited to a _single_ command - no use of cmd.exe's redirection and control operators; that said, I don't think --% was ever meant to emulate entire command _lines_).

As such, something like --$ - if implemented - would be the Unix _counterpart_ to --%.

In any event, at the very least the about_Parsing help topic deserves a conspicuous warning that --% will be useless on Unix in all but a few cases.

[SteveL-MSFT]

@iSazonov It may make sense to have Start-Native to handle some specific scenarios, but we should try to improve PowerShell so that using native exes is more natural and predictable

[SteveL-MSFT]

@PowerShell/powershell-committee reviewed this and agree that --% should mean treat the arguments as they would on their relevant platforms which means they behavior differently on Windows and Linux, but consistent within Windows and consistent within Linux. It would be more confusing to the user to introduce a new sigil. We will leave implementation to the engineer on how to enable this.

Cross platform scripts would have to be aware of this difference in behavior, but it seems unlikely users would hit this. If the user feedback is that there is a need due to more cross platform usage, then we can revisit introducing a new sigil.

[mpawelski]

For the second time I've been bitten by these native vs cmdlets string argument parsing differences when used ripgrep.

Here is result of powershell calls (echo.exe is from "C:\Program Files\Git\usrbin\echo.exe")

Now I know I should watch out for this " quirk:

> echo.exe '"test'
test

But this quirk is beyond me...

echo.exe '^\+.+;'
^\+.+;
echo.exe '^\+.*;'
^+.*;

In second case I need to put double \ to pass \ to native command, in first case I don't need to do it 😑

I understand this would be a breaking change to change this behavior so there won't be a difference between cmlets and native commands. However, I think quirks like this is something that puts people off from using powershell as default shell.

[SteveL-MSFT]

@mpawelski I just tried this on my Windows box with git 2.20.1.vfs.1.1.102.gdb3f8ae and it doesn't repro for me with 6.2-RC.1. Ran it several times and it consistently echos ^\+.+;

[mklement0]

@SteveL-MSFT, I think that @mpawelski accidentally specified the same command twice. You'll see the problem if you pass '^\"+.*;' for instance - note the \" part - with the - reasonable - expectation that the content of the single-quoted string will be passed through as-is, so that the external target program sees ^\"+.*; as the argument's value:

# Note how the "\" char. is eaten.
PS> bash -c 'printf %s "$1"' - '^\"+.*;'
^"+.*; 

#"# Running the very same command from Bash does NOT exhibit the problem:
$ bash -c 'printf %s "$1"' - '^\"+.*;'
^\"+.*; 

[rkeithhill]

--% was, if I understand correctly, to enable the reuse of existing cmd.exe command lines as-is.

That's not quite it. --% was introduced because many Windows command line utils take args that are interpreted by PowerShell which completely fubars the invocation of the utility. This can sour folks quick if they can no longer easily use their favorite native utils. If I had a quarter for every time I've answered questions on SO and elsewhere RE native exe commands that don't work right because of this issue, I could probably take the family out to dinner at Qoba. :-) For instance, tf.exe allows ; as part of specifying a workspace. Git allows {} and @~1, etc, etc, etc.

--% was added to tell PowerShell to NOT parse the rest of the command line - just send it "as-is" to the native exe. With the one rub, allow variables using the cmd env var syntax. It is a bit ugly but man, it really, really comes in handy still on Windows.

RE making this a cmdlet, I'm not sure I see how that works. --% is a signal to the parser to dumb down the parsing until EOL..

Frankly, as a long time user of PowerShell and this feature in particular, it makes sense to me to use the same operator on other platforms to simply mean - dumb down the parsing until EOL. There is the issue of how to allow some form of variable substitution. Even though it feels a bit ugly, on macOS/Linux you could take %envvar% and substitute the value of any corresponding env var. Then it could be portable between platforms.

The thing is, if you don't do this then you wind up with conditional code - not exactly what I'd call portable:

$env:Index = 1
if ($IsWindows) {
    git show --% @~%Index%
}
else {
    git show --$ @~$Index
}

I'd prefer this work on all platforms:

$env:Index = 1
git show --% @~%Index%

The behavior on Windows has to remain as-is because compatibility.

[mklement0]

@rkeithhill

That's not quite it.

By process of elimination, command lines that predate PowerShell in the Windows world were written for cmd.exe - and that's precisely what --% intends to emulate, as evidenced by the following:

• --% expands cmd.exe-style %...% environment-variable references, such as %USERNAME% (if no shell and no special logic were involved, such tokens would be passed _verbatim_)

• straight from the PowerShell's mouth (if you will; emphasis added):

The web is full of command lines written for Cmd.exe. These commands lines work often enough in PowerShell, but when they include certain characters, e.g. a semicolon (;) a dollar sign ($), or curly braces, you have to make some changes, probably adding some quotes. This seemed to be the source of many minor headaches.

To help address this scenario, we added a new way to “escape” the parsing of command lines. If you use a magic parameter --%, we stop our normal parsing of your command line and switch to something much simpler. We don’t match quotes. We don’t stop at semicolon. We don’t expand PowerShell variables. We do expand environment variables if you use Cmd.exe syntax (e.g. %TEMP%). Other than that, the arguments up to the end of the line (or pipe, if you are piping) are passed as is. Here is an example:

Note that this approach is a poor fit for the _Unix_ world (to recap from above):

• Globbing of unquoted arguments such as *.txt won't work.

• Traditional (POSIX-like) shells in the Unix world do _not_ use %...% to refer to environment variables; they expect $... syntax.

• There is (fortunately) no _raw_ command line in the Unix world: any external executable must be passed an _array_ of _literals_, so it is still PowerShell or CoreFx that must parse the command line into arguments first.

• Traditional (POSIX-like) shells in the Unix world accept '...' (single-quoted) strings, which --% doesn't recognize - see #10831.

But even in the Windows world --% has severe, non-obvious limitations:

• Most obviously, you cannot directly reference PowerShell variables in your command line if you use --%; the only - cumbersome - workaround is to temporarily define _environment_ variables, which you must then reference with %...%.
• You cannot enclose the command line in (...) - because the closing ) is interpreted as a literal part of the command line.
• You cannot follow the command line with ; and another statement - because the ; is interpreted as a literal part of the command line.
• You cannot use --% inside a single-line script block - because the closing } is interpreted as a literal part of the command line.
• You cannot use redirections - because they're treated as a literal part of the command line - however, you can use cmd --% /c ... > file, to let cmd.exe handle the redirection.
• You cannot use line-continuation characters - neither PowerShell's (`) nor cmd.exe's (^) - they will be treated as _literals_.
  
  
  
  • --% only ever parses (at most) to the end of the line.
  
  

---

Fortunately, we already _do_ have a cross-platform syntax: _PowerShell's own syntax_.

Yes, using that requires you to know what _PowerShell_ considers metacharacters, which is a _superset_ of both what cmd.exe and POSIX-like shells such as Bash consider metacharacters, but that is the price to pay for a richer, platform-agnostic command-line experience.

How unfortunate it is, then, that PowerShell handles quoting of " characters so poorly - which is the very subject of this issue, and which is summarized in this docs issue.

[mklement0]

@rkeithhill, I started a bit of a tangent; let me try to close it:

• --% actually already _is_ implemented as of PowerShell Core 6.2.0; e.g.,
  /bin/echo --% %HOME% prints the value of environment variable HOME; by contrast,
  /bin/ls --% *.txt will not work as expected, because the *.txt is passed as a literal.

• Ultimately, when not using --%, we need to help users diagnose what the command line / arguments array that is constructed _behind the scenes ultimately_ looks like (which takes us back to the venerable #1761):

  • Your helpful echoArgs.exe does just that, and in the linked issue you sensibly called for such functionality to be part of PowerShell itself.
  • @SteveL-MSFT pondered including the resulting command line in the error record.

Finally, to apply my previous argument - using PowerShell's own syntax as the inherently portable one - to your example:

# Works cross-platform, uses PowerShell syntax 
# Note: No need for an aux. *environment* variable (which should be cleaned up afterward)
$Index = 1
git show "@~$Index"

# Alternative, quoting just the '@'
git show `@~$Index

Yes, it requires you to know that a token-initial @ is a metacharacter that you therefore have to quote, but as PowerShell becomes more widely used, the awareness of such requirements should become more widespread too.

[Artoria2e5]

FYI, this problem should be almost the same on Windows and Unix-likes. CoreFx implementation of Unix S.D.Process has a thing called ParseArgumentsIntoList, which implements CommandLineToArgvW near-exactly without _setargv switches (and with the undocumented "" in quotes → " feature). Unix should not be an additional point of pain in this because in the current shape it is broken in the same way as Windows is.

_setargv is not something every program uses after all, and it's probably not worth considering it because, well, it is kinda effed with behavioral changes among CRT versions. The best we can and should do is to surround everything with double quotes, add some nice backslahes, and that's all.

[SteveL-MSFT]

Another example where args aren't being parsed correctly:

az "myargs&b"

In this case, az gets myargs and b is attempted to be executed as a new command.

Workaround is: az --% "myargs&b"

[mklement0]

@SteveL-MSFT, we can remove the Waiting - DotNetCore label, given that the requisite feature - the collection-based ProcessStartInfo.ArgumentList property has been available since .NET Core 2.1

@TSlivede is aware of the new method and plans to use it, but the associated RFC, https://github.com/PowerShell/PowerShell-RFC/pull/90, is languishing, unfortunately.

I suggest we continue the _implementation_ discussion there.

However, in the RFC discussion, @joeyaiello talks about making the changes an experimental feature, but it's becoming increasingly clear to me that you cannot fix the quoting behavior without massively breaking existing code:

Anyone who's had to _work around_:

• the inability to pass an empty-string argument (foo.exe "" currently passes _no_ arguments)
• the unexpected effective removal of double quotes due to lack of automatic escaping of embedded double quotes (foo.exe '{ "foo": "bar" }' being passed as improperly escaped "{ "foo": "bar" }")
• the quirks of certain CLIs such as msiexec that don't accept certain arguments double-quoted _as a whole_ (foo.exe foo="bar none" being passed as "foo=bar none").
  Note: It is msiexec that is to blame here, and with the proposed changes applied, passing the required foo="bar none" form of quoting will then require --%.

will be in trouble, because the workarounds will _break_ with the proposed changes applied.

Therefore, an additional question is:

• How can we make correct behavior available at least as an _opt-in_ feature?

  • In the case of broken quoting with Start-Process, we at least have the option of introducing a new parameter for the correct behavior, but with direct invocation that is clearly not an option (unless we introduce another "magic symbol" such as --%).

• A fundamental problem with such mechanisms - typically, by preference variable, but increasingly also by using statements - is PowerShell's dynamic scoping; that is, the opted-in behavior will by default be applied to code called _from_ one's opted-in code as well, which is problematic.

  • Perhaps it is time for generally introducing _lexical_ scoping of features, a generalization of the lexically scoped using strict proposal in the - equally languishing - lexical strict-mode RFC authored by @lzybkr.

  • Something like a lexically scoped using preference ProperArgumentQuoting? (Name obviously negotiable, but I'm struggling to come up with one).

[vexx32]

Given all those caveats, and that this is an issue with direct invocation as well, where we can't simply add a new parameter, I'm firmly in favour of breaking the old behaviour.

Yes, it'll probably break a lot. But really, only because it was so thoroughly broken to begin with. I don't think it's particularly feasible to prioritise maintaining what amounts to a huge pile of workarounds for broken behaviour over having a feature that _actually works_.

As a new major version I think v7 really is the only chance we'll get to rectify this situation properly for quite some time, and we should take the opportunity. Provided we make users aware, I don't think the transition will be poorly received overall.

[iSazonov]

If we feel that breaking change is inevitable, then perhaps we should design the ideal solution, taking into account that it should be easy to print in an interactive session and it is possible to have a script version that better works on all platforms.

[mklement0]

Agreed, @vexx32: Retaining the existing behavior, even if only by default, will remain a perennial pain point. Users do not expect the need for opt-in to being with, and, once they do, they are likely to either forget on occasion and/or resent the need to apply it every time.

A shell that doesn't reliably pass arguments to external programs is failing one of its core mandates.

You certainly have _my_ vote for making the breaking change, but I fear that others will feel differently, not least because v7 is being touted as allowing longterm WinPS users to migrate to PSCore.

---

@iSazonov: https://github.com/PowerShell/PowerShell-RFC/pull/90 describes the correct solution.

To recapitulate its spirit:

PowerShell, as a shell, needs to parse arguments according to _its_ rules and then pass the resulting, expanded argument values _verbatim_ to the target program - users should never have to think about how PowerShell makes that happen; all they should ever have to worry about is getting _PowerShell_ syntax right.

• On Unix-like platforms, ProcessStartInfo.ArgumentList now gives us a way to perfectly implement this, given that the _array_ of expanded argument values can be passed _as-is_ to the target program, because that's how argument passing - sensibly - works in this world.

• On Windows, we have to deal with the unfortunate reality of the anarchy that is Windows command-line parsing, but, as a shell, it behooves us to _just make it work behind the scenes, as much as possible_, which is what the RFC describes - though I've just discovered a wrinkle that makes sole use of ProcessStartInfo.ArgumentList not good enough, unfortunately (due to widespread of _batch files_ as CLI entry points, as demonstrated by @SteveL-MSFT's az[.cmd] example above). For those edge cases where doing the sensible thing isn't good enough, there's --%.

[iSazonov]

Perhaps PSSA can help to mitigate the breaking change by warning users that they use an argument format which will be changed.

[SteveL-MSFT]

I think we need to consider adopting something like Optional Features to move forward on this breaking change along with some others.

[vexx32]

Is there really any value in maintaining the existing behaviour? Other than backwards compatibility, I mean.

I don't think it's worthwhile to maintain two code paths for this, simply to retain a broken implementation because some old code might need it once in a while. I don't think it's unreasonable to expect folx to update their code once in a while. 😅

Doubly so if we expect PS7 to be a stand-in replacement for WinPS; the only reason I can see to retain the behaviour for v7 is if we expect folks to be using the same script to run commands on both 5.1 and 7, which (hopefully) should be a pretty rare case if PS7 is good replacement for 5.1.

And even then, it wouldn't be overly difficult for users to account for both. Provided we're not changing actual language syntax, it should be pretty easy to do something like this:

if ($PSVersionTable.PSVersion.Major -lt 7) {
    # use old form
}
else {
    # use new form
}

Provided we make users aware of the difference, I think it would be a welcome relief from the pain that it has been to handle weird native executables in PS up to now. 😄

As @TylerLeonhardt mentioned in the discussion of optional features -- implementing that means you now maintain _multiple_ distinct implementations which each need to be maintained and tested, plus also maintaining and testing the optional feature framework. Doesn't really seem worth it for this, tbh.

[SteveL-MSFT]

@vexx32 backwards compatibility is a big issue here. This isn't the only issue for args to native executables. I'd like to bucket them all and have them all be one "optional feature". For example, https://github.com/PowerShell/PowerShell/issues/1761 and https://github.com/PowerShell/PowerShell/issues/10675. "Fixing" interoperability with native commands is something I'd like to resolve for vNext. So if anyone sees any existing or new issues in this category, cc me and I'll tag it appropriately (or if you have triage permission, tag it like the others).

[iSazonov]

Optional features is modules :-) Having optional features in Engine is big headache for support, specially for Windows support. We could modularize Engine by reducing internal dependencies and replacing internal APIs with public - after this we could implement optional features in Engine in easy way.

[SteveL-MSFT]

@iSazonov one of the things my team will look at in vNext is to make the engine more modular :)

[musm]

What is the recommended solution here for end users?

[Artoria2e5]

This is contrived, but it is the most straightforward way to get to the correct* ArgumentList handling from the .NET framework itself:

Add-Type -AssemblyName "System"
function startProcess([string] $FileName, [string[]] $ArgumentList) {
    $proc = ([System.Diagnostics.Process]::new())
    $proc.StartInfo.UseShellExecute = $false
    $proc.StartInfo.FileName = $FileName
    $proc.StartInfo.CreateNoWindow = $true
    foreach ($a in $ArgumentList) { $proc.StartInfo.ArgumentList.Add($a) }
    $proc.Start()
    return $proc
}

startProcess -FileName 'C:\Program Files\nodejs\node.exe' -ArgumentList '-e','console.log(process.argv.join(''\n''))','--','abc" \" messyString'

Of course you can make it less contrived to use here by using positional parameters and some get-command trickery.

* DISCLAIMER: There is no single correct way to parse a cmdline on Windows. By "correct" I mean the MSVCRT style of cmdline from/to argv conversion, implemented by .NET on all platforms for ArgumentList handling, main (string[] args) processing, and external spawn calls on Unix. This sample is provided AS IS with no guarantee for general interoperability. See also the "windows command line" section of proposed NodeJS child_process documentation.

[choovick]

@Artoria2e5 That exactly the conclusion I came to. System.Diagnostics.Process is the only reliable way to run external executable, but escaping arguments can get tricky due to stdlib Rules:

2N backslashes + " ==> N backslashes and begin/end quote
2N+1 backslashes + " ==> N backslashes + literal " 
N backslashes ==> N backslashes

As the result I have come up with the following logic to escape arguments wrap them into double quotes " for process executuion:
https://github.com/choovick/ps-invoke-externalcommand/blob/master/ExternalCommand/ExternalCommand.psm1#L244

Also it can get tricky to obtain STDOUT and STDERR in realtime while external executable runs, so I have created this package

https://github.com/choovick/ps-invoke-externalcommand

that i'm using heavily on Windows, Linux and Mac and so far without issues and I can pass arguments with newline and other special characters in them.

GitHub

choovick/ps-invoke-externalcommand

Contribute to choovick/ps-invoke-externalcommand development by creating an account on GitHub.

GitHub

choovick/ps-invoke-externalcommand

Contribute to choovick/ps-invoke-externalcommand development by creating an account on GitHub.

[Artoria2e5]

@choovick the whole stdio redirect thing is great!

I disagree slightly on the escaping part though, since there is already a thing that does it for you called ArgumentList. I do understand that it is a relative recent (?) addition, and it does get disappointing since MS forgot to put a String,String[] initializer for S.D.ProcessStartInfo. (Is there a place for these… .NET Interface proposals?)

chitchat

My escape function from that NodeJS example is a bit different from yours: it uses the undocumented (but found in .NET core and MSVCRT) "" escape for quotation marks. Doing so sort of simplifies the backslash picking work. I did this mainly because it was used for the all mighty cmd, which does not understand that \" should not unquote the rest of the string. Instead of struggling with \^" I figured that I will be better off with something that has been in secret use since the beginning of time.

[choovick]

@Artoria2e5 unfortunately ArgumentList is not available in PowerShell 5.1 on windows, using your example i'm getting:

```You cannot call a method on a null-valued expression.
At C:Users\yser\dev\test.ps1:7 char:37

• ... oreach ($a in $ArgumentList) { $proc.StartInfo.ArgumentList.Add($a) }
• ~~~~~~~~
  
  
  
  • CategoryInfo : InvalidOperation: (:) [], RuntimeException
  
  
  • FullyQualifiedErrorId : InvokeMethodOnNull
    
    ```
  
  

Since custom argument escape logic....

chitchat

in regards to `\^"` in NodeJS, I think I had to do that several years back :) and I think got it working

[mklement0]

System.Diagnostics.Process is the only reliable way to run external executable

The problem is that you won't get integration with PowerShell's output streams and you won't get streaming behavior in the pipeline.

Here's the summary of the required workarounds if you still want to let PowerShell perform the invocation (which is definitely preferable):

• If you need to pass arguments with _embedded_ " chars., _double_ them on Windows, if possible, or \-escape them:

  • On Windows, when calling _batch files_ and if you know that the target program understands "" as an escaped ", use $arg -replace '"', '""'

    • Use of "" is preferable on Windows (it avoids the Windows PowerShell problem and works with CLIs that use batch files as _stubs_, such as Node.js and Azure), but not all executables support it (notably not Ruby and Perl).

  • Otherwise (always on Unix), use $arg -replace '"', '\"'

  • Note: In _Windows PowerShell_, this still doesn't always work properly if the value also contains _spaces_, because the presence of literal \" in the value does situationally does _not_ trigger enclosing double-quoting, unlike in PowerShell Core; e.g., passing '3\" of snow' breaks.

  • Additionally, before the above escaping, you must double \ instance immediately preceding ", if they are to be treated as literals:

    • $arg = $arg -replace '(\\+)"', '$1$1"'

• If you need to pass an _empty_ argument, pass '""'.

  • '' -eq $arg ? '""' : $arg (WinPS alternative: ($arg, '""')['' -eq $arg]

• Windows PowerShell only, don't do this in PS Core (where the problem has been fixed):

  • If your argument _contains spaces_ and _ends in_ (one or more) \, double the trailing\ instance(s).

    • if ($arg -match ' .*?(\\+)$') { $arg = $arg + $Matches[1] }

• If cmd / a batch file is being invoked with arguments that do _not_ have spaces (therefore _not_ triggering automatic double-quoting by PowerShell) but contain any of &|<>^,; (e.g., a&b), use _embedded enclosing double-quoting_ to ensure that PowerShell passes a double-quoted token and therefore doesn't break the cmd / batch-file call:

  • $arg = '"' + $arg + '"'

• If you need to deal with ill-behaved executables such as msiexec.exe, single-quote the argument:

  • 'foo="bar none"'

As stated, these workarounds will _break_, once the underlying problem gets fixed.

---

Below is simple (non-advanced) function iep (for "invoke external program"), which:

• Performs all of the escaping described above, including automatic special-casing for msiexec and preferring "" escaping over \" depending on the target program.
  
  
  
  • The idea is that you can pass any argument by focusing on _PowerShell_'s string syntax only, and rely on the function to perform the necessary escaping so that the verbatim value that PowerShell sees is also seen by the target program.
  
  
  • In PowerShell _Core_, this should work pretty robustly; in Windows PowerShell you still have edge cases with embedded double quotes that break if \" escaping must be used (as discussed above).
  
  
• Preserves shell-command invocation syntax.
  
  
  
  • Simply prepend iep  to your command line.
  
  
• As direct invocation would, it:
  
  
  
  • integrates with PowerShell's streams
  
  
  • sends output line by line through the pipeline
  
  
  • sets $LASTEXITCODE based on the external program's exit code; however, $? can _not_ be relied upon.
  
  

Note: The function is purposely minimalistic (no parameter declarations, no command-line help, short (irregular) name), because its meant to be as unobtrusive as possible: simply prepend iep to your command line, and things should work.

Example invocation using EchoArgs.exe (installable via Chocolatey from an _elevated_ session with choco install echoargs -y):

PS> iep echoargs '' 'a&b' '3" of snow' 'Nat "King" Cole' 'c:\temp 1\' 'a \" b'
Arg 0 is <>
Arg 1 is <a&b>
Arg 2 is <3" of snow>
Arg 3 is <Nat "King" Cole>
Arg 4 is <c:\temp 1\>
Arg 5 is <a \" b>

Command line:
"C:\ProgramData\chocolatey\lib\echoargs\tools\EchoArgs.exe" "" a&b "3\" of snow" "Nat \"King\" Cole" "c:\temp 1\\" "a \\\" b"

The above shows PowerShell Core output. Note how all arguments were correctly passed through as seen verbatim by PowerShell, including the empty argument.

In Windows PowerShell, the 3" of snow argument won't be passed correctly, because \" escaping is used due to calling an unknown executable (as discussed above).

To verify that batch files pass arguments correctly through, you can create echoargs.cmd as a wrapper for echoargs.exe:

'@echoargs.exe %*' | Set-Content echoargs.cmd

Invoke as iep .\echoargs.cmd '' 'a&b' '3" of snow' 'Nat "King" Cole' 'c:\temp 1\' 'a \" b'

Since a batch file is now called, ""-escaping is employed, which fixes the 3" of snow problem when calling from Windows PowerShell.

The function equally works on Unix-like platforms, which you can verify by creating a sh shell script named echoargs:

@'
#!/bin/sh
i=0; for a; do printf '%s\n' "\$$((i+=1))=[$a]"; done
'@ > echoargs; chmod a+x echoargs

Invoke as iep ./echoargs '' 'a&b' '3" of snow' 'Nat "King" Cole' 'c:\temp 1\' 'a \" b'

---

Important: A more complete version of this function has since been published as ie (Invoke (external) Executable) in I've just published a module Native, which I encourage you to use instead. Install the module with
Install-Module Native -Scope CurrentUser.
The module also contains an ins (Invoke-NativeShell) command that addresses the use case discussed in #13068 - see https://github.com/PowerShell/PowerShell/issues/13068#issuecomment-671572939 for details.

Function iep' s source code (use module Native instead - see above):

function iep {

  Set-StrictMode -Version 1
  if (-not (Test-Path Variable:IsCoreClr)) { $IsCoreCLR = $false }
  if (-not (Test-Path Variable:IsWindows)) { $IsWindows = $env:OS -eq 'Windows_NT' }

  # Split into executable name/path and arguments.
  $exe, [string[]] $argsForExe = $args

  # Resolve to the underlying command (if it's an alias) and ensure that an external executable was specified.
  $app = Get-Command -ErrorAction Stop $exe
  if ($app.ResolvedCommand) { $app = $app.ResolvedCommand }
  if ($app.CommandType -ne 'Application') { Throw "Not an external program, non-PS script, or batch file: $exe" }

  if ($argsForExe.Count -eq 0) {
    # Argument-less invocation
    & $exe
  }
  else {
    # Invocation with arguments: escape them properly to pass them through as literals.
    # Decide whether to escape embedded double quotes as \" or as "", based on the target executable.
    # * On Unix-like platforms, we always use \"
    # * On Windows, we use "" where we know it's safe to do. cmd.exe / batch files require "", and Microsoft compiler-generated executables do too, often in addition to supporting \",
    #   notably including Python and Node.js
    #   However, notable interpreters that support \" ONLY are Ruby and Perl (as well as PowerShell's own CLI, but it's better to call that with a script block from within PowerShell).
    #   Targeting a batch file triggers "" escaping, but in the case of stub batch files that simply relay to a different executable, that could still break
    #   if the ultimate target executable only supports \" 
    $useDoubledDoubleQuotes = $IsWindows -and ($app.Source -match '[/\\]?(?<exe>cmd|msiexec)(?:\.exe)?$' -or $app.Source -match '\.(?<ext>cmd|bat|py|pyw)$')
    $doubleQuoteEscapeSequence = ('\"', '""')[$useDoubledDoubleQuotes]
    $isMsiExec = $useDoubledDoubleQuotes -and $Matches['exe'] -eq 'msiexec'
    $isCmd = $useDoubledDoubleQuotes -and ($Matches['exe'] -eq 'cmd' -or $Matches['ext'] -in 'cmd', 'bat')
    $escapedArgs = foreach ($arg in $argsForExe) {
      if ('' -eq $arg) { '""'; continue } # Empty arguments must be passed as `'""'`(!), otherwise they are omitted.
      $hasDoubleQuotes = $arg.Contains('"')
      $hasSpaces = $arg.Contains(' ')
      if ($hasDoubleQuotes) {
        # First, always double any preexisting `\` instances before embedded `"` chars. 
        # so that `\"` isn't interpreted as an escaped `"`.
        $arg = $arg -replace '(\\+)"', '$1$1"'
        # Then, escape the embedded `"` chars. either as `\"` or as `""`.
        # If \" escaping is used:
        # * In PS Core, use of `\"` is safe, because its use triggers enclosing double-quoting (if spaces are also present).
        # * !! In WinPS, sadly, that isn't true, so something like `'foo="bar none"'` results in `foo=\"bar none\"` -
        #   !! which - due to the lack of enclosing "..." - is seen as *2* arguments by the target app, `foo="bar` and `none"`.
        #   !! Similarly, '3" of snow' would result in `3\" of snow`, which the target app receives as *3* arguments, `3"`, `of`, and `snow`.
        #   !! Even manually enclosing the value in *embedded* " doesn't help, because that then triggers *additional* double-quoting.
        $arg = $arg -replace '"', $doubleQuoteEscapeSequence
    }
      elseif ($isMsiExec -and $arg -match '^(\w+)=(.* .*)$') { 
        # An msiexec argument originally passed in the form `PROP="value with spaces"`, which PowerShell turned into `PROP=value with spaces`
        # This would be passed as `"PROP=value with spaces"`, which msiexec, sady, doesn't recognize (`PROP=valueWithoutSpaces` works fine, however).
        # We reconstruct the form `PROP="value with spaces"`, which both WinPS And PS Core pass through as-is.
        $arg = '{0}="{1}"' -f $Matches[1], $Matches[2]
      }
      # As a courtesy, enclose tokens that PowerShell would pass unquoted in "...", 
      # if they contain cmd.exe metachars. that would break calls to cmd.exe / batch files.
      $manuallyDoubleQuoteForCmd = $isCmd -and -not $hasSpaces -and $arg -match '[&|<>^,;]'
      # In WinPS, double trailing `\` instances in arguments that have spaces and will therefore be "..."-enclosed,
      # so that `\"` isn't mistaken for an escaped `"` - in PS Core, this escaping happens automatically.
      if (-not $IsCoreCLR -and ($hasSpaces -or $manuallyDoubleQuoteForCmd) -and $arg -match '\\') {
        $arg = $arg -replace '\\+$', '$&$&'
      }
      if ($manuallyDoubleQuoteForCmd) {
        # Wrap in *embedded* enclosing double quotes, which both WinPS and PS Core pass through as-is.
        $arg = '"' + $arg + '"'
      }
      $arg
    }
    # Invoke the executable with the properly escaped arguments.
    & $exe $escapedArgs
  }
}

[choovick]

@mklement0 Impressive, but here are couple that does not work for me on windows:

iep echoargs 'somekey="value with spaces"' 'te\" st'

Arg 0 is <somekey="value>
Arg 1 is <with>
Arg 2 is <spaces">
Arg 3 is <te\">
Arg 4 is <st>

Command line:
"C:\ProgramData\chocolatey\lib\echoargs\tools\EchoArgs.exe" somekey=\"value with spaces\" te\\\" st

here is my test array of arguments :)

$Arguments = @(
    'trippe slash at the end \\\',
    '4 slash at the end \\\\',
    '\\servername\path\',
    'path=\\servername\path\',
    'key="\\servername\pa th\"',
    '5 slash at the end \\\\\',
    '\\" double slashed double quote',
    'simple',
    'white space',
    'slash at the end \',
    'double slash at the end \\',
    'trippe slash at the end \\\',
    'trippe slash at the end with space \\\ ',
    '\\" double slashed double quote',
    'double slashed double quote at the end \\"',
    '\\\" triple slashed double quote',
    'triple slashed double quote at the end \\\"',
    # slash
    'single slashes \a ^ \: \"',
    'path="C:\Program Files (x86)\test\"'
    # quotes
    'double quote " and single quote ''',
    # windows env var syntax
    "env var OS: %OS%",
    # utf16
    ('"utf16 ETHIOPIC WORDSPACE: \u1361"' | ConvertFrom-Json),
    # special chars
    "newLine`newLine"
    "tab`tab"
    "backspace`bbackspace"
    "carriage`rafter",
    "formFeed`fformFeed",
    # JSON Strings
    @"
[{"_id":"5cdab57e4853ea7b5a707070","index":0,"guid":"25319946-950e-4fe8-9586-ddd031cbb0fc","isActive":false,"balance":"`$2,841.15","picture":"http://placehold.it/32x32","age":39,"eyeColor":"blue","name":{"first":"Leach","last":"Campbell"},"company":"EMOLTRA","email":"[email protected]","phone":"[phone]","address":"127 Beadel Street, Vivian, Vermont, 1991","about":"Ex labore non enim consectetur id ullamco nulla veniam Lorem velit cillum aliqua amet nostrud. Occaecat ipsum do est qui sint aliquip anim culpa laboris tempor amet. Aute sint anim est sint elit amet nisi veniam culpa commodo nostrud cupidatat in ex.","registered":"Monday, August 25, 2014 4:04 AM","latitude":"-12.814443","longitude":"75.880149","tags":["pariatur","voluptate","sint","Lorem","eiusmod"],"range":[0,1,2,3,4,5,6,7,8,9],"friends":[{"id":0,"name":"Lester Bender"},{"id":1,"name":"Concepcion Jarvis"},{"id":2,"name":"Elsie Whitfield"}],"greeting":"Hello, Leach! You have 10 unread messages.","favoriteFruit":"strawberry"},{"_id":"5cdab57e8cd0ac577ab534a4","index":1,"guid":"0be10c87-6ce7-46c4-8dd6-23b1d9827538","isActive":false,"balance":"`$1,049.56","picture":"http://placehold.it/32x32","age":33,"eyeColor":"green","name":{"first":"Lacey","last":"Terrell"},"company":"XSPORTS","email":"[email protected]","phone":"[phone]","address":"850 Franklin Street, Gordon, Virginia, 4968","about":"Eiusmod nostrud mollit occaecat Lorem consectetur enim pariatur qui eu. Proident aliqua sunt incididunt Lorem adipisicing ea esse do ullamco excepteur duis qui. Irure labore cillum aliqua officia commodo incididunt esse ad duis ea. Occaecat officia officia laboris veniam id dolor minim magna ut sit. Aute quis occaecat eu veniam. Quis exercitation mollit consectetur magna officia sit. Irure ullamco laborum cillum dolore mollit culpa deserunt veniam minim sunt.","registered":"Monday, February 3, 2014 9:19 PM","latitude":"-82.240949","longitude":"2.361739","tags":["nostrud","et","non","eiusmod","qui"],"range":[0,1,2,3,4,5,6,7,8,9],"friends":[{"id":0,"name":"Meyers Dillard"},{"id":1,"name":"Jacobson Franco"},{"id":2,"name":"Hunt Hernandez"}],"greeting":"Hello, Lacey! You have 8 unread messages.","favoriteFruit":"apple"},{"_id":"5cdab57eae2f9bc5184f1768","index":2,"guid":"3c0de017-1c2a-470e-87dc-5a6257e8d9d9","isActive":true,"balance":"`$3,349.49","picture":"http://placehold.it/32x32","age":20,"eyeColor":"green","name":{"first":"Knowles","last":"Farrell"},"company":"DAYCORE","email":"[email protected]","phone":"[phone]","address":"150 Bath Avenue, Marion, Oregon, 991","about":"Eiusmod sint commodo eu id sunt. Labore esse id veniam ea et laborum. Dolor ad cupidatat Lorem amet. Labore ut commodo amet commodo. Ipsum reprehenderit voluptate non exercitation anim nostrud do. Aute incididunt ad aliquip aute mollit id eu ea. Voluptate ex consequat velit commodo anim proident ea anim magna amet nisi dolore.","registered":"Friday, September 28, 2018 7:51 PM","latitude":"-11.475201","longitude":"-115.967191","tags":["laborum","dolor","dolor","magna","mollit"],"range":[0,1,2,3,4,5,6,7,8,9],"friends":[{"id":0,"name":"Roxanne Griffith"},{"id":1,"name":"Walls Moore"},{"id":2,"name":"Mattie Carney"}],"greeting":"Hello, Knowles! You have 8 unread messages.","favoriteFruit":"strawberry"},{"_id":"5cdab57e80ff4c4085cd63ef","index":3,"guid":"dca20009-f606-4b99-af94-ded6cfbbfa38","isActive":true,"balance":"`$2,742.32","picture":"http://placehold.it/32x32","age":26,"eyeColor":"brown","name":{"first":"Ila","last":"Hardy"},"company":"OBLIQ","email":"[email protected]","phone":"[phone]","address":"605 Hillel Place, Herald, Delaware, 9670","about":"Enim eiusmod laboris amet ex laborum do dolor qui occaecat ex do labore quis sunt. Veniam magna non nisi ipsum occaecat anim ipsum consectetur ex laboris aute ut consectetur. Do eiusmod tempor dolore eu in dolore qui anim non et. Minim amet exercitation in in velit proident sint aliqua Lorem reprehenderit labore exercitation.","registered":"Friday, April 21, 2017 6:33 AM","latitude":"64.864232","longitude":"-163.200794","tags":["tempor","eiusmod","mollit","aliquip","aute"],"range":[0,1,2,3,4,5,6,7,8,9],"friends":[{"id":0,"name":"Duncan Guy"},{"id":1,"name":"Jami Maxwell"},{"id":2,"name":"Gale Hutchinson"}],"greeting":"Hello, Ila! You have 7 unread messages.","favoriteFruit":"banana"},{"_id":"5cdab57ef1556326f77730f0","index":4,"guid":"f2b3bf60-652f-414c-a5cf-094678eb319f","isActive":true,"balance":"`$2,603.20","picture":"http://placehold.it/32x32","age":27,"eyeColor":"brown","name":{"first":"Turner","last":"King"},"company":"DADABASE","email":"[email protected]","phone":"[phone]","address":"915 Quay Street, Hinsdale, Texas, 9573","about":"Consequat sunt labore tempor anim duis pariatur ad tempor minim sint. Nulla non aliqua veniam elit officia. Ullamco et irure mollit nulla do eiusmod ullamco. Aute officia elit irure in adipisicing et cupidatat dolor in sint elit dolore labore. Id esse velit nisi culpa velit adipisicing tempor sunt. Eu sunt occaecat ex pariatur esse.","registered":"Thursday, May 21, 2015 7:44 PM","latitude":"88.502961","longitude":"-119.654437","tags":["Lorem","culpa","labore","et","nisi"],"range":[0,1,2,3,4,5,6,7,8,9],"friends":[{"id":0,"name":"Leanne Lawson"},{"id":1,"name":"Jo Shepard"},{"id":2,"name":"Effie Barnes"}],"greeting":"Hello, Turner! You have 6 unread messages.","favoriteFruit":"apple"},{"_id":"5cdab57e248f8196e1a60d05","index":5,"guid":"875a12f0-d36a-4e7b-aaf1-73f67aba83f8","isActive":false,"balance":"`$1,001.89","picture":"http://placehold.it/32x32","age":38,"eyeColor":"blue","name":{"first":"Petty","last":"Langley"},"company":"NETUR","email":"[email protected]","phone":"[phone]","address":"677 Leonard Street, Ticonderoga, Utah, 1152","about":"Nisi do quis sunt nisi cillum pariatur elit dolore commodo aliqua esse est aute esse. Laboris esse mollit mollit dolor excepteur consequat duis aute eu minim tempor occaecat. Deserunt amet amet quis adipisicing exercitation consequat deserunt sunt voluptate amet. Ad magna quis nostrud esse ullamco incididunt laboris consectetur.","registered":"Thursday, July 31, 2014 5:16 PM","latitude":"-57.612396","longitude":"103.91364","tags":["id","labore","deserunt","cillum","culpa"],"range":[0,1,2,3,4,5,6,7,8,9],"friends":[{"id":0,"name":"Colette Mullen"},{"id":1,"name":"Lynnette Tanner"},{"id":2,"name":"Vickie Hardin"}],"greeting":"Hello, Petty! You have 9 unread messages.","favoriteFruit":"banana"},{"_id":"5cdab57e4df76cbb0db9be43","index":6,"guid":"ee3852fe-c597-4cb6-a336-1466e8978080","isActive":true,"balance":"`$3,087.87","picture":"http://placehold.it/32x32","age":33,"eyeColor":"brown","name":{"first":"Salas","last":"Young"},"company":"PLAYCE","email":"[email protected]","phone":"[phone]","address":"927 Elm Place, Terlingua, North Carolina, 2150","about":"Laborum laboris ullamco aliquip occaecat fugiat sit ex laboris veniam tempor tempor. Anim quis veniam ad commodo culpa irure est esse laboris. Fugiat nostrud elit mollit minim. Velit est laborum ut quis anim velit aute enim culpa amet ipsum.","registered":"Thursday, October 1, 2015 10:59 AM","latitude":"-57.861212","longitude":"69.823065","tags":["eu","est","et","proident","nisi"],"range":[0,1,2,3,4,5,6,7,8,9],"friends":[{"id":0,"name":"Day Solomon"},{"id":1,"name":"Stevens Boyd"},{"id":2,"name":"Erika Mayer"}],"greeting":"Hello, Salas! You have 10 unread messages.","favoriteFruit":"apple"},{"_id":"5cdab57ed3c91292d30e141d","index":7,"guid":"ef7c0beb-8413-4f39-987f-022c4e8ec482","isActive":false,"balance":"`$2,612.45","picture":"http://placehold.it/32x32","age":36,"eyeColor":"brown","name":{"first":"Gloria","last":"Black"},"company":"PULZE","email":"[email protected]","phone":"[phone]","address":"311 Guernsey Street, Hatteras, New Mexico, 2241","about":"Laborum sunt exercitation ea labore ullamco dolor pariatur laborum deserunt adipisicing pariatur. Officia velit duis cupidatat eu officia magna magna deserunt do. Aliquip cupidatat commodo duis aliquip in aute dolore occaecat esse ad. Incididunt est magna in pariatur ut do ex sit minim cupidatat culpa. Voluptate eu veniam cupidatat exercitation.","registered":"Friday, June 26, 2015 7:59 AM","latitude":"38.644208","longitude":"-45.481555","tags":["sint","ea","anim","voluptate","elit"],"range":[0,1,2,3,4,5,6,7,8,9],"friends":[{"id":0,"name":"Abby Walton"},{"id":1,"name":"Elsa Miranda"},{"id":2,"name":"Carr Abbott"}],"greeting":"Hello, Gloria! You have 5 unread messages.","favoriteFruit":"strawberry"},{"_id":"5cdab57edc91491fb70b705d","index":8,"guid":"631ff8a0-ce4c-4111-b1e4-1d112f4ecdc7","isActive":false,"balance":"`$2,550.70","picture":"http://placehold.it/32x32","age":25,"eyeColor":"brown","name":{"first":"Deirdre","last":"Huber"},"company":"VERBUS","email":"[email protected]","phone":"[phone]","address":"814 Coles Street, Bartonsville, Tennessee, 7313","about":"Ipsum ex est culpa veniam voluptate officia consectetur quis et irure proident pariatur non. In excepteur est aliqua duis duis. Veniam consectetur cupidatat reprehenderit qui qui aliqua.","registered":"Monday, April 1, 2019 2:33 AM","latitude":"-75.702323","longitude":"45.165458","tags":["labore","aute","nisi","laborum","laborum"],"range":[0,1,2,3,4,5,6,7,8,9],"friends":[{"id":0,"name":"Genevieve Clarke"},{"id":1,"name":"Black Sykes"},{"id":2,"name":"Watson Hudson"}],"greeting":"Hello, Deirdre! You have 8 unread messages.","favoriteFruit":"strawberry"}]
"@
)

System.Diagnostics.Process does have limitations in when used, that's why I had to write my own runner with abilities to capture STDOUT STDERR and combination of then into variables, which is sufficient for my use cases, but not perfect.

Edit: like you said its goes seems have edge cases in Windows Poweshell 5. I tested on Powershell 6 and it works great! unfortunately i have to deal with Poweshell 5 until 7 takes it over in Windows case...

[mklement0]

Impressive, but here are couple that does not work for me on windows:

Yes, these limitations apply to _Windows PowerShell_, with executables for which support for "" escaping of embedded " cannot be assumed, as detailed in my previous comment.
If you're willing to assume support for "" escaping in all of your invocations (_most_, but not all executables on Windows do support it), you can easily tweak the function.

And, to confirm what you said in your edit: In PowerShell _Core_:

• iep echoargs 'somekey="value with spaces"' 'te\" st' works fine.
• Your test array of arguments seem to work fine too.

[choovick]

If you're willing to assume support for "" escaping in all of your invocations (_most_, but not all executables on Windows do support it), you can easily tweak the function.

Thanks I will give it a try in near future. I deal with sqlcmd on occasion and it does not support "" for sure. For that case - its easy to provide an option to skip escape logic for specific arguments.

[jiasli]

The way how Windows parses command-line arguments can be found at Parsing C++ command-Line arguments:

Microsoft C/C++ startup code uses the following rules when interpreting arguments given on the operating system command line:

• Arguments are delimited by white space, which is either a space or a tab.

• The caret character (^) is not recognized as an escape character or delimiter. The character is handled completely by the command-line parser in the operating system before being passed to the argv array in the program.

• A string surrounded by double quotation marks ("string") is interpreted as a single argument, regardless of white space contained within. A quoted string can be embedded in an argument.

• A double quotation mark preceded by a backslash (\") is interpreted as a literal double quotation mark character (").

• Backslashes are interpreted literally, unless they immediately precede a double quotation mark.

• If an even number of backslashes is followed by a double quotation mark, one backslash is placed in the argv array for every pair of backslashes, and the double quotation mark is interpreted as a string delimiter.

• If an odd number of backslashes is followed by a double quotation mark, one backslash is placed in the argv array for every pair of backslashes, and the double quotation mark is "escaped" by the remaining backslash, causing a literal double quotation mark (") to be placed in argv.

This is also discussed at _exec, _wexec Functions:

Spaces embedded in strings may cause unexpected behavior; for example, passing _exec the string "hi there" will result in the new process getting two arguments, "hi" and "there". If the intent was to have the new process open a file named "hi there", the process would fail. You can avoid this by quoting the string: "\"hi there\"".

How Python calls native executables

Python's subprocess.Popen can handle escaping correctly by converting args to a string in a manner described in Converting an argument sequence to a string on Windows. The implementation is subprocess.list2cmdline.

Hope this can shed some light on how PowerShell can handle this in a more elegant manner, instead of using --% or double escaping (following both PowerShell and CMD syntax). The current workarounds really bug Azure CLI (which is based on python.exe) customers.

[musm]

We still don't have a clear and concise method to understand how to handle this issue. Can anyone from the Powershell team shed light if this is simplified with v7?

[rkeithhill]

Well, the good news is that one of the focuses for PS 7.1 is to make it easier to invoke native commands. From a blog post on 7.1 investments:

Most native commands work just fine from within PowerShell, however, there are some cases where the argument parsing is not ideal (like handling quotes properly). The intent is to enable users to cut sample command lines for any popular native tool, paste it into PowerShell, and it just works without needing PowerShell specific escaping.

So maybe (hopefully) this will get addressed in 7.1

[mklement0]

Thanks, @rkeithhill, but doesn't:

The intent is to enable users to cut sample command lines for any popular native tool, paste it into PowerShell, and it just works without needing PowerShell specific escaping.

sound like another take on the inherently conceptually problematic --% (stop-parsing symbol)?

@SteveL-MSFT, can you tell us more about this upcoming feature?

---

To recap, the fix proposed here is that all you need to focus on is to satisfy _PowerShell_'s syntax requirements and that PowerShell takes care of all escaping _behind the scenes_ (on Windows; on Unix this is not even necessary anymore, now that .NET allows us to pass an array of verbatim tokens to the target program) - but there's no doubt that the fix would have to be opt-in, if backward compatibility must be maintained.

With this fix, _some_ fiddling with command lines for other shells will still be required, but it will be more straightforward - and, compared to --%, you retain the full power of PowerShell's syntax and variable expansions (expressions with (...), redirections, ...):

• You need to replace \" with `", but _only inside "..."_.

• You need to be aware that _no (other) shell_ is involved, so that Bash-style environment-variable references such as $USER will _not_ work (they will be interpreted as _PowerShell_ variables), unless you replace them with the equivalent PowerShell syntax, $env:USER.

  • As an aside: --% tries to compensate for that by - notably _invariably_ - expanding cmd.exe-style environment-variable references such as %USERNAME%, but note that it not only doesn't support Bash-style references ($USER) is passed _verbatim_ to the target program, but also unexpectedly expands cmd.exe-style references on Unix-like platforms, and doesn't recognize '...'-quoting.
  • See below for an alternative that _does_ involve the respective platform-native shell.

• You need to be aware that PowerShell has _additional_ metacharacters that require quoting/escaping for verbatim use; these are (note that @ is only problematic as an argument's _first_ char.):

  • for POSIX-like shells (e.g., Bash): @ { } ` (and $, if you want to prevent up-front expansion by PowerShell)
  • for cmd.exe: ( ) @ { } # `
  • Individually `-escaping such chars. is sufficient (e.g., printf %s `@list.txt).

A somewhat contrived example:

Take the following Bash command line:

# Bash
$ printf '"%s"\n' "3\" of snow"
"3" of snow"        # output

With the proposed fix in place , all that is needed is to replace the \" instances inside the "..."-enclosed argument with `":

# PowerShell - WISHFUL THINKING
PS> printf '"%s"\n' "3`" of snow"
"3" of snow"        # output

That is, you wouldn't need to worry about embedded " inside '...', and inside "..." you only need to escape them to make _PowerShell_ happy (which you could also do with "" here).

The above iep function implements this fix (as a stopgap), so that iep printf '"%s"\n' "3`" of snow" works as intended.

---

Contrast this with the current, broken behavior, where you need to jump through the following hoops to get the command to work as in Bash (inexplicably need for an _additional_ round of escaping with \):

# PowerShell - messy workaround to compensate for the current, broken behavior.
PS> printf '\"%s\"\n' "3\`" of snow"
"3" of snow"        # output

---

With the fix in place, those who want to use a given command line _as-is_ via the platform's _default shell_, will be able to use a verbatim _here-string_ to pass to sh -c (or bash -c) / cmd /c; e.g.:

# PowerShell - WISHFUL THINKING
PS> sh -c @'
printf '"%s"\n' "3\" of snow"
'@
"3" of snow"  # output

Note that use of --% does _not_ work here (printf --% '"%s"\n' "3\" of snow"), and the added advantage of the here-string-based approach is that the various --% limitations don't apply, notably the inability to use an output redirection (>).

If you switch to a _double_-quoted here-string (@"<newline>....<newline>"@), you can even embed _PowerShell variables and expressions_, unlike with --%; however, you then need to make sure that the expanded values don't break the syntax of the target shell.

We can think about a dedicated cmdlet with a succinct alias (e.g, Invoke-NativeShell / ins) for such calls (so that sh -c / cmd /c needn't be specified), but in order to pass complex command lines as-is, I don't think a way around using here-strings:

# PowerShell - WISHFUL THINKING
# Passes the string to `sh -c` / `cmd /c` for execution, as appropriate.
# Short alias: ins
PS> Invoke-NativeShell @'
printf '"%s"\n' "3\" of snow"
'@
"3" of snow"  # output

Of course, if you're relying on features of the platform-native shell, such calls will by definition be platform[-family]-specific - they won't work on _both_ Windows and Unix-like platforms.

This is the reason why relying on PowerShell _alone_, with its _own syntax_ is preferable in the long run: it provides a predictable cross-platform experience for calling external programs - even if that means that you cannot use command lines crafted for other shells as-is; as PowerShell gains in popularity, I expect the pain of discovering and knowing the required modifications to lessen, and I expect more and more documentation to show the PowerShell versions of command lines (too).

[aminya]

• You need to replace \" with `", but _only inside "..."_.

This does not work everywhere.

# working everywhere but polluted with \
❯ node -e 'console.log(\"hey\")'
hey

# working in Node:
❯ node -e 'console.log(`"hey`")'
hey

# not working in Julia:
❯ julia -e 'print(`"hey`")'
`hey`

# not working anywhere:
❯ node -e "console.log(`"hey`")"
❯ node -e "console.log("hey")"
❯ node -e "console.log(""hey"")"
❯ node -e 'console.log(""hey"")'

Bash syntax:

❯ node -e 'console.log("hey")'
hey

Powershell Suggestion:

If PowerShell team is just looking for a symbol, that does not break the previous syntax, why just not use something like a backticks ` for a Bash like behavior, which escapes literals automatically, and allows string interpolation. This is similar to JavaScript's syntax too.

❯ node -e `console.log("hey")`
hey

❯ $a=hey 
❯ node -e `console.log($hey)`
hey

[Artoria2e5]

why just not use something like a backticks ` for a Bash like behavior

Backticks are already used for escaping stuff, the same purpose as backslahes in POSIX shell. The comment you are referring to already points that out. We have used up all the ASCII quote-like stuff. Adding a $ prefix to normal string literals might work, but I don't think it makes enough sense.

---

The way how Windows parses command-line arguments can be found at...

The problem is that Windows MSVCR doesn't just do that: it handles corner cases in undocumented ways. The "" stuff is so solidly set that they even put it into CoreFX when they ported .NET to Unix. But anyway, it's always good enough for escaping, at least until someone asks for globbing.

There's also the classic problem of everyone doing it differently, but we don't need to worry about that because we always have .NET for raw cmdline.

[aminya]

why just not use something like a backticks ` for a Bash like behavior

Backticks are already used for escaping stuff, the same purpose as backslahes in POSIX shell. We have used up all the ASCII quote-like stuff.

There is a possibility to use a combination of symbols if the parser is not able to detect that here ` is introducing a string. Something like '' might even work.

[choovick]

@aminya here is a solution is you are not using old windows Powershell 5.1 and on 6+:
https://github.com/PowerShell/PowerShell/issues/1995#issuecomment-562334606

As I have to deal with PowerShell 5,1 on windows and 6+ on linux/mac, I have my own implementation that been working without issues for years that allows be to work with tools like kubectl, helm, terraform and others passing complex JSON objects within parameters:
https://github.com/choovick/ps-invoke-externalcommand

GitHub

choovick/ps-invoke-externalcommand

Contribute to choovick/ps-invoke-externalcommand development by creating an account on GitHub.

[AndrewSav]

@choovick somewhat shorter implementation was given above in this thread, I'm still to find a case where it would fail for me. This works in PS starting from v3.

[mklement0]

@AndrewSav @TSlivede's Run-Native function is very clever and concise, and commendably also works reliably in _Windows PowerShell_; a few things worth noting: of necessity, the child process sees the aux. commandlineargumentstring environment variable (probably rarely, if ever, a problem in practice), argument-less invocations are currently not handled correctly (easily fixed and not even a problem, if you make sure that you only ever use the function _with_ arguments, which is what it is for), _all_ arguments get double-quoted (e.g., something like 42 is passed as "42"), which on Windows can (unfortunately) have side effects for programs that interpret double-quoted (or partially double-quoted, as in the msiexec case) arguments differently.

@aminya, the only reason that node -e 'console.log(`"hey`")' (sort of) works is because of the current, broken behavior (see below). I assume what you meant to pass was _verbatim_ console.log("hey"), which, if PowerShell on Windows escaped things correctly as proposed here, you would pass as-is in single quotes: node -e 'console.log("hey")'. This should _automatically_ be translated to node -e "console.log(\"hey\")" (double-quoting, \-escaped verbatim ") _behind the scenes_.

Given how long this thread has become, let me try to recap:
You should only ever have to worry about _PowerShell_'s syntax requirements, and it is PowerShell's job _as a shell_ to ensure that the _verbatim argument values_ that result from PowerShell's own parsing are passed to the external program _as-is_.

• On Unix-like platforms, now that we have .NET Core support for it, doing this is trivial, as the verbatim values can just be passed as-is as the elements of an _array_ of arguments, which is how programs natively receive arguments there.
• On Windows, external programs receive arguments as a _single command-line string_ (a regrettable historical design decision) and must perform their own command-line parsing. Passing multiple arguments as part of a single string necessitates quoting and parsing rules in order to properly delineate arguments; while this is ultimately a free-for-all (programs are free to parse however they like), the most widely used syntax convention (as stated before and also proposed in the unfortunately now abandoned RFC) is what Microsoft's C/C++ compilers implement, so it makes sense to go with that.
  
  
  
  • _Update_: Even on Windows we can take advantage of the collection-of-verbatim-tokens ArgumentList property of System.Diagnostics.ProcessStartInfo in .NET Core: on Windows it is automatically translated to a properly quoted and escaped command-line string when the process is started; however, for _batch files_ we may still need special handling - see https://github.com/PowerShell/PowerShell-RFC/pull/90#issuecomment-552231174
  
  

Implementing the above is undoubtedly a massive breaking change, so it presumably requires an opt-in.
I think that going ahead with this is a must, if we want to git rid of all the current quoting headaches, which keep coming up and hamper PowerShell adoption, especially in the Unix world.

---

As for node -e 'console.log(`"hey`")': inside '...', don't use ` - unless you want that character to be passed through as-is. Because PowerShell currently _doesn't_ escape the verbatim " chars. in your argument as \" behind the scenes, what node sees on the command line is console.log(`"hey`"), which is parsed as two directly adjacent string literals: unquoted console.log(` and double-quoted "hey`". After stripping the ", which have _syntactic_ function due to not being \-escaped, the JavaScript code getting executed is ultimately console.log(`hey`), and that only happens to work because a `....` enclosed token is a form of string literal in JavaScript, namely a _template literal_.

[choovick]

@AndrewSav I have tested with my crazy test object and it did worked for me! Very elegant solution, tested on Windows 5.1 and PS 7 on linux. I'm ok with having everything double quoted, I don't deal with msiexec or sqlcmd also known to treat " explicitly.

My personal implementation also has simple escape logic similar to one you mention: https://github.com/choovick/ps-invoke-externalcommand/blob/master/ExternalCommand/ExternalCommand.psm1#L278

but I have wrote bunch of code to display and capture STDOUT and STDERR threads in realtime within that module... It probably can be greatly simplified, but I had no need...

@mklement0 this thread will never end (: We either need to provide published PowerShell module in the ps gallery that will suite most use cases and will be simple enough to use, or wait for upcoming shell improvements.

GitHub

choovick/ps-invoke-externalcommand

Contribute to choovick/ps-invoke-externalcommand development by creating an account on GitHub.

[aminya]

@mklement0 this thread will never end (: We either need to provide published PowerShell module in the ps gallery that will suite most use cases and will be simple enough to use, or wait for upcoming shell improvements.

If the PowerShell team decides to not fix this in the program itself, I would then say that a shell that cannot run external programs natively and correctly will not be the shell of my choice! These are the basic stuff that are missing from PowerShell.

[choovick]

@aminya yea, I found it that issue is very annoying myself when I had to move to it. But features like below made it worth it.

• Flexible parameters framework, easy to build reliable CMDlets.
• Modules and internal modules repositories to share common logic across organization, avoiding a lot of code duplication and centralization of core functionality that can be refactored in once place.
• Cross platform. I have folks running my tools on Windows in PS 5.1, and on linux/mac in PS 6/7

I really hope to PS team improves this in the future to make it less convoluted.

[iSazonov]

Implementing the above is undoubtedly a massive breaking change, so it presumably requires an opt-in.

Do you mean implementing https://github.com/PowerShell/PowerShell-RFC/pull/90?

[mklement0]

@aminya, I agree that calling external programs with arguments is a core mandate of a shell and must function properly.

A module, as suggested by @choovick, still makes sense for _Windows PowerShell_, which is in security-critical-fixes-only maintenance mode.
Complementarily, if/when the proposed conceptual help topic about calling external programs gets written - see https://github.com/MicrosoftDocs/PowerShell-Docs/issues/5152 - a helper function that corrects the problems in earlier versions / Windows PowerShell's, such as @TSlivede's above, could be posted there directly.

@iSazonov Yes, implementing https://github.com/PowerShell/PowerShell-RFC/pull/90 is what I meant.

As for the thread never ending and the breaking-change concerns:

The last official response to the linked RFC was this comment by @joeyaiello from 8 July 2019 (emphasis added):

but we think that this [RFC] makes a ton of sense irrespective of existing behavior and without regard for the breaking-ness of it. Now that we have experimental features, we think it's perfectly reasonable to go and implement this today behind an experimental feature flag, and we can figure out further down the line whether this is opt-in vs. opt-out behavior, whether there's some transition path, and if a preference variable is the right mechanism for turning it on and off.

_Personally_, I wouldn't mind fixing the behavior _by default_, even though that it is a breaking change; the RFC indeed proposes that and suggests an opt-in if you want the _old_ (broken) behavior.

I suspect those with legacy code to maintain will object, however, as _all existing workarounds will cease to function_ - see above - and maintaining backward compatibility still seems to be the overarching goal.

If the new, fixed behavior is made opt-in, you still have the awkwardness of having to do something just to get the right behavior, but at least existing code won't break.

But the existing opt-in mechanisms are themselves problematic:
Now that @KirkMunro's optional features RFC has been rejected, that pretty much leaves a _preference variable_, and the challenge there is PowerShell's dynamic scoping: third-party code called from a scope that opted-in that was not designed to use the new implementation could then break (unless the preference variable is temporarily reset).

_Lexical_ scoping of the opt-in is required here, which we currently don't have. The RFC for _lexical_ scoping of strict mode proposes implementation of a lexically scoped feature (or a different purpose), via a using statement (which, notably, is generally _dynamically_ scoped). Following this pattern, a lexically scoped using ProperExternalArgumentQuoting statement (the name is a WIP :) - if technically feasible - is worth considering.

We need the PowerShell committee to weigh in (again) and to provide clear guidance as to the way forward, with _timely_ feedback on questions as they arise. @SteveL-MSFT?

---

Note that a --%-like solution hinted at by the 7.1 blog post (see above) (which I personally think isn't worth pursuing - see above and this comment), would be a _separate_ feature - fixing PowerShell's native (non-emulation) behavior is still a must.

[musm]

Personally, I wouldn't mind fixing the behavior by default, even though that it is a breaking change; the RFC indeed proposes that and suggests an opt-in if you want the old (broken) behavior.

If the new, fixed behavior is made opt-in, you still have the awkwardness of having to do something just to get the right behavior, but at least existing code won't break.

Agreed, in fact I would argue it makes less sense to have a broken default, then a correctly implemented default. Given the fact that the current implementation is in fact a bug, the new behavior must be opt-in, not opt-out, since it really doesn't make sense to continue to encourage broken external shell calls that are prone to break in unexpected ways. In any case, PowerShell 7 should strive improve over the legacy Windows PowerShell.

[joeyaiello]

@SteveL-MSFT and I agreed we should close this one in favor of #13068. Anything we touch here is just too much of a breaking change, and we should address the problem with a new operator that serves as an opt-in mode.

[TSlivede]

I absolutely don't see how #13068 would resolve this: If that operator is introduced as intended we still have no way to properly call any native executable with a given array of arguments or with some explicit arguments whose content originates from variables.

The example, that @JustinGrote gave in that thread currently doesn't work reliably (if embedded quotes are possible in the argument payload) and adding that operator will not give any alternative that improves anything.

[TSlivede]

@joeyaiello Can you at least leave this issue open until that operator actually exists and somebody can show, how that operator would improve anything, that was mentioned in this thread?

Oh and also what about Linux? This issue is stupid and unexpected on Windows, but on Linux it makes even far less sense, especially as there is no loooong history of linux powershell scripts, that will break.

[Artoria2e5]

Making a special operator for this doesn't make sense for a command-line shell at all, since its main job is to launch programs and pass them arguments. Introducing a new operator that does system() for this job is like Matlab introducing a way to call calc.exe because it has a bug in its arithmetics. What should instead be done is that:

• The pwsh team prepares for a new major release that fixes the command-line stuff, moving the current behavior behind a built-in cmdlet.
• As a stop-gap solution, the upcoming pwsh version gets a built-in cmdlet that uses the new, correct behavior for command-line passing.

The same applies to Start-Process. (Actually it's a pretty good candidate for the "new" cmdlet with some options like -QuotingBehavior Legacy...) See #13089.

[yecril71pl]

Why is Powershell behaving differently in these two situations? Specifically, it is inconsistently wrapping args containing spaces in double-quotes.

I get consistent results in v.7. Seems fixed.

PING 'A \"B'

Ping request could not find host A "B.

PING 'A\" B'

Ping request could not find host A" B.

[mklement0]

It' isn't fixed, because the verbatim hostnames that ping should see are A \"B and A\" B - _with_ the \ characters.

PowerShell, as a shell, should parse the arguments according _its_ rules - only - and then transparently ensure that the target process sees the same verbatim values that were the result of PowerShell's own parsing.

That _other_ shells - and those poor programs running on Windows that must act like their own shell, in a manner of speaking, by having to parse a _command line_ just to extract the individual arguments passed - use \ as the escape character shouldn't enter the picture here - accommodating that (necessary on Windows only, on Unix you just pass the verbatim arguments directly as an array) is PowerShell's job as a shell, _behind the scenes_.

As an aside, just like PowerShell itself doesn't require escaping of " _inside '...'_ (single-quoted strings), neither do POSIX-compatible shells such as bash: executed from bash, for instance, /bin/echo 'A \"B' (sensibly) prints A \"B (the \ are treated as literals in single-quoted strings) - that executing the very same command from PowerShell (unexpectedly) yields
A "B - the \ is missing - is a manifestation of the problems discussed here.

[mklement0]

I should clarify:

• From the perspective of wanting to ultimately pass A "B _verbatim_, you should be able to use 'A "B' from PowerShell.

• The command line that PowerShell currently constructs behind the scenes contains "A "B" - which the target process see as A B - that is, the blind enclosure in "...", without escaping the _embedded_ " resulted in the effective loss of the embedded ". What PowerShell _should_ use in the behind-the-scenes command line in this case is "A \"B" - that is, the embedded " needs \-escaping.

• Similarly, the same blind enclosure causes 'A \"B' to be represented as "A \"B" on the behind-the-scenes command line, which just so happens to turn the _embedded_ \" into an _escaped_ " character, which the target process therefore sees as A "B; that is, the lack of _automatic_ escaping resulted in the effective loss of the embedded \. What PowerShell _should_ use in the behind-the-scenes command line in this case is "A \\\"B" - that is, both the \ and the " need escaping.

[yecril71pl]

It' isn't fixed, because the verbatim hostnames that ping should see are A \"B and A\" B - _with_ the \ characters.

"It" refers here to the quoted complaint, which I fortunately cannot reproduce.

[mklement0]

@yecril71pl, I see: my (incorrect) assumption was that the "inconsistently wrapping args containing spaces in double-quotes" refers to the _lack of automatic escaping of embedded " and \ characters_, as explained in my previous comment - and that is the crux of this issue.

There have been minor fixes in PowerShell Core that Windows PowerShell doesn't have; I can only think of one right now:

• Windows PowerShell uses blind double-quoting in case of a trailing \: 'A B\' turns into (broken) "A B\" - PS Core handles that correctly ("A B\\").

Given that this repo is for PS Core only, it's sufficient to focus on what's still broken in PS Core (it can be helpful to mention differences _as an aside_, but it's best to make that aspect explicit).

---

And even the scenario you had in mind _is_ still broken in PS Core - but only _if you omit the \ from the argument_:

Passing 'A" B' still results in _non_-double-quoted A" B behind the scenes (whereas 'A\" B' results in"A\" B" - which is also broken, as discussed - only differently).

[yecril71pl]

Given that this repo is for PS Core only, it's sufficient to focus on what's still broken in PS Core (it can be helpful to mention differences _as an aside_, but it's best to make that aspect explicit).

Meta aside: I find it useful to know that wrong behaviour mentioned in another user’s comment does not apply. Of course, we could have dismissed the comment merely because the user did not bother to check the current version. Well. Unruly reporters being unruly, it is still better to be sure. IMHO.

[mklement0]

No argument there, but providing _proper framing and context_ to such _asides_ is important, especially in a looooong thread where the original comment - needed for context - was posted a long time ago and is, in fact, now _hidden_ by default (it never hurts to actually _link_ to the original comment being quoted).

[aminya]

I wonder if these discussions make a difference. Clearly the committee's decisions are independent of what the community wants. Look at the tags: Resolution- won't fix. But since PowerShell is open source (MIT), the community can fix this in a separate fork, and call this PowerShellCommunity (pwshc) for short.

This removes the need for backward compatibility. Later in PowerShell 8, the committee might integrate the fork.

About backward compatibility: PowerShell does not come preinstalled on any operating system, and to me, the hardship of installing PowerShellCommunity is the same as PowerShell. I prefer to install the community version and use it right away instead of waiting for some future 8 version (or making the code more complex with a new operator).

[yecril71pl]

Since the Committee has decided to keep things broken, it is better to know how badly broken they are. I think the community can live with Invoke-Native that does the right thing. It is not the community’s job to save Microsoft’s face against their will.

[mklement0]

it is better to know how badly broken they are

I fully agree - even if the problem can't be helped in the moment, knowing how to do things right _in principle, if and when the time comes_ is important - even if that time _never_ comes in the context of a given language.

the community can live with Invoke-Native that does the right thing

To be clear:

• Something like Invoke-NativeShell addresses a _different use case_ - which is the subject of #13068 - and for that use case such a cmdlet is _not_ a stopgap: it is the proper solution - see https://github.com/PowerShell/PowerShell/issues/13068#issuecomment-656781439

• _This_ issue is about fixing _PowerShell itself_ (it is not about platform-_native_ functionality), and the _stopgap_ solution to that is to provide a low-ceremony _opt-in_ - hence the proposal to ship function iep as a built-in function, pending a _proper_ fix in a future version that is permitted to substantially break backward compatibility.

It is not the community’s job to save Microsoft’s face

I don't think @aminya's concern is about saving anyone's face - it's about fixing fundamentally broken behavior in an area that is a shell's core mandate.

That said, I'm not sure that fragmenting the PowerShell ecosystem with a fork is the right way to go.

[joeyaiello]

I don't have time to respond to all this at just this moment, but I think this is reasonable, so I'm re-opening:

Can you at least leave this issue open until that operator actually exists and somebody can show, how that operator would improve anything, that was mentioned in this thread?

[iSazonov]

As I said in related issue, native call operator adds complexity in UX and it is wrong direction.
At the same time, the whole discussion here is about simplifying interaction with native applications.

The only thing that stops us is that it is a breaking change. We have said many times that this is too much destruction, but let's weigh it.

Let's look at an interactive session. An user will install new PowerShell version and discover new behavior when invoking native applications. What will he say? I will speculate that he will say - "Thanks - finally I can just type and it works!".

Let's look at script execution/hosting scenario. Any new version of an application (even with one minor change!) can break a business process. We always expect this and check our processes after any updates. If we find a problem, then we have several ways:

• rollback the update
• prepare fast fix for script
• turn off a feature which break our script until these things are fixed or they themselves die over time.

_Since version updates always break something, the last option is the best that we could have and accept._

(I want to point out that right now PowerShell Core is not a component of Windows and therefore cannot spoil hosting applications directly, only scripts are affected directly.)

I want to remind you what Jason said above - this is a bug fix.
Let us fix it and simplify everything for everyone. Let users works powershell-y.

[yecril71pl]

As I said in related issue, native call operator adds complicity in UX and it is wrong direction.

complexity ❓

[yecril71pl]

Let's look at an interactive session. An user will install new PowerShell version and discover new behavior when invoking native applications. What will he say? I will speculate that he will say - "Thanks - finally I can just type and it works!".

I have a different scenario: new users give PowerShell a try, realise that it mysteriously fails, move it to the Recycle Bin and never return. That is what I would do.

prepare fast fix for script

That is something we should do to cover the obvious cases in user scripts.

[iSazonov]

realise that it mysteriously fails

The whole discussion here is just about getting rid of this "mysteriously fails". And it’s best to do this by simplifying but not adding new mysterious things.

[yecril71pl]

The whole discussion here is just about getting rid of this. And it’s best to do this by simplifying but not adding new mysterious things.

We do not want to get rid of the ability to directly invoke executable programs.

[Artoria2e5]

The old invocation is not direct to cmdline either with its guessing of whether something is already quoted. In addition, the said ability would still be retained with --%.

[yecril71pl]

In addition, the said ability would still be retained with --%.

--% requires a predefined command line, so its utility is limited.

[mklement0]

@yecril71pl

We do not want to get rid of the ability to directly invoke executable programs.

I think @iSazonov means getting rid _of the buggy behavior_, i.e. fixing the bug properly (without opt-in via extra syntax), even though it means breaking existing workarounds. Correct, @iSazonov?

@Artoria2e5:

The old invocation is not direct to cmdline either with its guessing of whether something is already quoted

[_Update_: I misread the quoted line, but hopefully the information is still of interest.]

• You don't have to _guess_, and the rule is simple:

  • _Only if_ your command name or path is _quoted_ - '/foo bar/someutil' - and/or contains _variable references (or expressions) - $HOME/someutil - & is _required_.

• While you may consider this need unfortunate, it is at the very heart of the PowerShell language, and necessitated by needing to able to syntactically distinguishing between the two fundamental parsing modes, argument mode and expression mode.

  • Note that the issue is not specific to calling _external programs_; native PowerShell commands too require & for invocation if specified via a quoted string / variable reference.

• If you don't want to memorize this simple rule, the simpler rule is: _always_ use &, and you'll be fine - it's somewhat less convenient than _not_ needing anything, but not exactly a hardship (and shorter than --%, which is absolutely the wrong solution - see below)

the said ability would still be retained with --%.

[_Update_: I misread the quoted line, but hopefully the information is still of interest.]

No, despite the unfortunate conflation of #13068 with _this_ issue, --% - the ability to call the _native shell_, using _its_, invariably _platform-specific_ syntax - is in no way a workaround for the issue at hand and discussing it as such only adds to the confusion.

--% is a very different use case and, as currently proposed, has severe limitations; If there's something _not_ worth introducing an _operator_ for (or changing the behavior of an existing one), it's the ability to pass a verbatim command line to the native shell (which, of course, you can already do yourself, with sh -c '...' on Unix, and cmd /c '...', _but only robustly if this issue gets fixed_; a binary Invoke-NativeShell / ins cmdlet implementation, while primarily abstracting away the details of the target shell CLI syntax, would avoid the issue at hand (by direct use of System.Diagnostics.ProcessStartInfo.ArgumentList), and can therefore be implemented independently).

[TSlivede]

@yecril71pl

I have a different scenario: new users give PowerShell a try, realise that it mysteriously fails, move it to the Recycle Bin and never return. That is what I would do.

Could you clarify: Do you fear, that the current behavior of powershell could lead to this unpleasant user experience, or do you fear that the proposed change leads to that userexperience.

Because in my opinion the current behavior of powershell is much more likely to generate such a experience. As said above: The current behavior of powershell should be considered a bug.

[yecril71pl]

Why would a new user, without any knowledge of PowerShell being broken, issue commands with contorted arguments?

[TSlivede]

@yecril71pl Just to be absolutely sure: You consider the currently required form for arguments "contorted", not the suggested fix.

[yecril71pl]

I consider your question as originating from your supposition that I am an incurable nerd. That is correct—but I have retained the ability to imagine what a normal random chap would consider contorted.

[TSlivede]

@mklement0
I think, that @Artoria2e5 was talking about converting the array of arguments to the single lpCommandLine string when saying

The old invocation is not direct to cmdline either with its guessing of whether something is already quoted

Because when calling

echoargs.exe 'some"complicated_argument'

you indeed have to more or less guess, whether powershell adds quotes around some"complicated_argument.

Example 1: In most powershell versions
echoarg.exe 'a\" b' and echoarg.exe '"a\" b"' will be translated to
"C:\path\to\echoarg.exe" "a\" b" (tested versions 2.0; 4.0; 6.0.0-alpha.15; 7.0.1)
but my default powershell on Win10 (version 5.1.18362.752) translates
echoarg.exe 'a\" b' to "C:\path\to\echoarg.exe" a\" b and
echoarg.exe '"a\" b"' to "C:\path\to\echoarg.exe" ""a\" b"".

Example 2: Older powershell versions translate
echoarg.exe 'a"b c"' to "C:\path\to\echoarg.exe" "a"b c"" (tested versions 2.0; 4.0;)
whereas newer versions translate
echoarg.exe 'a"b c"' to "C:\path\to\echoarg.exe" a"b c" (tested versions 5.1.18362.752; 6.0.0-alpha.15; 7.0.1).

---

As the behavior was clearly already changed multiple times, I don't get why it can't be changed one more time to get the expected behavior.

[mklement0]

I see, @TSlivede, thanks for clarifying, and sorry for the misinterpretation, @Artoria2e5.

As for the real problem: we're 100% in agreement - in fact, you should never even have to think about what lpCommandLine ends up being used behind the scenes on Windows; if PowerShell did the right thing, no one would have to (except for edge cases, but they are not PowerShell's fault, and that's when --% (as currently implemented) can help; with a proper fix, there will never be edge cases on Unix-like platforms).

As for simply fixing the problem properly: you certainly have my vote (but existing workarounds will break).

[yecril71pl]

TL;DR: The assumption that we can reliably pass any value as an argument to any program in the Microsoft Windows NT subsystem is wrong, so we should stop pretending that this is our goal. However, there is still much to be rescued if we consider the extent of the argument.

When invoking native Windows executables, we should preserve original quoting. Example:

CMD /CSTART="WINDOW TITLE"

The system cannot find the file WINDOW.

 { CMD /CSTART="WINDOW TITLE" }. Ast. EndBlock. Statements. PipelineElements. CommandElements[1]

StringConstantType

BareWord

Value

/CSTART=WINDOW TITLE

StaticType

System.String

Extent

/CSTART="WINDOW TITLE"

Parent

CMD /CSTART="WINDOW TITLE"

If we took the extent as the template, we would not lose anything and we could call the native executable as expected. The workaround of using a string argument works here but I do not think it is strictly technically necessary to do so, provided proper support gets implemented within PowerShell. This approach would work for all cases.

Quotation marks within quotations present an insurmountable problem because there are tools that interpret backslash escape (TASKLIST "\\\"PROGRAM FILES") and tools that do not (DIR "\""PROGRAM FILES" /B) and tools that do not bother (TITLE A " B). However, if we were to escape, the standard escape with backslashes poisons all common file managing tools because they simply do not support quotation marks at all and double backslashes \\ mean something entirely different to them (try DIR "\\\"PROGRAM FILES" /B), so sending an argument with a quotation mark inside should be a run-time error. But we cannot throw an error because we do not know which one is which. While using the normal escaping mechanism should not cause any harm to arguments that do not contain quotation marks, we cannot be sure that, when applied to arguments that do contain them and fed to a tool that does not support quotation marks as values, it would necessarily cause an aborting error rather than unexpected behaviour, and unexpected behaviour would be very bad indeed. This is a serious burden we place upon the user. In addition, we shall never be able to provide for ‘don’t care’ tools (CMD /CECHO='A " B').

Note that environment variables do not represent values in CMD, they represent code fragments that are reparsed as environment variables are expanded, and there is no provision for reliably treating them as arguments to other commands. CMD just does not operate on objects on any kind, not even strings, which seems to be the root cause of the present conundrum.

[Dabombber]

TL;DR: The assumption that we can reliably pass any value as an argument to any program in the Microsoft Windows NT subsystem is _wrong_, so we should stop pretending that this is our goal.

That should be the goal though shouldn't it? It's not PowerShell's problem if a program can't interpret arguments it receives.

When invoking native Windows executables, we should preserve original quoting. Example:

CMD /CSTART="WINDOW TITLE"

Are you suggesting calling a program should dynamically change the language from PowerShell to whatever the invoked program uses? You wrote that example in PowerShell, which means it should be equivalent to any of the following

CMD "/CSTART=WINDOW TITLE"
CMD '/CSTART=WINDOW TITLE'
CMD /CSTART=WINDOW` TITLE

[yecril71pl]

TL;DR: The assumption that we can reliably pass any value as an argument to any program in the Microsoft Windows NT subsystem is _wrong_, so we should stop pretending that this is our goal.

That should be the goal though shouldn't it? It's not PowerShell's problem if a program can't interpret arguments it receives.

The program CMD can interpret the argument /CECHO=A " B but PowerShell cannot pass it without distorting it.

When invoking native Windows executables, we should preserve original quoting. Example:

CMD /CSTART="WINDOW TITLE"

Are you suggesting calling a program should dynamically change the language from PowerShell to whatever the invoked program uses? You wrote that example in PowerShell, which means it should be equivalent to any of the following

CMD "/CSTART=WINDOW TITLE"
CMD '/CSTART=WINDOW TITLE'
CMD /CSTART=WINDOW` TITLE

I tried to suggest that, when interfacing with external programs under Microsoft Windows NT subsystem, PowerShell has a myriad ways to encode the arguments that are all equivalent to PowerShell but not equivalent to the receiving program. Being blunt and forcing the One True Way™ of encoding arguments, without paying attention to what quoting arrangement the user actually used, is not helpful, to say it mildly.

[imgx64]

@yecril71pl I'm really confused by your comments. What exactly are you proposing here? Your use cases are all covered by --%. You dismissed it earlier by saying

--% requires a predefined command line, so its utility is limited.

But in fact you can use environment variables with --%. Try This:

PS > $env:mytitle='WINDOW TITLE'
PS > cmd --% /CSTART="%mytitle%"

So what am I missing?

[yecril71pl]

We are missing the syntax CMD /CSTART="$mytitle", without leaking things to ENV:.

[Artoria2e5]

As a terrible idea, we do have the option of replacing Environment.ExpandEnvironmentVariables with something else. There's no native implementation on Unix anyways, and I don't believe the stuff it processes would become performance-critical when rewritten in C#.

Since equal signs are not allowed in env var names anyways, we can have %=$a% mean $a. This wouldn't break anything existing while allowing for some very flexible (and possibly bad) extensions like making it work like JS's template strings. Hell, we can define %VARNAME=$var% as some sort of fallback syntax too.

As for the documentation hell this would cause... I apologize.

[mklement0]

• We do _not_ have a parsing problem.

• What we do have is a problem with _how PowerShell passes the verbatim, stringified arguments that have resulted from _its_ parsing to external (native) executables_:

  • On _Windows_, the problem is that the command line to invoke the external executable with that is constructed behind the scenes does _not_ adhere to the most widely used convention for quoting arguments, as detailed in the the Microsoft C/C++ compiler documentation's Parsing C++ command-Line arguments section.

  • What happens currently isn't even that a _different_ convention is used: presumably due to an oversight, the command lines that are constructed are situationally _syntactically fundamentally broken_, depending on the specifics of the arguments, relating to a combination of embedded double quotes and spaces as well as empty-string arguments.

  • Ultimately, the problem is the fundamental architecture of process creation on Windows: You're forced to encode the arguments to pass to a process _as a command line_ - a single string representing _all_ arguments - rather than passing them as an _array_ of arguments (which is how Unix-like platforms do it). The need to pass a command line requires _quoting and escaping rules_ to be implemented, and it is ultimately _up to each program_ how to interpret the command line it is given. In effect, this amounts to needlessly forcing programs to be a mini-shell of sorts: they're forced to _re-perform_ the task that the shell has already performed, which is task that should be the purview of a _shell only_ (as is the case on Unix), namely parsing a command line into individual arguments. In a nutshell, this is the anarchy that is argument passing on Windows.

  • In practice, the anarchy is mitigated by _most_ programs adhering to the aforementioned convention, and new programs being developed are highly likely to adhere to that convention, primarily because widely used runtimes underpinning console applications implement these conventions (such as the Microsoft C/C++ / .NET runtimes). The sensible solution is therefore:

    • Make PowerShell adhere to this convention when building the command line behind the scenes.
    • For "rogue" programs that do _not_ adhere to this convention - which notably includes cmd.exe, batch files, and Microsoft utilities such as msiexec.exe and msdeploy.exe - provide a mechanism to explicitly control the command line passed to the target executable; this is what --%, the stop-parsing symbol provides - albeit quite awkwardly.

  • On _Unix_, the problem is that _a command line is being constructed at all_ - instead, the array of verbatim arguments should be passed _as-is_, which .NET Core now supports (since v2.1, via ProcessStartInfo.ArgumentList; it should _always_ have supported this, given that - sensibly - _there are no command lines_, only argument arrays, when a process is created on Unix-like platforms).

  • Once we use ProcessStartInfo.ArgumentList, all problems on Unix go away.

Fixing these issues is what @TSlivede's https://github.com/PowerShell/PowerShell-RFC/pull/90 is all about.

In https://github.com/PowerShell/PowerShell-RFC/pull/90#issuecomment-650242411 I've proposed additionally automatically compensating for the "roguishness" of batch files, given their still very widespread use as CLI entry points for high-profile software such as Azure (CLI az is implemented as a batch file, az.cmd).
Similarly, we should consider doing the same for msiexec.exe and msdeploy.exe and perhaps other high-profile "rogue" Microsoft CLIs.

---

I've just published a module, Native, (Install-Module Native -Scope CurrentUser) that addresses all of the above via its ie function (short for invoke (external) executable; it is a more complete implementation of the iep function introduced above).

It also includes ins (Invoke-NativeShell), which addresses #13068, and dbea (Debug-ExecutableArguments) for diagnosing argument passing - see https://github.com/PowerShell/PowerShell/issues/13068#issuecomment-671572939 for details.

In other words: ie can serve as an unobtrusive stopgap while we wait for this issue to be fixed, simply by prefixing invocations with ie as the command:

Instead of:

# This command is currently broken, because the '{ "name": "foo" }' argument isn't properly passed.
curl.exe -u jdoe  'https://api.github.com/user/repos' -d '{ "name": "foo" }'

you'd use the following:

# OK, thanks to `ie`
ie curl.exe -u jdoe  'https://api.github.com/user/repos' -d '{ "name": "foo" }'

---

As for the CMD /CSTART="WINDOW TITLE" example (whose more idiomatic form is cmd /c start "WINDOW TITLE", which does already work):

It is in essence the same problem as with prop="<value with spaces>" arguments for msiexec / msdeploy: PowerShell - justifiably - transforms /CSTART="WINDOW TITLE" into "/CSTART=WINDOW TITLE", which, however breaks the cmd.exe invocation.

There are two ways to resolve this:

• Delegate to ins / Invoke-NativeShell (note that the use of cmd.exe /c is effectively implied):

  • ins 'START="WINDOW TITLE"'
  • If you use an _expandable_ string, you can embed PowerShell values in the command string.
    
    
    
    • $title = 'window title'; ins "START=`"$title`""
    
    

• Alternatively, use the current --% implementation, but beware its limitations:

  • cmd --% /CSTART="WINDOW TITLE"
  • As discussed, a problematic limitation of --% is that the only way to embed _PowerShell_ values is to use an _aux. environment variable_ and reference it with %...% syntax:
    
    
    
    • $env:_title = 'window title'; cmd --% /CSTART="%_title%"
    
    
    • To avoid this limitation, --% should always have been implemented with a _single_ string argument - e.g.,
      
      cmd --% '/CSTART="WINDOW TITLE"' or cmd --% "/CSTART=`"$title`"" - but this can't be changed without breaking backward compatibility, so a _new_ symbol would have to be introduced - personally, I don't see the need for one.
    
    

[yecril71pl]

• they're forced to _re-perform_ the task that the shell has already performed

I do not think CMD.EXE splits command lines into arguments, the only thing that is needed is to find out which executable to call and the rest is just the command line as written by the user (after environment variable substitutions, which are done without any regard for argument boundaries). Of course, internal shell commands are an exception here.

It is in essence the same problem as with prop="<value with spaces>" arguments for msiexec / msdeploy

I am not a confident user of either, so I preferred to bring up something I am more familiar with.

[mklement0]

To be clear: the following has no impact on the points made in my previous comment.

I do not think CMD.EXE splits command lines into arguments

• It can get away without _explicit_ splitting when calling _external executables_ (commands run by another executable in a child process), but it does have to do it for _batch files_.

• Even when calling external executables it needs to be _aware_ of argument boundaries, so as to determine whether a given metacharacter (e.g. &) has _syntactic_ function or whether it is part of a double-quoted argument and therefore to be treated as a literal:

:: OK - the "..." around & tells cmd.exe to use it verbatim
C:\>echoArgs.exe one "two & three"
Arg 0 is <one>
Arg 1 is <two & three>

Command line:
"C:\ProgramData\chocolatey\lib\echoargs\tools\EchoArgs.exe" one "two & three"

Also, cmd.exe recognizes _embedded_ " chars. in "..." strings are recognized if escaped as "":

:: OK - the "" is recognized as an escaped "
C:\>echoArgs.exe "3"" of rain & such."
Arg 0 is <3" of rain & such.>

Command line:
"C:\ProgramData\chocolatey\lib\echoargs\tools\EchoArgs.exe" "3"" of rain & such."

Unfortunately, cmd.exe _only_ supports (the Windows-only) "" and not also the more widely used \" (which is what POSIX-like _shells_ on Unix _exclusively_ use - note: _shells_, not _programs_, because programs just see the array of verbatim arguments that result from the shell's parsing).

While most CLIs on Windows support _both_ "" and \", some _only_ understand \" (notably Perl and Ruby), and then you're in trouble:

:: !! BROKEN: cmd.exe misinterprets the & as *unquoted*, thinks it's the statement-sequencing operator, 
:: !! and tries to execute `such`:
C:\>echoArgs.exe "3\" of rain & such."
Arg 0 is <3" of rain >

Command line:
"C:\ProgramData\chocolatey\lib\echoargs\tools\EchoArgs.exe" "3\" of rain

'such."' is not recognized as an internal or external command,
operable program or batch file.

Therefore:

• Avoid calling cmd.exe directly, if possible.

  • Call external executables directly (once this issue is fixed) or via ie (for now), using _PowerShell's_ syntax.

• If you do have to call cmd.exe, use ins / Invoke-NativeShell for general simplicity and, specifically, for how easy it is to embed PowerShell variable and expression values into the command line.

  • A legitimate reason to still call cmd.exe directly is to compensate for PowerShell's lack of support for raw byte data in the pipeline - see this SO answer for an example.

[joeyaiello]

I know I'm going to catch a lot of flak here, and I really appreciate the depth of the discussion happening, but...ducks...does anyone have an example of any of this actually mattering in a real-world scenario?

It's my take that we are not empowered in PowerShell to solve the "anarchy" that currently exists with Windows argument parsing. And for many of the same reasons that we can't solve the problem, there's a good reason that Windows and the VC++ compilers have chosen not to break this behavior. It's rampant, and we're only going to create a really long tail of new (and largely undecipherable) problems if we change things.

For those utilities which are already cross-platform and in heavy use between Windows and Linux (e.g. Docker, k8s, Git, etc.), I don't see this problem manifesting in the real world.

And for those "rogue" applications that do a poor job: they're largely legacy, Windows-only utilities.

I agree that what you've described @mklement0 is largely a "correct" solution. I just don't know how to get there without really screwing things up.

[cspotcode]

Pretty basic usages break:

❯ git commit --allow-empty -m 'this is what we call a "commit message" which contains arbitrary text, often with punctuation'
error: pathspec 'message which contains arbitrary text, often with punctuation' did not match any file(s) known to git
❯ $a = 'this is what we call a "commit message" which contains arbitrary text, often with punctuation'
❯ git commit --allow-empty -m "$a"
error: pathspec 'message which contains arbitrary text, often with punctuation' did not match any file(s) known to git

❯ $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.0.3
PSEdition                      Core
GitCommitId                    7.0.3
OS                             Microsoft Windows 10.0.19042
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

[yecril71pl]

git.exe is a well-behaved application, so the fix in PowerShell will be straightforward, albeit requiring all scripters out there to revert their smart work-arounds. cmd.exe is harder to adapt to and requires a much more considerate approach that may solve a few problems but probably not all of them. Which is actually appalling, considering that PowerShell started as a Windows NT tool. I understand this question as _whether there is a real-life scenario when an ill-behaved legacy utility like cmd.exe will be called from PowerShell in a way that causes problems in the interface_. PowerShell tried to approach this problem by duplicating most of the functionality in cmd.exe, so as to make cmd.exe redundant. This is also possible for other tools, for example MSI can be operated via ActiveX, although doing so requires considerable knowledge. So is there anything essential that is not covered?

[SteveL-MSFT]

@PowerShell/powershell-committee discussed this. We appreciate the git example which clearly shows a real world compelling example. We agreed that we should have an experimental feature early in 7.2 to validate the impact of taking such a breaking change. An additional test example shows that even --% has a problem even though it should have been unparsed:

PS> testexe --% -echoargs 'a b c "d e f " g h'
Arg 0 is <'a>
Arg 1 is <b>
Arg 2 is <c>
Arg 3 is <d e f >
Arg 4 is <g>
Arg 5 is <h'>

This appears to be a problem in the native command parameter binder.

[joeyaiello]

Yeah, thank you @cspotcode. That example was definitely an aha moment for me (especially considering I've actually hit that one in the real world).

I'm still concerned about the breaking change aspect, and it's my take that this is a could candidate for an experimental feature that may remain experimental over multiple versions of PowerShell, and that is absolutely not something we're sure will eventually make it.

I also need to dig in more to understand the allow list / "rouge app" aspect of your RFC, @mklement0, as I'm not sure how much we want to sign up to maintain a list like that.

[mklement0]

@joeyaiello and @SteveL-MSFT, let me make a meta observation first:

While it's good to see that @cspotcode's example gave you a _glimpse_ of the problem, your responses still betray a fundamental lack of understanding and appreciation of the (magnitude of the) underlying problem (I will argue this point in a later comment).

This is not a _personal_ judgment: I fully recognize how difficult it must be to be stretched very thin and to have to make decisions on a very wide range of subjects in a short amount of time.

However, this points to a _structural_ problem: To me it seems that decisions are routinely made by the @PowerShell/powershell-committee on the basis of a superficial understanding of the problems being discussed, to the detriment of the community at large.

To me, the committee's response to the issue being discussed here is the most consequential example of this structural problem to date.

Therefore, I ask you to consider this:

How about appointing subject-matter-specific _sub_-committees that the committee consults with that _do_ have the required understanding of the issues involved?

[musm]

can you share the content of testexe SteveL-MSFT, just want to make sure !

[mklement0]

@TSlivede summarized the problem aptly in https://github.com/PowerShell/PowerShell/issues/13068#issuecomment-665125375:

PowerShell on the other hand claims to be a shell (until #1995 is solved, I won't say that it _is_ a shell)

As stated many times before, a core mandate of a shell is to call external executables with arguments.

PowerShell currently fails to fulfill this mandate, given that arguments with embedded double quotes and empty-string arguments aren't passed correctly.

As stated before, this may have been less of a problem in the Windows-only days, where the lack of capable external CLIs rarely surfaced this problem, but these days are gone, and if PowerShell wants to establish itself as a credible cross-platform shell, it must address this problem.

@cspotcode's git example is a good one; any executable to which you want to pass a JSON string - e.g., curl - is another:

# On Unix; on Windows, 
#   echoArgs.exe '{ "foo": "bar" }' 
# would show the same problem.
PS> /bin/echo '{ "foo": "bar" }'
{ foo: bar }  # !! Argument was incorrectly passed.

Leaving backward compatibility aside:

• On Unix, the problem is trivially and _completely_ solved by using ProcessStartInfo.ArgumentList behind the scenes.

• On Windows, the problem is trivially and _mostly_ solved by using ProcessStartInfo.ArgumentList behind the scenes.

  • For edge cases ("rogue" CLIs), there's the (poorly implemented) --%
  • As a _courtesy_, we can compensate for certain well-known edge cases to lessen the need for --% - see below.

Therefore, _as soon as possible_, one of the following choices must be made:

• Realize the importance of making argument-passing work properly and _fix it at the expense of backward compatibility_.

• If backward compatibility is really paramount, provide a new operator or a function such as the ie function from the Native module that fixes the problem, and widely publicize it as the only reliable way to invoke external executables.

Proposing an _experimental_ feature to address a badly broken fundamental feature is wholly inadequate.

---

@SteveL-MSFT

Even considering use of --% as the solution to this problem is fundamentally misguided:

It is a _Windows-only_ feature that knows only "..." quoting and %...%-style environment-variable references.

On Unix, the concept of "stopping parsing" fundamentally doesn't apply: _there is no command line_ to pass to child processes, only arrays of arguments.

Thus, _someone_ has to parse the command line into arguments _before_ invocation, which is implicitly delegated to the ProcessStartInfo class, via its .Arguments property, which on Unix uses the _Windows_ conventions for parsing a command line - and therefore recognizes "..." quoting (with escaping of embedded " as "" or \") only.

--% is a Windows-only feature whose only legitimate purpose is to call "rogue" CLIs.

---

@joeyaiello

that Windows and the VC++ compilers have chosen not to break this behavior.

The VC++ compiler _imposes a sensible, widely observed convention_, to bring order to the anarchy.

It is precisely adherence to this convention that is being advocated for here, which use of ProcessStartInfo.ArgumentList would automatically give us.

_This alone will cover the vast majority of calls. Covering ALL calls is impossible and indeed not PowerShell's responsibility._

As stated, for "rogue" CLIs that require non-conventional forms of quoting, --% must be used (or ins / Invoke-NativeShell from the Native module).

_As a courtesy_, we can automatically compensate for well-known "rogue" scenarios, namely calling batch files and certain high-profile Microsoft CLIs:

• The batch-file case is a generic one, and easily explained and conceptualized (e.g, pass a&b as "a&b", even though it shouldn't require quoting) - it will avoid the need for use of --% with all CLIs that use batch files as their entry point (which is quite common), such as Azure's az.cmd

• The alternative to hard-coding exceptions for specific CLIs - which admittedly can get confusing - is to detect the following _pattern_ in the arguments that result from PowerShell's parsing - <word>=<value with spaces> - and, instead of passing "<word>=<value with spaces>", as currently happens, to pass <word>="<value with spaces>"; the latter satisfies the "rogue" CLIs, while also being accepted by convention-adhering CLIs; e.g., echoArgs "foo=bar baz" ultimately sees the same first argument as echoArgs --% foo="bar baz"

[daxian-dbw]

@musm You can find the source code of TestExe at https://github.com/PowerShell/PowerShell/blob/master/test/tools/TestExe/TestExe.cs.

GitHub

PowerShell/PowerShell

PowerShell for every system! Contribute to PowerShell/PowerShell development by creating an account on GitHub.

[Dabombber]

I think that accommodating exceptions by default is just going to lead to a similar situation as the current one, where people need to revert PowerShell's "helpfulness". If there are exceptions, it should be obvious they're being applied.

Maybe something like:

# Arguments passed correctly, without regard for the program's ability to handle them
& $program a "" 'c "d e" f'
# Try to pass the arguments intelligently based on the program being called
&[] $program a "" 'c "d e" f'
# Escape the arguments for a batch file, eg) " -> ""
&[bat] $program a "" 'c "d e" f'

I'm really struggling to find syntax for this which isn't broken. At least this sort of makes sense if you think of it as casting the program, but casting the actual variable containing the program would require enclosing parenthesis.

That, in addition to allowing people to add exceptions for whatever broken behaviour they desire should hopefully eliminate the need for --%. The exception class they register would then have a method for determining if it's applicable to the program (for intelligent invocation), and an escaping method where you just throw the command abstract syntax tree at it and it returns the argument array/string.

[yecril71pl]

• instead of passing "<word>=<value with spaces>", as currently happens, to pass <word>="<value with spaces>"

The quotes used to construct the command line should follow the way the call is quoted in PowerShell. Therefore:

1. A call that does not contain double quotes in its text should put double quotes around the whole value.
2. A call that contains double quotes should retain them as written in the script _if possible_.

In particular:
| script argument | command line |
| ----------------- | ---------------- |
| p=l of v | "p=l of v" |
| p=l` of` v | "p=l of v" |
| p="l of v" | p="l of v" |
| p="l of v"'a s m' | p="l of v"a" s m" |
| p="l of v"' s m' | p="l of v s m" |

The last row shows an example where it will not be possible to retain the original double quotes.

[mpawelski]

I know I'm going to catch a lot of flak here, and I really appreciate the depth of the discussion happening, but..._ducks_...does anyone have an example of any of this actually mattering in a real-world scenario?

I already mentioned it in this thread a year ago (it's hidden now...) that I used to get a lot of WFT moments when using ripgrep in Powershell. I couldn't understand why I couldn't search quoted strings. It ignored my quotes:

rg '"quoted"'

and in git bash it didn't.

Now I get less this WTF moments because sadly I found this long github issue and found that passing " to Powershel is totally broken. Recent "git.exe" example is also great.

To be honest, now I don't even dare to use Powershell to call native command when I know I might be passing " in string as parameter. I know I might get wrong result or error.

Really, @mklement0 summed it up great (this should be engraved in stone somewhere)

As stated many times before, a core mandate of a shell is to call external executables with arguments.
PowerShell currently fails to fulfill this mandate, given that arguments with embedded double quotes and empty-string arguments aren't passed correctly.
As stated before, this may have been less of a problem in the Windows-only days, where the lack of capable external CLIs rarely surfaced this problem, but these days are gone, and if PowerShell wants to establish itself as a credible cross-platform shell, it must address this problem.

And about breaking changes.
Recently coworker wrote to me that my script didn't worked on his machine. I was running it only on Powershel Core and he was running it on Windows Powershell. Turns out Out-File -Encoding utf8 encoded file with "BOM" on Windows Powershell and without BOM on Powershel Core. This is somehow unreleted example but show that there are already subtle breaking changes in Powershel and this is good because we are eliminating quirks and intuitive behavior from a language that is famous for it. It would be great if Powershel team was a bit more lenient when it comes to breaking changes now that we have cross platform Powershell that is shipping outside of Windows and that we know that Windows Powershell is in "maintenence" mode and will be usable forever if you really want it to run some old script that broke in newer version of Powershell.

[vexx32]

RE: that last point of breaking changes -- I fully agree. There are _many_ breaking changes we've tolerated for various reasons. However, more and more often it seems to be the case that some breaking changes are simply frowned upon for reasons of preference and not given proper gravity of consideration for their actual value.

There are some changes like this which would _massively_ improve the overall shell experience for anyone who needs to reach outside of PowerShell to get things done, which happens all the time. It's been agreed time and time again that the current behaviour is untenable and already largely broken for anything but the simplest usages. And yet, we're still facing this reticence to breaking changes even while there are scores of already accepted breaking changes, some of which have similarly large impact.

For those asking for examples -- take a minute to visit Stack Overflow for once. I'm sure @mklement0 has a litany of examples where community help is required to help explain a breaking change in newer versions. It happens _all the time_. We have no excuse to not make helpful breaking changes.

[iSazonov]

Whenever MSFT team repeats the same thing over and over again, we can be sure they know more than they can publicly say. We should respect their inner discipline and not pressure them. _Maybe we can find a compromise._ I hope I have time today to describe an alternative path with lazy migration.

[vexx32]

I do recognise that, and it's why I rarely make a point of questioning it.

However, this is an open source project; if there's no possible visibility into those decisions, folks _will_ inevitably end up frustrated. No blame to cast on either side of that coin, that's just the reality of the situation here, IMO. So yeah, having a migration path may ease that pain somewhat, but we need clear policies defined on how that has to work that will make things work for as many folks as possible. Compromise is difficult to reach when lacking information, though.

I look forward to seeing what you have up your sleeve. 😉

[joeyaiello]

@mklement0 you're absolutely right, and so much so that I can only respond to your meta-point right now. Unfortunately, in the cases where we aren't able to reach the level of depth required to answer a question like this, the safer approach is often to defer or reject the breaking change until we have more time to

I want to make another meta-point about breaking changes, though: our telemetry implies that most PowerShell 7 users are not managing their own versions. They're running automated scripts in a managed environment that's comfortable e.g. upgrading their users from 6.2 to 7.0 (see the 2-day jump in 6.2 users becoming 7.0 users starting on 8/3; this isn't our only data point here, but it's a convenient one right now that makes the point). For these users, a breaking change that turns a perfectly working script into a non-working script is unacceptable.

I also owe the community a blog on how I think about the impact of breaking changes: namely trading off the prevalence of existing usage and severity of the break against the ease of identifying and correcting the break. This one is extremely prevalent in existing scripts, confusing to identify and fix, and the breaking behavior is from total success to total failure, hence my extreme reticence to do anything here.

I think it's fair to say we're not going to do anything here in 7.1, but I'm definitely open to making this an investigative priority for 7.2 (i.e. we spend more than just our Committee time discussing this.)

How about appointing subject-matter-specific sub-committees that the committee consults with that do have the required understanding of the issues involved?

We're working on this. I know I've said that before, but we're extremely close (as in, I'm crafting the blog and you're probably going to see some new labels show up soon that we'll be playing with).

I appreciate everyone's patience and I recognize that it's annoying to get a pithy reply out of the Committee every couple weeks when folks are pouring an immense amount of thought and consideration into the discussion. I know it looks like that means we're not thinking deeply about things, but I think we're just not expressing the depth of our discussions in as much detail as folks do here. In my own backlog, I've got a whole set of blog topics like the breaking change one around how I think about making decisions from within the Committee, but I've just never gotten the chance to sit down and pump them out. But I can see here that maybe folks would find a lot of value in that.

Hope I didn't get the rails too far off in this discussion. I don't want this issue to totally become a meta-issue about the project's management, but I did want to address some of the understandable frustration I see here. I implore anyone that wants to talk about this in more detail with us to join the Community Call next week (add your questions and thoughts here and I'll be sure to address them in the call).

[mklement0]

Just a quick note on the meta-point: I appreciate the thoughtful response, @joeyaiello.

As for the severity of the breaking change: The following statements seem to be at odds:

does anyone have an example of any of this actually mattering in a real-world scenario

vs.

This one is extremely prevalent in existing scripts, confusing to identify and fix

If it is already prevalent, the awkwardness and obscurity of the necessary workarounds are all the more reason to finally fix this, especially given that we should expect the number of cases to increase.

I do realize that all existing workarounds will break.

If avoiding that is paramount, this previously suggested approach is the way to go:

provide a _new operator or a function_ such as the ie function from the Native module that fixes the problem, and widely publicize it as the only reliable way to invoke external executables.

A _function_ such as ie would allow people to opt-into the correct behavior with _minimal fuss_, _as a stopgap_, without burdening the _language_ with a new syntactical element (an operator), whose sole raison d'être would be to work around a legacy bug deemed too breaking to fix:

• The stopgap would provide _officially sanctioned_ access to the correct behavior (no reliance on _experimental_ features).
• For as long as the stopgap is necessary, it would need to be widely publicized and properly documented.

If/when the default behavior gets fixed:

• the function can be modified to just defer to it, so as not to break code that uses it.
• new code can be written without needing the function anymore.

[iSazonov]

A function such as ie would allow people to opt-into the correct behavior with minimal fuss, as a stopgap

We can simplify adoption by mean of #13428. We can inject this with @mklement0's investigations in Engine transparently.

[mklement0]

@Dabombber

I think that accommodating exceptions by default is just going to lead to a similar situation as the current one, where people need to revert PowerShell's "helpfulness". If there are exceptions, it should be obvious they're being applied.

The accommodations I'm proposing make the vast majority of calls "just work" - they are useful abstractions from the Windows command line anarchy that we should do our best to shield users from.

The justifiable assumption is that this shielding is a one-time effort for _legacy_ executables, and that newly created ones will adhere to the Microsoft C/C++ conventions.

It's impossible to do that in _all_ cases, however; for those that can't be accommodated automatically, there's --% .

Personally, I don't want to have to think about wether a given utility foo happens to be implemented as foo.bat or foo.cmd, or whether it requires foo="bar none" arguments specifically, without also accepting "foo=bar none", which for convention-compliant executables are equivalent.

And I certainly don't want a separate syntax form for various exceptions, such as &[bat]
Instead, --% is the (Windows-only) catch-all tool for formulating the command line exactly as you want it passed - whatever the target program's specific, unconventional requirements are.

Specifically, the proposed accommodations are the following:

Note:

• As stated, they are required _on Windows_ only; on Unix, delegating to ProcessStartInfo.ArgumentList is sufficient to solve all problems.

• At least at a high level, these accommodations are easy to conceptualize and document.

• Note that they will be applied _after_ PowerShell's usual parsing, in the (Windows-only) translation-to-the-process-command-line step. That is, PowerShell's own parameter parsing won't be involved - and _shouldn't_ be, @yecril71pl.

• Any then truly exotic cases not covered by these accommodations would have to be handled by users themselves, with
  --% - or, with the Native module installed, with ins / Invoke-NativeShell, which makes it easier to embed PowerShell variable and expression values in the call.

  • The dbea (Debug-ExecutableArguments) command from the Native module can help with diagnosing and understanding what process command line is ultimately used - see the example section below.

List of accommodations:

• For batch files (as noted, the importance of accommodating this case automatically is the prevalence of high-profile CLIs that use batch files _as their entry point_, such as az.cmd for Azure).

  • Embedded double quotes, if any, are escaped as "" (rather than \") in all arguments.
  • Any argument that contains no spaces but contains either double quotes or cmd.exe metacharacters such as & is enclosed in double quotes (whereas PowerShell by default only encloses arguments with spaces in double quotes); e.g., a verbatim argument seen by PowerShell as a&b is placed as "a&b" on the command line passed to a batch file.

• For high-profile CLIs such as msiexec.exe / msdeploy.exe and cmdkey.exe (without hard-coding exceptions for them):

  • Any invocation that contains at least one argument of the following forms triggers the behavior described below; <word> can be composed of letters, digits, and underscores:

    • <word>=<value with spaces>
    • /<word>:<value with spaces>
    • -<word>:<value with spaces>

  • If such an argument is present:

    • Embedded double quotes, if any, are escaped as "" (rather than \") in all arguments. - see https://github.com/PowerShell/PowerShell/pull/13482#issuecomment-677813167 for why we should _not_ do this; this means that in the rare event that <value with spaces> has _embedded_ " chars., --% must be used; e.g.,
      msiexec ... --% PROP="Nat ""King"" Cole"
    • Only the <value with spaces> part is enclosed in double quotes, not the argument as a whole (the latter being what PowerShell - justifiably - does by default); e.g., a verbatim argument seen by PowerShell as foo=bar none is placed as foo="bar none" on the process' command line (rather than as "foo=bar none").
  • Note:
    
    
    
    • If the target executable happens _not_ to be an msiexec-style CLI, no harm is done, because convention-adhering CLIs sensibly consider <word>="<value with spaces>" and "<word>=<value with spaces>" _equivalent_, both representing verbatim <word>=<value with spaces>.
    
    
    • Similarly, the vast majority of executables accept "" interchangeably with \" for escaping embedded " chars., with the notably exception of PowerShell's own CLI, Ruby, and Perl (_not_ performing the accommodation is worthwhile at least if the PowerShell CLI is being called, but I think hard-coding Ruby and Perl would also make sense). https://github.com/PowerShell/PowerShell/pull/13482#issuecomment-677813167 shows that all applications that use the CommandLineToArgvW WinAPI function do _not_ support ""-escaping.
    
    

All other cases on Windows can also be handled with ProcessStartInfo.ArgumentList, which implicitly applies the Microsoft C/C++ convention (which notably means \" for "-escaping).

---

The ie function from the current version (1.0.7) of the Native module implements these accommodations (in addition to fixing the broken argument parsing), for PowerShell versions 3 and up (Install-Module Native).

I invite you and everyone here to put it through its paces to test the claim that it "just works" for the vast majority of external-executable calls

Currently unavoidable limitations:

• Note: These technical limitations come from ie being implemented as a _function_ (a proper fix in the engine itself would _not_ have these problems):

  • While $LASTEXITCODE is properly set to the process' exit code, $? ends up always $true - user code currently cannot set $? explicitly, though adding this ability has been green-lit - see https://github.com/PowerShell/PowerShell/issues/10917#issuecomment-550550490. Unfortunately, this means that you cannot currently use ie meaningfully with && and ||, the &&, the pipeline-chain operators.
    However, if _aborting_ a script on detecting a nonzero exit code is desired, the iee wrapper function can be used.

  • -- as an argument is invariably "eaten" by the PowerShell parameter binder; simply pass it _twice_ in order to pass -- through to the target executable (foo -- -- instead of foo --).

  • An unquoted token with , must be quoted lest it be interpreted as an array and passed as multiple arguments; e.g., pass 'a,b' rather than a,b; similarly, pass -foo:bar (something that looks like a named PowerShell argument) as '-foo:bar' (this shouldn't be necessary, but is owed to a bug: #6360); similarly '-foo.barmust be passed as'-foo.bar'` (another bug, which affects direct calls to external executables too: #6291)

• I expect the function to work robustly in PowerShell _Core_. Due to changes in _Windows PowerShell_ over time, there may be edge cases that aren't handled properly, though I'm only aware of two:

  • The partial-quoting accommodation for msiexec-style CLIs cannot be applied in version 3 and 4, because these versions wrap the entire argument in an additional set of double quotes; it does work in v5.1, however.

  • ""-escaping is used by default, to work around problems, but in cases where \" is necessary (PowerShell CLI, Perl, Ruby), a token such as 3" of snow is mistakenly passed as _3_ arguments, because all Windows PowerShell versions neglect to enclose such an argument in double quotes; this seems to happen for arguments with non-initial " characters that aren't preceded by a space character.

---

Examples, with output from PowerShell Core 7.1.0-preview.5 on Windows 10:

Note: The dbea (Debug-ExecutableArguments) function is used to illustrate how the arguments would be received by external executables / batch files.

Current, broken argument passing:

• Calling a convention-compliant application (a .NET console application used by dbea behind the scenes by default):

# Note the missing 2nd argument and the effective loss of embedded double quotes,
# due to the embedded " chars. not having been escaped.
PS> dbea -- 'a&b' '' '{ "foo": "bar" }'

2 argument(s) received (enclosed in <...> for delineation):

  <a&b>
  <{ foo: bar }>

Command line (helper executable omitted):

  a&b  "{ "foo": "bar" }"

• Calling a batch file:

Note the use of -UseBatchFile to make dbea pass the arguments to a helper _batch file_ instead.

# Note that only *part of the first argument* is passed and that the `&` is interpreted as cmd.exe's
# statement separator, causing `b` to be run as a command (which fails).
PS> dbea -UseBatchFile -- 'a&b' '' '{ "foo": "bar" }'

1 argument(s) received (enclosed in <...> for delineation):

  <a>

'b' is not recognized as an internal or external command,
operable program or batch file.

• Calling a msiexec-style CLI, cmdkey.exe:

# The call fails, because `cmdkey.exe` requires the password argument to 
# to be quoted exactly as `/password:"bar none"` (double-quoting of the option value only), 
# whereas PowerShell - justifiably - passes `"/password:bar none"` (double-quoting of the whole argument).
PS> cmdkey.exe /generic:foo /user:foo /password:'bar none'

The command line parameters are incorrect.

Fixing the problem with ie:

Note the use of -ie in the dbea calls, which causes the use of ie for the invocations.

• Calling a convention-compliant application (a .NET console application used by dbea behind the scenes by default):

# OK
# Note that the empty 2nd argument is correctly passed, and that \" is used for embedded "-escaping.
PS> dbea -ie -- 'a&b' '' '{ "foo": "bar" }'

3 argument(s) received (enclosed in <...> for delineation):

  <a&b>
  <>
  <{ "foo": "bar" }>

Command line (helper executable omitted):

  a&b "" "{ \"foo\": \"bar\" }"

• Calling a batch file:

# OK
# - `a&b` was enclosed in "...", due to the presence of metacharacter `&`
# - "" is used for escaping of embedded " chars.
# Note that `echo %1`, for instance, prints the argument exactly as passed on the command line, including quoting.
# `echo %~1` strips the surrounding double quotes, but embedded escaped ones still print as "".
# However, if you pass these arguments (`%*`) through to convention-compliant CLIs, they are parsed correctly.
PS> dbea -ie -UseBatchFile -- 'a&b' '' '{ "foo": "bar" }'

3 argument(s) received (enclosed in <...> for delineation):

  <"a&b">
  <"">
  <"{ ""foo"": ""bar"" }">

• Calling a msiexec-style CLI, cmdkey.exe:

# The call now succeeds, because `ie` ensure the value-only double-quoting that cmdkey.exe requires.
# (Use `cmdkey /del:foo` to remove the credentials again.)
PS> ie cmdkey.exe /generic:foo /user:foo /password:'bar none'

CMDKEY: Credential added successfully.

To show that the value-only double-quoting was applied in the actual command line, via dbea:

PS> dbea -ie -- cmdkey.exe /generic:foo /user:foo /password:'bar none'

  <cmdkey.exe>
  </generic:foo>
  </user:foo>
  </password:bar none>

Command line (helper executable omitted):

  cmdkey.exe /generic:foo /user:foo /password:"bar none"

[yecril71pl]

The following code causes a data loss:

{ PARAM($A) $A } | OUT-FILE A.PS1
PWSH A.PS1 -A:(1,2)

1

[SteveL-MSFT]

@JamesWTruher has a proposed fix and is validating if it addresses the concerns raised in this issue

[TSlivede]

Is there a pull request of that proposed fix? It would be nice if we could comment on the PR. Because fixing this was IMO never the complicated part. The complicated part was how to handle backwardscompatibility. And it would be nice, if we could see how the proposed fix handles that....

[mklement0]

That's good to hear, @SteveL-MSFT - a fix for _all_ problems discussed here? For v7.1, after all? And I second @TSlivede's request.

@yecril71pl, that's a good find, although this one truly relates to PowerShell's own parsing (which does have _some_ special-casing for external executables), not to how the native command line is constructed _after_ parsing (which is where the problems previously discussed come from).

A more succinct repro of the problem, on Unix:

PS> printf '<%s>\n' -a:(1,2,3)
<-a:1>
<2>
<3>

That is, only the _first_ array element was directly attached to -a:, the others were passed as separate arguments.

There are related problems with arguments that _look like_ PowerShell parameters, but aren't:

There is a related issue that only affects calls to _PowerShell_ commands that use $args / @args: #6360

• & { $args.Count; $args } -foo:bar yields 2, '-foo:', 'bar'

There's also #6291, which affects both PowerShell commands and external executables (note the .).

• & { $args.Count; $args } -foo.bar yields 2, '-foo', '.bar'

One thing to note is that (...) as part of a bareword normally results in (...)'s output as a whole becoming a _separate_ argument, so the fact that the first element _is_ attached in the printf command above is evidence of special-casing for external-executable calls; e.g.,
& { $args.Count; $args.ForEach({ "$_" }) } foo('bar', 'baz') yields 2, 'foo', 'bar baz', the second argument being the stringification of array 'bar', 'baz'.

[yecril71pl]

When PowerShell has to pass -A:(1,2) for an external executable, it figures out that -A: is a string and (1,2) is an array which must be marshalled as ‘1 2’. PowerShell tries to preserve the original syntax of the invocation, so, putting it all together, we get ‘-A:1 2’, whereas the correct result would be ‘-A:"1 2"’. It looks like a trivial omission in the marshalling code to me.

[TSlivede]

I wouldn't say, that @yecril71pl's specific problem is related to parsing (although I agree, that it doesn't have anything to do with the problem "converting array to commandline", which is discussed in this issue).

When PowerShell has to pass -A:(1,2) for an external executable, it figures out that -A: is a string and (1,2) is an array

Almost: -A: is a named parameter and the array is the value of that parameter (To test that: remove the - in front and you see, that it is quoted differently). But the problem isn't, that the array is incorrectly converted to a string - the problem is, that for native executables arguments are (almost) always splatted, even when using $ and not @ and even if the array originates from a expression such as (1,2).

Test for example printf '<%s>\n' -a:('a b',2): As the string a b contains a space it is correctly quoted, but as 2 is in the next array element and the array is splatted, 2 is not part of the first argument.

The magic happens in NativeCommandParameterBinder.cs

At line 170 powershell tries to get an enumerator for the current argument value.

IEnumerator list = LanguagePrimitives.GetEnumerator(obj);

If list is not null, powershell adds each element of the list (possibly quoted if containing spaces) to the lpCommandLine.

The elements are separated with spaces (line 449) by default. The only exception is if the array was a literal
(as in printf '<%s>\n' -a:1,2).
Then powershell tries to use the same separator in the lpCommandLine, that was used in the script line.

[SteveL-MSFT]

I expect a PR when ready. If it makes it into 7.1, we'll take it, otherwise it'll be 7.2. Backwards compatibility is something he is addressing. Perhaps what would help is some help writing Pester tests (using testexe -echoargs which can be built using publish-pstesttools from build.psm1).

[TSlivede]

I expect a PR when ready

That is exactly, what I wanted to avoid - please show the code that is not ready (mark PR as work in progress).

Or at least comment, what he wants to do.

It would be nice if we could discuss how he wants to handle backwards compatibility.

[mklement0]

@TSlivede, note that since the PowerShell _CLI_ is called - an external executable - -A:(1,2) is parsed _before_ knowing that this token will eventually bind to _named_ -A parameter - that such a parameter _eventually_ comes into play is incidental to the problem.

@yecril71pl:

It figures out that -A: is a string

No, it is special-cased during parsing, because it happens to _look like_ a PowerShell parameter.

This special-casing happens for calls to PowerShell commands that use $args as well (as opposed to having actual declared parameters that are bound), but it happens _differently_ for external executables (what is normally a separate argument stays attached, but in case of a collection only its _first_ element).

You can actually opt out of this special-casing if you pass -- beforehand, but, of course, that will also pass --, which is only removed for calls to _PowerShell_ commands:

PS> printf '<%s>\n' -- -a:(1,2,3)
<-->   # !! not removed
<-a:>
<1>    # array elements are *all* now passed as indiv. arguments, because (...) output is separate (splatted) argument
<2>
<3>

If the argument _doesn't_ look like a PowerShell parameter, the usual behavior (output from (...) becomes a separate argument) kicks in even for external executables (with the usual behavior of an _array_ being splatted, i.e. being turned into individual arguments in the external-executable case).

# Note: No "-" before "a:" -> output from `(...)` becomes separate argument(s)
PS> printf '<%s>\n' a:(1,2,3)
<a:>
<1>
<2>
<3>

Applying this behavior _consistently_ would make sense - a (...) expression as part of a bareword should _always_ become a separate argument - see #13488.

In order to pass a single argument '-A:1 2 3', with the array _stringified_, use a(n implicit) _expandable string_, in which case you need $(...) rather than (...) _and_ - surprisingly - currently also "...":

PS> printf '<%s>\n' "-a:$(1,2,3)"  # quotes shouldn't be needed; `-a:"$(1,2,3)"` would work too.
<a:1 2 3> # SINGLE argument with stringified array.

You _shouldn't_ also need "..." in this case - it is again necessary because of the anomalies relating to tokens that _look_ like parameters (which in general apply to _both_ PowerShell and external-executable calls - see #13489); if they don't, you needn't quote:

# Anomaly due to looking like a parameter: $(...) output becomes separate argument
PS> Write-Output -- -a:$(1,2,3)
-a:
1
2
3

# Otherwise (note the absence of "-"): no quoting needed; treated implicitly like 
# "a:$(1,2,3)"
PS> Write-Output -- a:$(1,2,3)
a:1 2 3  # SINGLE argument with stringified array.

The world of compound tokens in argument mode is a complex one, with several inconsistencies - see #6467.

[mklement0]

@SteveL-MSFT

In its current form, testexe -echoArgs only prints the individual arguments that the .NET Core executable parsed from the raw command line (on Windows), not the raw command line itself.

It therefore cannot test accommodations with selective quoting for batch files and msiexec-style CLIs - assuming such accommodations will be implemented, which I strongly recommend; for instance, you won't be able to verify that PROP='foo bar' was passed as PROP="foo bar", with double-quoting just around the value part.

However, in order to print the raw command line, testexe mustn't be a .NET _Core_ executable, because .NET Core _recreates a hypothetical command line_ that always uses \"-escaping for embedded " chars., even if "" was used, and generally doesn't faithfully reflect which arguments were double-quoted and which weren't - for background, see https://github.com/dotnet/runtime/issues/11305#issuecomment-674554010.

Only a .NET _Framework_-compiled executable shows the true command line in Environment.CommandLine, so testexe would have to be compiled that way (and altered to (optionally) print the raw command line).

To test the accommodations for batch files, a separate test _batch_ file is needed, to verify that 'a&b' is passed as "a&b" and 'a"b' as "a""b", for instance.

[SteveL-MSFT]

@mklement0 compiling for .NET Framework isn't going to be possible and would probably need to run under .NET Framework to get the right behavior. We deliberately removed all the native code compilation out of PS repo, and I don't think we want to add it back... One option is to have pre-built native testexe (which would have to be compiled for Windows, macOS, and various Linux distros (like Alpine separately). Writing testexe is easy, doing all that work to publish it will take time...

Alternatively, can we just rely on a simple bash script for Linux/macOS to emit the args?

#!/bin/bash
for i; do
   echo $i
done

And on Windows something similar with a batch file.

[cspotcode]

How about using node with a .js script?

console.log(process.execArgv.join('\n') or whatever string handling you
wanna do to make the output look nice?

[mklement0]

@cspotcode, to get the raw command line we need a WinAPI call.

@SteveL-MSFT:

On Windows, you can delegate the compilation to _Windows PowerShell_ via its CLI, which is what I do in dbea; here's a simple example that produces a .NET _Framework_ executable that echoes the raw command line (only), ./rawcmdline.exe:

powershell.exe -noprofile -args ./rawcmdline.exe -c {

  param([string] $exePath)

  Add-Type -ErrorAction Stop -OutputType ConsoleApplication -OutputAssembly $exePath -TypeDefinition @'
using System;
static class ConsoleApp {
  static void Main(string[] args) {
    Console.WriteLine(Environment.CommandLine);
  }
}
'@

}

Sample call:

PS> ./rawcmdline.exe --% "a&b" PROP="foo bar"
"C:\Users\jdoe\rawcmdline.exe"  "a&b" PROP="foo bar"

As for a _batch file_ that echoes its arguments, dbea also creates one on demand.

On Unix, a simple shell script, as shown in your comment, is indeed sufficient, and you can even use an ad hoc script that you pass to /bin/sh as an _argument_.

[SteveL-MSFT]

@PowerShell/powershell-committee discussed this today, we are asking @JamesWTruher to update his PR to also include as part of his experimental feature to skip the step in the native command processor which reconstructs the array of args back to a string and just pass that to the new array args in ProcessStartInfo (there is a bit of code to make sure parameter names and values are matched appropriately). Also, we accept that we may need an allowlist to special case known commands that still fail with the proposed change and that is something that can be added later.

[mklement0]

For those who may not have noticed: the PR has been published (as a WIP) and is already being discussed: https://github.com/PowerShell/PowerShell/pull/13482

P.S., @SteveL-MSFT, regarding getting the raw command line on Windows: of course, an alternative to delegating the compilation to Windows PowerShell / .NET _Framework_ is to enhance the existing .NET _Core_ console application to make a (platform-conditional) P/Invoke call to the GetCommandLine() WinAPI function, as demonstrated below.

using System;
using System.Runtime.InteropServices;

namespace demo
{
  static class ConsoleApp
  {
    [DllImport("kernel32.dll")]
    private static extern System.IntPtr GetCommandLineW();

    static void Main(string[] args)
    {
      Console.WriteLine("\n{0} argument(s) received (enclosed in <...> for delineation):\n", args.Length);
      for (int i = 0; i < args.Length; ++i)
      {
        Console.WriteLine("  <{0}>", args[i]);
      }

      // Windows only: print the raw command line.
      if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
      {
        Console.WriteLine("\nCommand line:\n\n  {0}\n", Marshal.PtrToStringUni(GetCommandLineW()));
      }

    }

  }

}

[mklement0]

@SteveL-MSFT

Backwards compatibility is something he is addressing.

I think it is implied by your later clarification that ProcessStartInfo.ArgumentList (a _collection_ of _verbatim_ arguments to used as-is on Unix and translated to a MS C/C++-convention-compliant command line on Windows by .NET Core itself) should be used, but let me state it explicitly:

• Fixing this problem properly, once and for all, _precludes any concessions to backward compatibility_.

• @JamesWTruher's PR is on the right track as of this writing, the only problem seems to be that empty arguments are still not passed through.

  • Once that is addressed, the fix is complete on _Unix_ - but lacks the important accommodations for CLIs on _Windows_ (see below).

we may need an allowlist to special case known commands that still fail with the proposed change and that is something that can be added later.

I urge you not to put this off until later.

Instead of an _allowlist_ (special-casing for specific executables), simple general rules can govern the accommodations, refined from the ones above after some more discussion with @TSlivede.

These accommodations, which are _needed on Windows only_ :

• will make the vast majority of calls on Windows work properly.

• are easy to conceptualize and document, in the proposed about_Native_Calls help topic.

• are straightforward to implement - here's a proof-of-concept written in PowerShell, along with tests.

Specifically, these are:

• Just like on Unix, ProcessStartInfo.ArgumentList is used by default - except if _one or both_ of the following conditions are met, _in which case the process command line must be constructed manually_ (and assigned to ProcessStartInfo.Arguments as currently):

  • A batch file (.cmd, .bat) or cmd.exe directly is called:
  • In that event, _embedded_ " are escaped as "" (rather than as \"), and space-less arguments that contain any of the following cmd.exe metacharacters are also double-quoted (normally, only arguments _with spaces_ get double-quoted): " & | < > ^ , ; - this will make calls to batch files work robustly, which is important, because many high-profile CLIs use batch files as _entry points_.

  • Independently (and possibly in addition), if at least one argument that matches the regex
    '^([/-]\w+[=:]|\w+=)(.*? .*)$' is present, all such arguments must apply _partial_ double-quoting around the _value part only_ (what follows the : or =)

    • E.g., msiexec.exe / msdeploy.exe and cmdkey.exe-style arguments seen by PowerShell verbatim as
      FOO=bar baz and /foo:bar baz / -foo:bar baz would be placed on the process command line as
      foo="bar baz" or /foo:"bar baz" / -foo:"bar baz" making _any_ CLIs that require this style of quoting happy.

  • Verbatim \ characters in the arguments must be handled in accordance with the MS C/C++ conventions.

What _isn't_ covered by these accommodations:

• msiexec.exe (and presumably msdeploye.exe too) supports _only_ ""-escaping of _embedded_ " chars., which the above rules would _not_ cover - except if you happen to call via a batch file or cmd /c.

  • This should be rare enough to begin with (e.g.,
    msiexec.exe /i example.msi PROPERTY="Nat ""King"" Cole"), but it is probably made even rarer due to the fact that misexec invocations are usually _made synchronous_ to await the end of an installation, in which case you can avoid the problem in one of two ways:
  • cmd /c start /wait msiexec /i example.msi PROPERTY='Nat "King' Cole' - rely on calls to cmd.exe (then) triggering ""-escaping
  • Start-Process -Wait msiexec '/i example.msi PROPERTY="Nat ""King"" Cole"' - rely on the -ArgumentList (-Args) parameter passing a single string argument through verbatim as the process command line (even though that's not how this parameter is supposed to work - see #5576).

• Any other non-conventional CLIs for which the above accommodations aren't sufficient - I'm personally not aware of any.

At the end of the day, there is always a workaround: call via cmd /c, or, for non-console applications, via Start-Process, or use --%; if and when we provide an ins (Invoke-NativeShell) cmdlet, it is another option; a dbea (Debug-ExecutableArguments cmdlet with echoArgs.exe-like abilities, but on demand also for batch files, would also help to _diagnose_ problems.

---

As for the path to a breaking change vs. opt-in:

• Does implementing this as an experimental feature mean that if enough interest is shown that it will become the _default_ behavior and will therefore amount to a (nontrivial) breaking change?

• Can you please make sure that this experimental feature is widely publicized, given its importance?

  • A general concern I have about experimental features is that their use can often be _unwitting_ in preview versions, given that _all_ experimental features are turned on by default. We definitely want people to know about and exercise this feature deliberately.