Powershell: Start-Process doesn't pass arguments with embedded whitespace and double quotes correctly

[mklement0]

^{Updated later to include the problem with embedded double quotes.}

Steps to reproduce

Embedded whitespace:

'"Hi!"' > './t 1.ps1'; Start-Process -Wait -NoNewWindow pwsh -ArgumentList '-noprofile', '-file', './t 1.ps1'

Embedded double quotes:

Start-Process -Wait -NoNewWindow pwsh -ArgumentList '-noprofile', '-command', '"Hi!"'

Expected behavior

In both cases:

Hi!

That is, script file ./t 1.ps1 should execute, and double-quoted string literal "Hi!" should print.

Actual behavior

Embedded whitespace:

Invocation fails, because the ./t 1.ps1 is passed as _two_ arguments:

The argument './t' is not recognized as the name of a script file. Check the spelling of the name, or if a path was included, verify that the path is
correct and try again.

The only way to make this currently work is to _embed_ (potentially escaped) _double quotes_: '"./t 1.ps1"' or "`"./t 1.ps1`""; e.g.:

Start-Process -Wait -NoNewWindow pwsh -ArgumentList '-noprofile', '-file', "`"./t 1.ps1`""

_Update_: Overall, the best workaround is to pass a _single_ string containing _all_ arguments to -ArgumentList and use embedded quoting and escaping as necessary:

Start-Process -Wait -NoNewWindow pwsh -ArgumentList '-noprofile -file "./t 1.ps1"'

Embedded double quotes:

The embedded double quotes are unexpectedly _removed_.

The only way to make this currently work is to \-escape the embedded double quotes:

Start-Process -Wait -NoNewWindow pwsh -ArgumentList '-noprofile', '-command', '\"Hi!\"'

_Update_: Again, the best workaround is to use a _single_ string:

Start-Process -Wait -NoNewWindow pwsh -ArgumentList '-noprofile -command \"Hi!\"'

Environment data

PowerShell Core v6.0.0-rc (v6.0.0-rc) on Microsoft Windows 7 Enterprise  (64-bit; v6.1.7601)
Windows PowerShell v5.1.14409.1012 on Microsoft Windows 7 Enterprise  (64-bit; v6.1.7601)

[ZSkycat]

I think it just executes {filename} {ArgumentList.join(' ')}

try this code

Start-Process -Wait -NoNewWindow pwsh.exe -ArgumentList '-noprofile', '-file', '"./t 1.ps1"'
Start-Process -Wait -NoNewWindow pwsh.exe -ArgumentList '-noprofile', '-file', '"', './t 1.ps1', '"'

[bergmeister]

The code where it deconstructs -Argumentlist to the Argument string for ProcessStartInfo.Arguments here and it simply joins the arguments by separating it with a space character. Therefore it sees ./t and 1.ps1 as two separate arguments. As pointed out by @ZSkycat one has to correctly quote the file.

[iSazonov]

I see the feature is documented:

Arguments are parsed and interpreted by the target application, so must align with the expectations of that application. For.NET applications as demonstrated in the Examples below, spaces are interpreted as a separator between multiple arguments. A single argument that includes spaces must be surrounded by quotation marks, but those quotation marks are not carried through to the target application. In include quotation marks in the final parsed argument, triple-escape each mark.

I don't understand how we can fix this for all platforms and for an indefinite number of applications. If there is no standard to which we should follow I would rather expect a common way of passing arguments so that they reach the goal application in exactly the user specified form.

[bergmeister]

Well, I guess we can only improve it (see my PR 5703) or maybe the better solution is to add an -Arguments parameter that is just a string to reduce complexity and leave the -ArgumentList parameter there only for legacy reasons.

[mklement0]

@bergmeister: Thanks for taking this on in your PR, but there are additional edge cases to consider, such as tokens ending in \ - see https://github.com/PowerShell/PowerShell/issues/1995#issuecomment-330675421

Ultimately, the single string passed to ProcessStartInfo.Arguments should be constructed the same way that a direct call to an external program is handled, namely based on inverse of the MSVC++ command-line parsing rules, as discussed in @TSlivede's RFC proposal.

Longer-term, once https://github.com/dotnet/corefx/issues/23592 is implemented in CoreFx, PowerShell will simply be able to pass the array _through_. The linked issue also points to a utility method in the CoreFx code where the kind of array-to-command-line transformation that would be needed here is used internally.

And, yes, this will break things, inevitably and in multiple scenarios, but I think if PowerShell wants to be taken seriously as a multi-platform shell, there's no way around that.

While we may consider adding a new -Arguments parameter for someone who wants to pass a _single-string_, _pre-escaped_ command line (directly assignable to ProcessStartInfo.Arguments - which, notably, still needs to be split back into an array _before_ creating a process on _Unix_), it is -ArgumentList that must be fixed - which breaks things.

[iSazonov]

@mklement0 As far as I understand the shortest way to get this is to implement https://github.com/dotnet/corefx/issues/23592, isn't it?

[mklement0]

@iSazonov: Yes, with said proposal implemented, PowerShell could just use the new ProcessStartInfo.ArgumentList property directly and, on Windows, let CoreFX translate that into a single command line string with appropriate quoting.

[bergmeister]

Ok. Shall I close the PR then or do we want to consider it as an intermediate improvement?

[iSazonov]

I think it is not critical and not a secure hole so we can wait and it is better to direct our efforts to other areas or contribute directly in CoreFX now.

[Artoria2e5]

On a related note, it seems to truncate at the > character too:

PS> Start-Process -Wait -NoNewWindow node -ArgumentList '-e', 'process.argv.slice(1).forEach((x) => console.log(x))'
[eval]:1
process.argv.slice(1).forEach((x)
                                ^
SyntaxError: missing ) after argument list

(Windows PS 7.0.0-preview3)

---

read the reply. ho boy... this is bad

[mklement0]

@Artoria2e5: The problem isn't >, the problem is that process.argv.slice(1).forEach((x) => console.log(x)) isn't passed through as a _single argument_, it is broken into _3_ arguments by _whitespace_.

That is, your node command is effectively executed as follows, which explains the symptom:

node -e 'process.argv.slice(1).forEach((x)' '=>' 'console.log(x))'

[felixfbecker]

This is very annoying when wrapping Linux command line tools. Could this be considered for PowerShell 7? It seems like the fix would be simple (just use ProcessStartInfo.ArgumentList). Using System.Diagnostics.Process in PowerShell directly works as expected, but is incredibly cumbersome.

[rjmholt]

It seems like the fix would be simple (just use ProcessStartInfo.ArgumentList

Having used ProcessStartInfo.Arguments recently (had to support netstandard2.0), it's a very hard API to use.

I'd second the recommendation to switch to ArgumentList if we can.

[SteveL-MSFT]

I have a PR to fix this, just need to add tests.

In the second example:

Start-Process -Wait -NoNewWindow pwsh -ArgumentList '-noprofile', '-command', '"Hi!"'

This becomes: pwsh -noprofile -command "Hi!" so it should only print out Hi! without the quotes. The quotes are needed to tell PowerShell this is a string and not a command. After this fix, you still need to have nested double quotes if you want quotes:

Start-Process -Wait -NoNewWindow pwsh -ArgumentList '-noprofile', '-command', '"""Hi!"""'

[mklement0]

Glad to see that this is getting tackled.

This becomes: pwsh -noprofile -command "Hi!" so it should only print out Hi! without the quotes.

As a _shell command_ / Windows command line, pwsh -noprofile -command "Hi!", _fails_, because the the enclosing " are removed during argument parsing, causing PowerShell to attempt to execute Hi!.

What Start-Process -Wait -NoNewWindow pwsh -ArgumentList '-noprofile', '-command', '"Hi!"' _should_ translate to is:

On Windows:

pwsh -noprofile -command "\"Hi!\""

On Unix, the array of verbatim tokens needs to be:

pwsh, -noprofile, -command, "Hi!"

The collection-based ProcessStartInfo.ArgumentList - not the single-string ProcessStartInfo.Arguments - should do all that for us.

[SteveL-MSFT]

Unfortunately, some bad news. Use of ArgumentList makes this a breaking change. Previously, you could do something like:

start-process ping -argumentlist "-n 2 localhost"

But this becomes:

ping "-n 2 localhost"

This means I can't rely on ArgumentList. Since the current code simply joins all elements of ArgumentList into a single string separated by a space, there is no way to determine what the user intended so I can't arbitrarily add quotes. Seems like the only way to solve this in a non-breaking manner is to introduce a new parameter that is strictly treated as an array of arguments. It's unfortunate that the current parameter is called ArgumentList.

On the topic of:

Start-Process -Wait -NoNewWindow pwsh -ArgumentList '-noprofile', '-command', '"Hi!"'

Again, this becomes:

pwsh -noprofile -command "Hi!"

We could double escape the quotes to get the desired behavior, but seems like something else will be broken.

[vexx32]

From a pure UX perspective, I think it would be much better to have -ArgumentList be strictly an actual list of arguments (so, it would map directly to ArgumentList in the code), and then introduce an additional parameter that mimics legacy behaviour.

Yes, it would break something, but given the difficulty of using the current behaviour correctly... I think we're rather better off breaking it here. 😕

[bergmeister]

Although I wish the parameter behaviour was better, I agree with Steve, that we cannot make a breaking change in such an important cmdlet because it is the foundation of a lot of code and use cases are quite often very generic. A classic example is the software of CI agents that needs to spawn processes, this software usually delegates some of its work from .net/java/C++ to something simpler like node/python/powershell for doing that. Because this piece of software is implicitly used by other users, it is not known how users will call into it and implicitly break peoples build or release pipelines.
The change will need to be in a non-breaking way or at least have some way to opt back into legacy behaviour.

[SteveL-MSFT]

It's way too likely people are doing:

start-process ping -argumentlist "-n",2,"localhost"
start-process ping -argumentlist "-n 2","localhost"
start-process ping -argumentlist "-n 2 localhost"

where all 3 have the same behavior today since it's a simple concatenation with whitespace in between. This is apparent to me looking at the test failures in my attempt to use ProcessStartInfo.ArgumentList that our tests do this. Maybe -LiteralArgumentList is ok.

[mklement0]

Good points, @SteveL-MSFT and @bergmeister, as much as I wish we could go with @vexx32's suggestion.

It does seem like a new parameter is the only non-breaking solution, but, to better reflect the distinction, I propose the following names

• -IndividualArguments
• alias -iArgs

Additionally, we could consider renaming -ArgumentList to -Arguments, making the former an alias of the latter, and highlighting the latter in the help topic (the -Args alias luckily already has the better name).

[felixfbecker]

It's also not just Start-Process, argument splatting is also affected. I recently had this problem where I was calling a utility on macOS like foo @bar. One would expect this to call foo with exactly the array $bar as arguments, instead it joins the array into a string, then splits the string by spaces and passes that array as arguments. This broke the tool whenever I wanted to pass a string as first parameter that contained a space.

A different parameter for Start-Process would not solve this usage, and it would be really annoying if we basically have to tell macOS and Linux users they cannot rely on calling commands directly and always need to use Start-Process.

Also related: #1995

[SteveL-MSFT]

@felixfbecker that's a different issue than this one, can you open a new issue or see if there is an existing one for that?

[vexx32]

That seems like the same problem with #1995 (or at least the same possible fix, anyway).

[mklement0]

I agree that splatting is incidental to the issue and that it comes down to #1995:

Assume a de.exe executable that prints the full command line with which it is invoked:

PS> de.exe '"hi there"' 
"c:\path\to\de.exe" "hi there"   # broken: embedded " weren't preserved

# Ditto with splatting  (or direct passing of $a)
PS> $a = , '"hi there"'; de.exe @a  
"c:\path\to\de.exe" "hi there"

Yes, passing arguments with embedded double quotes is fundamentally broken, and always has been - because of existing workarounds, it's a problem to fix it now without breaking lots of code.

@felixfbecker, please start reading at https://github.com/PowerShell/PowerShell/issues/1995#issuecomment-552223508 for the current state of the debate.

[iSazonov]

Since #1995 is still a big question, we definitely need to resolve the issue with Start-Process in the 7.1 milestone.

I believe we can add a new -Arguments parameter. Internally, it will use ProcessStartInfo.ArgumentList, but this internal implementation is invisible to users and should not confuse them.

As option, we could rename (with alias) ArgumentList to ArgumentString.

[mklement0]

While I definitely welcome a fix for this, note that fixing Start-Process is no substitute for fixing #1995, given that Start-Process serves a different purpose than direct invocation, and notably isn't integrated with PowerShell's output streams -
it is #1995 that needs urgent attention; you can work around the Start-Process bug at hand by supplying a _single_ string containing all arguments, with embedded "-quoting.

I like the idea of renaming (aliasing) to ArgumentString.

As for naming the new parameter -Arguments, I see two problems:

• We also need to think about a short parameter alias that parallels -Args - what would that be for -Arguments?

• While PowerShell in general needn't mirror the underlying .NET APIs, I find it problematic that the semantics in this case would be the _exact opposite_ of the underlying .NET API's.

-IndividualArguments / -iArgs solves / mitigates these problems. [_update_: see the tab-completion-friendlier alternative [ below](https://github.com/PowerShell/PowerShell/issues/5576#issuecomment-666566501)]

---

One more thing we could do to avoid confusion, to complement the renaming to -ArgumentString:

• Change the -ArgumentList parameter type to string, so that the syntax diagram and the documentation can suggest that only a _single string with all arguments_ should be passed.

• So as not to break backward compatibility, decorate the re-typed parameter with a transformation attribute that stringifies an array argument by joining its elements with spaces - which is effectively what happens behind the scenes at the moment; something along the following lines (in the real implementation, existing internal helper methods and error messages would need to be used):

c# public class StringArrayToScalarTransformationAttribute : ArgumentTransformationAttribute { public override object Transform(EngineIntrinsics engineIntrinsics, object inputData) { return inputData switch { Array a => string.Join(' ', (object[]) inputData), object o when o is IEnumerable && !(o is IDictionary) => throw new ArgumentException("Input type not supported."), _ => inputData }; } }

[iSazonov]

I think it is better to avoid unneeded complicity - best is the enemy of good.

We also need to think about a short parameter alias that parallels -Args - what would that be for -Arguments?

With IntelliSense and tab-completion it is minor.

While PowerShell in general needn't mirror the underlying .NET APIs, I find it problematic that the semantics in this case would be the exact opposite of the underlying .NET API's

Yes, PowerShell users do not see and don't think about underlying .NET API's. For developers we add docs and comments too.

If we blog post and enhance PSSA to recommend the new parameter I believe user adaptation will be easy.

[iSazonov]

/cc @SteveL-MSFT @daxian-dbw for PowerShell Committee review.

[mklement0]

@iSazonov

better to avoid unneeded complicity

What unneeded complexity? If you're referring to re-typing -ArgumentList to string, there is minor implementation complexity and no added complexity to end users - on the contrary, it makes things clearer for them.

With IntelliSense and tab-completion it is minor.

We want go give power users an official short alias, irrespective of tab-completion, just like we do with -Args for -ArgumentList.

add docs and comments too.

Docs, blog posts, and comments are only part of the puzzle: the parameter names themselves shouldn't be counterintuitive:

We can't fix the -ArgumentList / -Args name anymore (except to de-emphasize -ArgumentList in favor of
-ArgumentString), but we can be more descriptive in the _new_ parameter's name:

-Arguments is ambiguous, -IndividualArguments is not [_update_: see the tab-completion-friendlier alternative [ below](https://github.com/PowerShell/PowerShell/issues/5576#issuecomment-666566501)]

[iSazonov]

With tab completion Arguments is more discoverable and more easy for adoption. Everything else does not make sense until the user reads the parameter description.

[mklement0]

If your concern is that -IndividualArguments doesn't start with the word "Argument" - a good point - here's another option:

-ArgumentArray / -Arga

Note that the s in the existing -Args alias can then be interpreted as referring to the "String" part of the renamed
-ArgumentString (we could capitalize the final letter for clarity without breaking anyone).

Here's a quick proof of concept:

Add-Type @'
    using System;
    using System.Management.Automation;

    [Cmdlet("Start", "Process", DefaultParameterSetName = "ArgString")]
    public class StartProcessCommand : PSCmdlet {

        [Parameter(Position = 0)]
        public string FilePath { get; set; }

        [Parameter(ParameterSetName = "ArgArray", Position = 1)]
        [Alias("Arga")]
        public string[] ArgumentArray { get; set; }

        [Parameter(ParameterSetName = "ArgString", Position = 1)]
        [Alias("Args", "ArgumentList")]
        // string-array-to-string-scalar transformation attribute would go here.
        public string ArgumentString { get; set; }

        protected override void ProcessRecord() {
          WriteObject(ParameterSetName);
        }
    }
'@ -PassThru | % Assembly | Import-Module

Typing Start-Process foo -a<tab> then cycles through the parameters in the following order:
-ArgumentArray, -Arga, -ArgumentString, -Args, and, finally, the de-emphasized original parameter name,
-ArgumentList.

That is, we would correctly prioritize the new parameter.

Sadly, the _positional_ use of the arguments parameter must continue to default to -ArgumentList (-ArgumentString), so as not to break existing code.

[SteveL-MSFT]

@PowerShell/powershell-committee need to review a new proposed parameter to accept an array of args passed as an array of args

[SteveL-MSFT]

@PowerShell/powershell-committee reviewed this, we would prefer to have a switch rather than a new parameter that changes the behavior to -ArgumentList rather than cause confusion to users between -ArgumentList vs -ArgumentArray. However, we couldn't come up with a good switch name to describe the behavior to the user without being overly verbose so open to any suggestions.

[iSazonov]

New behavior is a preferred behavior so the new switch looks like an extra parameter in scripts.
How are we going to get rid of it after the transition period? Maybe revert the switch logic? For backward compatibility users could add the switch.
Although I would just prefer the new parameter ArgumentArray. I don't think that it will confuse users if we explicitly say in helps that user should use ArgumentArray instead of ArgumentList.

[rjmholt]

The path through a new parameter rather than a switch seems clearer to me:

• Add a new parameter ArgumentArray (or Arguments, or whatever) with a separate parameter set
• Positional binding continues to be the old parameter
• In a later PowerShell version we make the breaking change to make the positional argument refer to the new parameter

[mklement0]

I also think that a new switch is the wrong way to go.

Note that the feared name confusion could be minimized based on the above proposal: rename -ArgumentList to
-ArgumentString and make -ArgumentList an _alias_ for it, for _programmatic_ backward compatibility.

With the renamed parameter, the syntax diagram would then only show -ArgumentString, which can be documented as such.

[Artoria2e5]

The new parameter thing sounda good to me. Adding it to my PR to reduce the scope of that experimental switch.