Powershell: Command line parsing bug: spaces and trailing backslash

[akervinen]

This has already been posted on UserVoice (here), but appears to still happen in 6.0.

Steps to reproduce

Using any .exe capable of outputting its command line arguments:

PS X:\scratch> .\ps-args-test.exe .\test1\ '.\test 2\' .\test1\

Auto-completion uses this format with a trailing backslash, so it's expected to work correctly.

Expected behavior

.\test1\
.\test 2\
.\test1\

Actual behavior

.\test1\
.\test 2" .\test1\

Environment data

> $PSVersionTable
Name                           Value
----                           -----
PSVersion                      6.0.0-beta
PSEdition                      Core
GitCommitId                    v6.0.0-beta.4
OS                             Microsoft Windows 10.0.14393
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

[mklement0]

@iSazonov:

This is only indirectly related to IntelliSense (which _suggests_ passing the arguments like that).

The real issue is how the command line is rebuilt _behind the scenes_ by PowerShell:

Argument list .\test1\ '.\test 2\' .\test1\ is translated to .\test1\ ".\test 2\" .\test1\ behind the scenes, which any external utility that uses \ as the escape character (most of them) parses as demonstrated in the initial post, causing the initial argument partitioning to be misinterpreted:

• The 2nd " in ".\test 2\", due to being preceded by \ is interpreted as an _escaped_ ", causing the remainder of the string - despite a then-missing closing " to be interpreted as part of the same argument.

---

The - cumbersome - workaround on Windows is to add an extra \, though, given that PowerShell does _re-quoting_ behind the scenes (as is _necessary_ on Windows), the fix is to let _PowerShell_ do that _automatically_:

> .\ps-args-test.exe .\test1\ '.\test 2\\' .\test1\  # Note the '\\'

The - proper, hopefully-soon - fix on Unix platforms is to pass the arguments as an _array of literals_ to the target utility, as is the norm on Unix platforms, bypassing such issues altogether. (Re-quoting behind the scenes is _unnecessary_ on Unix platforms and should be _avoided altogether_.)

cc @SteveL-MSFT

[SteveL-MSFT]

Unless I hear otherwise, this doesn't seem to be a common use case. I think we can fix in 6.1.0 unless someone from the community submits a PR.

[akervinen]

I'd argue it's common since it happens every time I auto-complete a directory name with a space unless I remove the trailing slash manually. My example was a bit incomplete I think, since the extra quote is also inserted with just one argument.

PS X:\scratch> .\ps-args-test.exe '.\test 2\'

.\test 2"

A shell that has a broken auto-complete for directories is... inconvenient.

[SteveL-MSFT]

@akervinen I see, every time with a space is definitely a concern

[daxian-dbw]

The trailing slash is added by PSReadline, not by powershell tab completion:

PS> $s = TabExpansion2 -inputScript 'git t' -cursorColumn 'git t'.Length
PS> $s.CompletionMatches[0].CompletionText
'.\test 2'

So this is more of a PSReadline issue, maybe PSReadline should have an option to no adding trailing slash for directories?

[mklement0]

@daxian-dbw:

Not appending a \ would be a band-aid, especially given that nothing stops a user from _typing_ a path that ends in \.

There is no good reason not to properly support passing of whitespace-containing arguments that end in \.

At the risk of repeating myself: The problem is a fundamental problem with how PS rebuilds the command line behind the scenes on Windows and that it even _does so at all_ on Unix platforms, and it should be fixed at that level.

[daxian-dbw]

@mklement0 I see your point now.
The problem is in NativeCommandParameterBinder.cs, which cause the argument passed in native commands like this:

PS> testexe.exe -echoargs '.\test 2\'
Arg 0 is <.\test 2">

[JVimes]

Just lost an hour wondering why git status wouldn't work with -C. And a number of RoboCopy folks lost time to it.

This use case gets hit!

[daxian-dbw]

The - cumbersome - workaround on Windows is to add an extra \, though, given that PowerShell does re-quoting behind the scenes (as is necessary on Windows), the fix is to let PowerShell do that automatically:
.\ps-args-test.exe .\test1\ '.\test 2\' .\test1\ # Note the '\'

@mklement0 Do you think we should do this fix for now or shall we wait until https://github.com/dotnet/corefx/issues/23592 is addressed?

I'm trying to do this fix now but having some trouble figuring out the reasonable behavior handling the trailing backslashes of an arg string. When the arg string needs to be quoted, the trailing backslashes will escape each other and if there are odd number of backslashes, then the closing quote will be misinterpreted, for example:

PS> testexe.exe -echoargs 'test \'
Arg 0 is <test ">
PS> testexe.exe -echoargs 'test \\'
Arg 0 is <test \>
PS> testexe.exe -echoargs 'test \\\'
Arg 0 is <test \">
PS> testexe.exe -echoargs 'test \\\\'
Arg 0 is <test \\>

Now, we think the resulting " is confusing, and the ending \ should be preserved when passing to the native command. As a result, we will see behaviors like this:

PS> testexe.exe -echoargs 'test \'
Arg 0 is <test \>
PS> testexe.exe -echoargs 'test \\'
Arg 0 is <test \>
PS> testexe.exe -echoargs 'test \\\'
Arg 0 is <test \\>
PS> testexe.exe -echoargs 'test \\\\'
Arg 0 is <test \\>

You can see that when there is an odd number of backslashes, the ending backslash will be preserved, which makes the odd number of trailing backslashes always behave as _(odd number + 1)_ trailing backslashes. Does this look like a reasonable behavior?

[mklement0]

@daxian-dbw:

My sense is that once this RFC proposal by @TSlivede is implemented, it will cover this scenario too, so we probably don't need a separate fix for this specific issue.

For background, see https://github.com/PowerShell/PowerShell/issues/1995#issuecomment-325231274

[TSlivede]

Yes, that's true. My RFC-Proposal (or any other fix for https://github.com/PowerShell/PowerShell/issues/1995) should also fix this. Basically we could rename https://github.com/PowerShell/PowerShell/issues/1995 to something like "Arguments for external executables aren't correctly escaped". In that case this issue could be closed as duplicate.

[SteveL-MSFT]

@TSlivede makes sense. Dupe of https://github.com/PowerShell/PowerShell/issues/1995

[SteveL-MSFT]

I'm separating this out from the uber issue of args to native commands which might be resolved by a future corefx api. I'll see if there is something we can do to solve this one sooner.