Msys2-packages: Add option to disable path mangling when calling non-msys2 binaries

[gintas]

MSYS2 binaries automatically mangle arguments when calling non-MSYS binaries, applying a set of heuristics. Heuristic transformations are ugly; not being able to disable the heuristic transformations is just nasty. I've run into a few cases already in a just few days where the path mangling was causing trouble, and it is particularly annoying to have to work around.

For reference, the arguments are mangled here:

https://github.com/Alexpux/Cygwin/blob/6e95f9f5eab763ba36c8d5c62a22a4db9cca1d7f/winsup/cygwin/spawn.cc#L416

How about making it possible to disable the mangling by setting an environment variable? Something like CYGWIN_DISABLE_ARGUMENT_MANGLING=1. The environment variable would be accessible from anywhere in the code and thus should be unintrusive regarding the rest of the cygwin code, the patch should be just a couple lines.

[Alexpux]

First, it would be helpful if you provide such examples that not correctly mangled to be able try fix its converting.

Second, you can disable mangling some arguments with MSYS2_ARG_CONV_EXCL environment variable. This variable can contain semicolon separated list of arguments started with values that don't need to be converted. See for example:
expux/MINGW-packages/blob/master/mingw-w64-python-sphinx/PKGBUILD#L61

[mingwandroid]

Heuristic transformations are ugly

Unfortunately they're also essential.

Something like CYGWIN_DISABLE_ARGUMENT_MANGLING=1

Name mangling has nothing to do with Cygwin. It is the main feature that makes MSYS* different from Cygwin.

[gintas]

Ah, I see. I was looking at the master branch of Alexpux/Cygwin, but I should have been looking at the develop branch. Sorry about that! The exception handling code there is clear to see, looks like that variable should be enough for my purposes.

Is MSYS2_ARG_CONV_EXCL documented anywhere? A google search brought up nothing, and grep in the repository only found matches in the source code.

[mingwandroid]

Is MSYS2_ARG_CONV_EXCL documented anywhere?

We're too busy fixing bugs to document stuff, our documentation lives in the source code and places like this. In fact these github issues for MSYS2-packages and MINGW-packages are a gold mine of 'stuff'.

Some useful pointers for what's what:

https://github.com/Alexpux/MSYS2-packages/tree/master/msys2-runtime/
https://github.com/Alexpux/MINGW-packages/tree/master/

We would still like examples of paths that got mangled incorrectly if possible? We have a project in which this stuff is developed with a test suite and we'd like to add your cases to that if they are fairly unambiguously being misconverted:

https://github.com/Alexpux/path_convert

[mingwandroid]

Of course you are right about documentation, so I wrote some. Would you care to review it and suggest any improvements?

https://sourceforge.net/p/msys2/wiki/Contributing%20to%20MSYS2/

[gintas]

Regarding incorrectly mangled paths, those were not paths but strings that looked like paths. One was of the shape //foo/bar; that seems to work fine in a recent version of MSYS2 actually. The other was, I think, a sed expression, but I don't remember exactly. Thanks for taking this seriously, I will let you know if I come up with an interesting case of bad behavior.

Also thanks a lot for adding some documentation. At least a google search for MSYS2_ARG_CONV_EXCL should now bring up _something_ :)

One question regarding MSYS2_ARG_CONV_EXCL: is there a value that would simply turn off argument mangling in all cases? I think the use case makes sense, being able to say that the caller will take care of the arguments being in the right shape.

[mingwandroid]

Not at present, we could add MSYS2_ARG_CONV_EXCL="*" as a special case I guess?

[mingwandroid]

Path mangling has been suffering some turbulence recently as we've implemented the new MSYS2 code to replace the legacy MSYS1 version we had been using. A few corner cases continue to come out of the woodwork. Hopefully that'll stop soon.

[gintas]

• as a special case sounds perfectly reasonable to me.

Thanks for your work on this.

[mingwandroid]

That's done, next time msys2-runtime is released it will be included.

[lpintes]

I have one question concerning this:
I have my path set up so that I can use git with another tools from MSYS2.
Recently I tried magit, and it always failed with "xxx is not a directory", because paths are mangled and Emacs is confused.
No matter what I did, it always failed.

Now I suppose that I just cannot use git from MSYS2 with magit, is this true?
Note I don't want to use Emacs from MSYS2, because I made my Emacs "portable", that only means I configured "home" directory to be under emacs\home.

[DavidEGrayson]

Documentation for MSYS2_ARG_CONV_EXCL

For anyone who is landing on this page because it's the top Google hit for MSYS2_ARG_CONV_EXCL and you are looking for documentation, go here:

https://github.com/msys2/msys2/wiki/Porting

Update: I changed the link to point to the new GitHub Wiki.

[573]

@DavidEGrayson https://sourceforge.net/p/msys2/wiki/Porting/ is behind an auth wall.

[mati865]

@573 moved here https://github.com/msys2/msys2/wiki/Porting

[parleer]

Example of MSYS incorrectly mangling paths:

docker run --rm mysql:5.7 /usr/bin/mysqldump --host=hostname--user=user--password=password-v dbname > db.sql

And pretty much any other command involving docker. Docker containers expect real POSIX paths, but docker on windows is not an MSYS executable, so paths are getting mangled and the container doesn't know what to do.

Example error:

/usr/local/bin/docker-entrypoint.sh: line 215: /C:/Program Files/Git/usr/bin/mysqldump: No such file or directory

[mingwandroid]

There's no correct or incorrect about this I am sorry to say. At least not in general. It's a simple set of rules that are followed. This thread also explains how to disable conversion. Did I miss somethig?

[mbacs]

Here is an example needing a "master disable" of path mangling.
All other Amazon ssm parameter management commands seem to work.
For AWS SSM, , the path is NOT a file path. It is an AWS SSM Parameter path.
(I did not find in this thread a way to disable path conversion for all arbitrary AWS SSM parameter commands)

$ aws ssm get-parameter --name "/subfolder/PARAMETER1" --with-decryption
An error occurred (ParameterNotFound) when calling the GetParameter operation:

To AWS SSM get-parameter work, change to mangled form.

This is now different than AWS CLI on any other platform including CMD and PowerShell.

$ aws ssm get-parameter --name "//subfolder\PARAMETER1" --with-decryption {
"Parameter": {
"Name": "/subfolder/PARAMETER1",
"Type": "SecureString",
"Value": "TestSetParameter",
"Version": 2,
"LastModifiedDate": 1564611241.75,
"ARN": "arn:aws:ssm:us-west-2:999999999999:parameter/subfolder/PARAMETER1"
}
}