Yarn: Globally installed binaries do not work in PowerShell after 1.13.0

[ArcanoxDragon]

Since the 1.13.0 update, globally installed binaries do not work in PowerShell. The offending commit is 5419606, which is the update to the cmd-shim package. The version of cmd-shim that yarn updated to added default support for PowerShell, meaning that every binary writes a .ps1 shim in addition to the non-extension and the .cmd ones. It appears Yarn is first calling cmdShim during the "Linking Dependencies" phase to create the initial bin shims for the package. This is fine and dandy here, but later in the install step (after the success Installed <package> with binaries message is printed), Yarn calls cmdShim again once per binary, which causes cmdShim to write a shim for each of the shims. This results in files like binary.cmd.ps1, binary.ps1.ps1, etc, because cmd-shim thinks that the base name of the binary is actually binary.ps1. This causes the bash shim to be written to binary.cmd and binary.ps1, breaking those two shims (PowerShell finds the ps1 file which is actually a bash script). Here's a visual after yarn global add ts-node:

I added a console.trace to cmdShim in my local copy of Yarn to print out the arguments passed, and this is the output:

https://gist.github.com/briman0094/d7f256d1158d6e8f96afbb74e85840bb

[vladimiry]

Same issue https://ci.appveyor.com/project/vladimiry/email-securely-app/build/job/p9cy9yoosgfylw50 right?

/home/appveyor/.yarn/lib/cli.js:45726
  let {
      ^
SyntaxError: Unexpected token {
    at exports.runInThisContext (vm.js:53:16)
    at Module._compile (module.js:373:25)
    at Object.Module._extensions..js (module.js:416:10)
    at Module.load (module.js:343:32)
    at Function.Module._load (module.js:300:12)
    at Module.require (module.js:353:17)
    at require (internal/module.js:12:17)
    at Object.<anonymous> (/home/appveyor/.yarn/bin/yarn.js:24:13)
    at Module._compile (module.js:409:26)
    at Object.Module._extensions..js (module.js:416:10)
> Yarn was installed, but doesn't seem to be working :(.
Command exited with code 1

cli.js:45726:

45726:  let {
45727:    win32: nodePath,
45728:    posix: shNodePath
45729:  } = normalizePathEnvVar(opts.nodePath)

[ArcanoxDragon]

No, that looks like a syntax error in the actual JS file. This is causing a syntax error is coming from PowerShell because it's finding a PS1 file that actually contains a Unix Bash script

[cstrachan88]

I am having the exact same issue and can confirm downgrading to 1.12.3 after clearing the global bin does fix the issue.

Here is a gist for the files created by Yarn 1.12.3 for @vue/cli.
Here is a gist for the files created by Yarn 1.13.0 for the same package.

As @briman0094 stated, the xxx.ps1 file contains the Unix bash script.

[choclaudio]

bump

[farzadmf]

This is so annoying. Bash syntax in ps1 file??!!

[tats-u]

I bumped into the similar problem. 😕
I'd like you to fix this offending issue ASAP

(node) PS C:\Users\tatsu\Documents> ls (split-path -parent (gcm yo.ps1).path)


    Directory: C:\Users\tatsu\scoop\apps\yarn\current\bin


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----       2019/01/17      0:58            416 express
-a----       2019/01/17      0:58             91 express.cmd
-a----       2019/01/17      0:58            382 express.cmd.ps1
-a----       2019/01/17      0:58            415 express.ps1
-a----       2019/01/17      0:58            292 express.ps1.cmd
-a----       2019/01/17      0:58            600 express.ps1.ps1
-a----       2019/01/17      0:58            406 yo
-a----       2019/01/17      0:58            424 yo-complete
-a----       2019/01/17      0:58             95 yo-complete.cmd
-a----       2019/01/17      0:58            386 yo-complete.cmd.ps1
-a----       2019/01/17      0:58            423 yo-complete.ps1
-a----       2019/01/17      0:58            300 yo-complete.ps1.cmd
-a----       2019/01/17      0:58            608 yo-complete.ps1.ps1
-a----       2019/01/17      0:58             86 yo.cmd
-a----       2019/01/17      0:58            377 yo.cmd.ps1
-a----       2019/01/17      0:58            405 yo.ps1
-a----       2019/01/17      0:58            282 yo.ps1.cmd
-a----       2019/01/17      0:58            590 yo.ps1.ps1

(node) PS C:\Users\tatsu\Documents> cat (gcm yo.ps1).path
#!/bin/sh
basedir=$(dirname "$(echo "$0" | sed -e 's,\\,/,g')")

case `uname` in
    *CYGWIN*) basedir=`cygpath -w "$basedir"`;;
esac

if [ -x "$basedir/pwsh" ]; then
  "$basedir/pwsh"  "$basedir/../../../../../AppData/Local/Yarn/Data/global/node_modules/.bin/yo.ps1" "$@"
  ret=$?
else
  pwsh  "$basedir/../../../../../AppData/Local/Yarn/Data/global/node_modules/.bin/yo.ps1" "$@"
  ret=$?
fi
exit $ret
(node) PS C:\Users\tatsu\Documents> cat (gcm yo.ps1.ps1).path
#!/usr/bin/env pwsh

if ($PSVersionTable.PSVersion -lt "6.0" -or $IsWindows) {
  # Fix case when both the Windows and Linux builds of Node
  # are installed in the same directory
  $exe=".exe"
}
$ret=0
if (Test-Path "$basedir/pwsh$exe") {
  & "$basedir/pwsh$exe"  "$basedir/../../../../../AppData/Local/Yarn/Data/global/node_modules/.bin/yo.ps1" $args
  $ret=$LASTEXITCODE
} else {
  & "pwsh$exe"  "$basedir/../../../../../AppData/Local/Yarn/Data/global/node_modules/.bin/yo.ps1" $args
  $ret=$LASTEXITCODE
}
exit $ret
(node) PS C:\Users\tatsu\Documents> cat (gcm yo.cmd.ps1).path
#!/usr/bin/env pwsh
$basedir=Split-Path $MyInvocation.MyCommand.Definition -Parent

$exe=""
if ($PSVersionTable.PSVersion -lt "6.0" -or $IsWindows) {
  # Fix case when both the Windows and Linux builds of Node
  # are installed in the same directory
  $exe=".exe"
}
& "$basedir/../../../../../AppData/Local/Yarn/Data/global/node_modules/.bin/yo.cmd"   $args
exit $LASTEXITCODE
(node) PS C:\Users\tatsu\Documents> cat (gcm yo.cmd).path
@"%~dp0\..\..\..\..\..\AppData\Local\Yarn\Data\global\node_modules\.bin\yo.cmd"   %*
(node) PS C:\Users\tatsu\Documents> cat (join-path (split-path -parent (gcm yo.cmd).path) "yo")
#!/bin/sh
basedir=$(dirname "$(echo "$0" | sed -e 's,\\,/,g')")

case `uname` in
    *CYGWIN*) basedir=`cygpath -w "$basedir"`;;
esac

if [ -x "$basedir//bin/sh" ]; then
  "$basedir//bin/sh"  "$basedir/../../../../../AppData/Local/Yarn/Data/global/node_modules/.bin/yo" "$@"
  ret=$?
else
  /bin/sh  "$basedir/../../../../../AppData/Local/Yarn/Data/global/node_modules/.bin/yo" "$@"
  ret=$?
fi
exit $ret
(node) PS C:\Users\tatsu\Documents> yarn --version
1.13.0
(node) PS C:\Users\tatsu\Documents> node --version
v11.6.0
(node) PS C:\Users\tatsu\Documents> npm --version
6.5.0-next.0
(node) PS C:\Users\tatsu\Documents> gcm yarn

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Application     yarn.cmd                                           0.0.0.0    C:\Users\tatsu\scoop\apps\yarn\current\Yarn\bin\yarn.cmd

tatsu@TATSU-NB-3RD:/mnt/c/Users/tatsu/Documents$ diff -up /mnt/c/Users/tatsu/scoop/apps/yarn/current/bin/yo{,.ps1}
--- /mnt/c/Users/tatsu/scoop/apps/yarn/current/bin/yo   2019-01-17 00:58:48.299169900 +0900
+++ /mnt/c/Users/tatsu/scoop/apps/yarn/current/bin/yo.ps1       2019-01-17 00:58:48.485789500 +0900
@@ -5,11 +5,11 @@ case `uname` in
     *CYGWIN*) basedir=`cygpath -w "$basedir"`;;
 esac

-if [ -x "$basedir//bin/sh" ]; then
-  "$basedir//bin/sh"  "$basedir/../../../../../AppData/Local/Yarn/Data/global/node_modules/.bin/yo" "$@"
+if [ -x "$basedir/pwsh" ]; then
+  "$basedir/pwsh"  "$basedir/../../../../../AppData/Local/Yarn/Data/global/node_modules/.bin/yo.ps1" "$@"
   ret=$?
 else
-  /bin/sh  "$basedir/../../../../../AppData/Local/Yarn/Data/global/node_modules/.bin/yo" "$@"
+  pwsh  "$basedir/../../../../../AppData/Local/Yarn/Data/global/node_modules/.bin/yo.ps1" "$@"
   ret=$?
 fi
 exit $ret
tatsu@TATSU-NB-3RD:/mnt/c/Users/tatsu/Documents$ diff -up /mnt/c/Users/tatsu/scoop/apps/yarn/current/bin/yo.{cmd,ps1}.ps1
--- /mnt/c/Users/tatsu/scoop/apps/yarn/current/bin/yo.cmd.ps1   2019-01-17 00:58:48.447668900 +0900
+++ /mnt/c/Users/tatsu/scoop/apps/yarn/current/bin/yo.ps1.ps1   2019-01-17 00:58:48.485165400 +0900
@@ -7,5 +7,12 @@ if ($PSVersionTable.PSVersion -lt "6.0"
   # are installed in the same directory
   $exe=".exe"
 }
-& "$basedir/../../../../../AppData/Local/Yarn/Data/global/node_modules/.bin/yo.cmd"   $args
-exit $LASTEXITCODE
+$ret=0
+if (Test-Path "$basedir/pwsh$exe") {
+  & "$basedir/pwsh$exe"  "$basedir/../../../../../AppData/Local/Yarn/Data/global/node_modules/.bin/yo.ps1" $args
+  $ret=$LASTEXITCODE
+} else {
+  & "pwsh$exe"  "$basedir/../../../../../AppData/Local/Yarn/Data/global/node_modules/.bin/yo.ps1" $args
+  $ret=$LASTEXITCODE
+}
+exit $ret

[tats-u]

How to avoid this problem provisionally:

1. Remove global packages and (scoop uninstall yarn) -and (scoop install [email protected])
2. Run express.ps1.ps1 express.cmd or express.cmd.ps1 instead of express

I wonder why ONLY global packages cause this probrem. There is no problem in packages in every node_modules/.

[tats-u]

I found the mechanism of this bug.

1. getBins (called at https://github.com/yarnpkg/yarn/blob/1af4c5f50c4c146a833c0c2b20d10be1f5a61803/src/cli/commands/global.js#L144 and defined in https://github.com/yarnpkg/yarn/blob/1af4c5f50c4c146a833c0c2b20d10be1f5a61803/src/cli/commands/global.js#L60-L81) lists all shims including .cmd and .ps1 in node_modules\.bin in the global folder.
2. linkBin is called every for all shims. (Called at: https://github.com/yarnpkg/yarn/blob/1af4c5f50c4c146a833c0c2b20d10be1f5a61803/src/cli/commands/global.js#L168)
3. cmdShim (called by linkBin at https://github.com/yarnpkg/yarn/blob/f3fb1102ff83fbdef53855767ee3902c53d17639/src/package-linker.js#L37) in @zkochan/cmd-shim (defined in https://github.com/pnpm/cmd-shim/blob/b3c57c06211afd3738ecdfd6fde94ea9d49b7de1/index.js#L40-L44) is run with every shims (no extension, .cmd, and .ps1) and creates sh, .cmd, and .ps1 script shims for every src (scripts). As the result, junk scripts (sh scripts with extensions .cmd and .ps1, and batch or PowerShell scripts with double extensions) are created.
4. Scripts with the same names are overwritten by ones that tried to be created later.

https://github.com/yarnpkg/yarn/blob/1af4c5f50c4c146a833c0c2b20d10be1f5a61803/src/cli/commands/global.js#L169-L171 suppressed this bug before 1.123.

[xPixv]

Same problem.

Rolled back to 1.12.3, :(

[hellraisercenobit]

+1

[ArcanoxDragon]

Fun fact: after fixing the duplicate-extension issue, I determined that the @zkochan/cmd-shim PowerShell shims are just broken altogether. They appear to try and determine which version of PowerShell is installed which binary is called from the shim, but not correctly, resulting in nonexistent binaries such as pwsh.exe and sh.exe being called instead of the typical node.exe. It's easier just to disable them for now and fall back to the .cmd shims like before, because they work.

[tats-u]

@briman0094 It's not a problem that @zkochan/cmd-shim tries to create PowerShell shims because shims to JS files (in node_modules/.bin) work fine.
WHAT'S A PROBLEM IS THAT IT CAN'T CREATE SHIMS OF SHIMS!
You must understand that the PR #6954 is just a makeshift, not the solution to the root of this problem.

[tats-u]

As this issue is likely to be closen by the makeshift, it's necessary to create a new issue to spread of the root of this problem.

[ArcanoxDragon]

Yarn was creating shims of shims before the 1.13.0 update. The difference is that it took care of the duplicates after the fact. I agree that the root cause needs to be addressed (which would most likely allow the new PowerShell shims to actually work correctly), but every PowerShell user who updates to 1.13.0 is left unable to use any globally-installed packages whatsoever. It would make sense to regress the change that broke the shims as a minor update until a proper fix can be applied, as opposed to making all PowerShell users suffer until then.

[tats-u]

All right, this issue must be fixed ASAP and even a hotfix must be merged. I can wait for the root of this problem solved.

[cfanoulis]

Why is yarn using .ps1 files to begin with? Aren't the generated .cmd files enough?

[ArcanoxDragon]

Yarn isn't generating the .PS1 files; cmd-shim is. Yarn updated its dependency on that package to a new major version, which happened to include support for PS1 files. I agree that .cmd files suffice, but I suppose the author of cmd-shim felt that .PS1 support was necessary

[Fernando-Marquardt]

It seems this update (on Yarn) was not done properly, so why this was not reverted yet? At least, until someone takes the time required to do it right and test it?

[ExE-Boss]

That’s odd, it works fine for me and pnpm.
(Edit: I don't use yarn for global packages, and pnpm doesn't generate shims of shims)

PowerShell support was added to fix the https://github.com/pnpm/pnpm/issues/1199 and https://github.com/npm/npm/issues/20699 issues.

---

Also, pwsh.exe is the PowerShell 6 binary

/cc @zkochan

[ArcanoxDragon]

It's not a cmd-shim issue; it's an issue with yarn. I have an outstanding PR to fix it

[tats-u]

cmd-shim is also to blame.
The problem will fixed w/o any PRs if it supports shims of shims.
Your PR (s)must be merged ASAP, but is just to adapt Yarn for the current behaviors of cmd-shim.

[nowarzz]

I'm also having the same issue after upgrading yarn

[sueblue]

I am having the exact same issue and can confirm downgrading to 1.12.3 after clearing the global bin does fix the issue.

Here is a gist for the files created by Yarn 1.12.3 for @vue/cli.
Here is a gist for the files created by Yarn 1.13.0 for the same package.

As @briman0094 stated, the xxx.ps1 file contains the Unix bash script.

mark.tks

[hediet]

This is why I don't like yarn - they don't care about windows at all. Here is another example.

[DavidM77]

@torifat, @arcanis - The fix for this issue (#6954) seems to have been waiting to be merged for the last month and at least one major release.

Is there anything that can be done to expedite this. Is it worth while creating an alternate pull request if there is some problem blocking this one?

I might add that not only does this issue make yarn global unusable on Windows, accidentally running it also hoses npm -g installations which I had hoped would provide a workaround.

[ArcanoxDragon]

I actually submitted two possible fixes in hopes that one might be accepted sooner than the other but neither has been reviewed by a maintainer...it's certainly surprising that nobody has looked at them yet. This causes major widespread issues on Windows.

[xPixv]

So, this problem still not solved in Release Candidate v1.15.0? Not solved after 2 candidate version released?
What are you thinking Yarn Team?
You known this issue, but you do NOT willing to solve(review & merge) it?

It's time to say goodbye (before this issue solved or forever). :)

[tats-u]

Yarn Team MUST make a decision to merge either of @briman0094 's patches (#6954, #6959 ) ASAP.

[arcanis]

Let's clear up a misconception.

What are you thinking Yarn Team?
You known this issue, but you do NOT willing to solve(review & merge) it?

First, we don't have enough day-to-day core contributors - I've been the only one actively merging PRs this past year. If there's anything I should be spending my time on, it would be finding new contributors willing to help managing this repository.

Second, I'm working on a major release that overhaul most of the components affected in those issues. My priorities currently are long-term issues more than implementation ones - since the implementation will change anyway.

Third, I'm rewriting our documentation. Think of it as writing a book - it's really not something you have just by wishing it with all your heart - you need to spend time on it.

Fourth, I have an actual job I'm being paid to do, and maintaining Yarn is but a part of it. I've literally been spending my days and nights working on Yarn this past year, and you really don't get to tell me what should my priorities be - or at least not in this way.

Fifth, the problem of this PR is that it's for Windows and I don't use Windows. That doesn't make its priority lower because it's Windows, mind you, but I personally lack a good chunk of context about the problem it fixes and the way it solves it. Ideally someone with Windows should review & merge it, but cf my first point.

It's time to say goodbye (before this issue solved or forever). :)

Tough luck, I don't know how I'll do without having the joy of receiving your aggressive messages right after getting out of bed.

The only one here that has any right to be slightly corrosive would be @briman0094, who actually spend time to help the project. To him I'd like to apologize - anyone else is strongly suggested to stay civil and remember there are human beings behind the screen.

End of the digression, back to the topic.

[choclaudio]

Hi @arcanis. I do agree with your post, I do not think that you should get flamed. but I also have some comments to it.

1) If you need more maintainers, maybe @briman0094 would be a good bet?
2) If you need more maintainers, maybe calling a wide request for it could be a good idea?
3) I understand and respect that you are working on a future release, and new documentation. Having that said this issue isolates all of your windows users. There might be multiple companies that actually use Yarn and/or want to use Yarn that with the current state just cant come up with great arguments to keep using or adopting Yarn.

I understand that windows users can downgrade, but if no path forward (like an upcomming overhaul that will solve the issues) is presented to a wider audience, then it is practically the same as the issue being ignored.

I wish all good things for you and thank you for taking your time.

[xPixv]

I'm sorry for my aggressive words. and apologize to Yarn Team.

I have no right push you do so. Yarn is a cross-platform package management. All the Windows users who want add global package(s) will encounter this problem during these two months (even longer). As you can see, many issues refer to this problem including vue.js users.

If this issue still remaining, your long-term issues will make no sense for Windows users. Maybe you and your team are working on Typescript rewrite or yaml lockfile and pushing Yarn to v2. They're loooong-term work. I wish you release a stable version before digging in them.

If you really have no time on solving it. I recommend leaving a message on Yarn Official Website to warn Windows users and put v1.12 msi file on it for them. So that we can avoid this annoying problem.

I apologize again for what I saw before. I'm so sorry.

[jthile]

I’m very sorry to hear that Yarn is maintained by a single, overworked developer! You have my sympathy.

As a Windows user, I will be moving back to NPM. It’s a little disingenuous to claim to support Windows when the project is so undersupported that the single dev involved can’t or won’t keep it working on the platform.

[arcanis]

Regarding the main topic: #6954 seems good enough (and easy enough to revert if it breaks something else, which is important) and depending what @zkochan says I'm ok to go with it in the next patch release.

---

I apologize again for what I saw before. I'm so sorry.

Thanks for your understanding 🙂 we all want things to get better, and I can appreciate that.

It’s a little disingenuous to claim to support Windows when the project is so undersupported that the single dev involved can’t or won’t keep it working on the platform.

It's a bit unfair - note that the problem isn't coming from Yarn, technically speaking, but rather from a third-party package (cmd-shim). Part of the reason why these PRs didn't get merged was that I was hoping that the fix would be made upstream. The whole goal of cmd-shim is precisely to abstract this logic so that we don't have to care about it, after all! 😄

If you need more maintainers, maybe calling a wide request for it could be a good idea?

A call for maintainers is a good idea, but we first need to figure out some parts of our governance model before being ready to do it. I've opened https://github.com/yarnpkg/yarn/issues/7110 to discuss it further, if you're interested.

[cstrachan88]

Wouldn't #6959 be the better PR to merge? That PR seems to take care of the root problem whereas #6954 is just a band aid fix as @briman0094 says.

[arcanis]

Keeping open until the 1.15.1 ships.

@cstrachan88 I prefer to merge the safest PR. The other one instantiates a whole new Install object, which complexifies the codebase enough that I wouldn't feel comfortable merging it as is without doing some back and forth to at least add comments explaining the process.

[ArcanoxDragon]

I figured the more complex fix in #6959 would require quite a bit more discussion, hence why I made the bandaid fix in #6954 first ;)

I'm absolutely willing to participate in discussion and review of #6959 to evaluate if it's the best solution to the root problem or to see if there's another better solution

[arcanis]

The 1.15.2 (small infra hiccups for the 1.15.1) has just been released - I'd appreciate if someone here could confirm the problem is resolved on your side, then I'll go ahead and close this issue while we work on #6959 🙂

[Ezerous]

It's resolved!! 🎉

[curtisrutland]

I can also confirm; after installing 1.15.2, I can run binaries from globally installed packages. Thank you for your efforts in resolving this issue!

[cstrachan88]

Also confirming 1.15.2 correctly adds global packages. No more shims of shims!

[arcanis]

Awesome 👍