Volta: Support for non-binary global installs

[andrewplummer]

Volta throws an error Volta error: Global package installs are not supported. on adding a global yarn/npm package. I understand the reasoning for this for projects but a valid use case of global installs are scripts to be run locally outside a project directory, and volta fundamentally breaks this.

I've seen VOLTA_UNSAFE_GLOBAL env posted, however this still doesn't allow requiring the package globally, so it doesn't really serve as a workaround. Is there any other recourse?

[charlespierce]

Hi @andrewplummer, thanks for reaching out! Can you expand a bit on the use-case you’re needing global installs for? In general, a library that’s installed globally by npm or yarn can’t be imported into an arbitrary script with require().

It can generally be used by _other_ globally installed scripts, and there are a few cases where that can come up, so I’d like to know if your use-case is one we’ve run into before or something new.

At the moment, there isn’t a good story for allowing global installs to share information with one-another, that’s something we need to figure out. Knowing a wider range of uses would make it easier to see the whole picture, so we can come up with a good solution.

[andrewplummer]

In general, a library that’s installed globally by npm or yarn can’t be imported into an arbitrary script with require()

Sure it can! I have a number of one-off scripts for when I need to do things that bash just isn't quite up to (manipulating JSON data is a good example).

Here's a good example from a script I have called pretty-print-json

#!/usr/bin/env node

const fs   = require('fs');
const path = require('path');
const argv = require('argv');

const args = argv
  .option([
    {
      short: 'r',
      name: 'replace',
      type: 'boolean',
      description: 'Replace inline (default false).'
    },
    {
      short: 't',
      name: 'tabs',
      type: 'boolean',
      description: 'Use tabs (default false).'
    }
  ]).run();

if (!args.targets.length) {
  throw new Error('No targets provided.');
}

args.targets.forEach(function(p) {
  const fullPath = path.resolve(p);
  const obj = require(fullPath);
  const tabs = args.options.tabs ? '\t' : 2;
  const str = JSON.stringify(obj, null, tabs)
  if (args.options.replace) {
    fs.writeFileSync(fullPath, str);
  } else {
    console.log(str);
  }
});

[charlespierce]

Interesting! That goes against my testing on this issue. Out of curiosity, where is that pretty-print-json script in your filesystem? And are you using npm or yarn for global installs, as they put the files in different spots.

If we can reproduce this generally, then that’s _definitely_ a new use-case, so thanks!

[andrewplummer]

I have it along with a few other scripts in ~/.bin/ ...

I use both, would prefer to use yarn but I'll take what I can get!

[charlespierce]

Thanks for the info, I’ll do some testing when I’m back at my desk tomorrow and see if I can reliably reproduce that behavior! If so, we’ll need an even better story for interop with global packages.

[ljharb]

That script isn’t installed and doesn’t have any dependencies; how does it relate to globally installed npm packages?

[andrewplummer]

@ljharb to clarify argv is the package that I would be installing globally here.

[ljharb]

@andrewplummer ah, thanks, i missed that.

The problem here though is you aren't declaring which version of argv your script is compatible with, which means it can silently break at any time. The widely accepted way to handle this in the industry is to make a package for your script, with argv as a dep and your script as a "bin", and then while you could globally install it, you could also use npx to invoke it directly at any time.

[andrewplummer]

The problem here though is you aren't declaring which version of argv your script is compatible with, which means it can silently break at any time.

Yep, it's not meant to be stable, just a one off script for local use.

The widely accepted way to handle this in the industry is to make a package for your script [...]

Sure, if you're writing a library for consumption by others. But this is intended to be quick and dirty and there's a use case for that too (it's part of the reason package managers allow global installs in the first place). The hassle of having to write a package simply to create a shell script is definitely non-trivial.

[charlespierce]

Hi @andrewplummer, I just tested this locally on both MacOS and Linux, and regardless of whether or not I have argv installed globally, I always get an error about not being able to find the package when trying to run your script above.

Do you possibly have a custom NODE_PATH environment variable set on your system?

[ljharb]

global modules are indeed not requireable by default (nor should be), you'd have to have NODE_PATH set up to be able to do that deprecated thing.

[andrewplummer]

@charlespierce of course, NODE_PATH has always been required for global packages. I believe nvm adds it for you

[andrewplummer]

Actually scratch that, I believe they used to but it appears they don't anymore. Actually a good workaround here would be to simply set it in each of the required shell scripts. The only issue then is that volta is blocking a global install... it could simply allow it with some kind of override flag, I realize this isn't standard behavior.

[charlespierce]

@andrewplummer We actually do support an override for this behavior. If you run the yarn global add or npm i -g command with the environment variable VOLTA_UNSAFE_GLOBAL=1 set, then it will bypass the check and allow the global install:

$ VOLTA_UNSAFE_GLOBAL=1 yarn global add argv

And then whatever work you need to do on the script side to ensure that the require happens correctly.

[andrewplummer]

@charlespierce ok that worked for me, however I don't see any way to set the node path to allow the script to find the globally installed version

[charlespierce]

@andrewplummer I believe that yarn global add will install the packages into ~/.config/yarn/global/node_modules, so you should be able to import them by either setting NODE_PATH=$HOME/.config/yarn/global/node_modules or by importing them using an absolute path:

const argv = require('/home/username/.config/yarn/global/node_modules/argv');

[frangio]

What is the reason for not simply forwarding the install -g command to npm? I believe the user should be allowed to run this if they want to, and Volta shouldn't block standard functionality of the tools it installs.

[andrewplummer]

Another potential issue here, I run eslint from vim which isn't project based but works on the assumption of the existence of a global package install... this could potentially mess with that as well.

[charlespierce]

@frangio There are a few reasons, but the biggest is that it likely won't work the way a user would expect it to work. Since Volta manages multiple Node versions simultaneously, the "global install directory" isn't on the PATH, so any globally installed tools won't be callable (which is why Volta provides a volta install <tool> command that provides similar functionality).

For non-tool library packages, as discussed earlier in this issue, they aren't generally available to scripts anyway, so the use-cases for them are limited. We do still have some work to do around global packages needing peer-dependency libraries (either explicitly or implicitly), as well as interop between global tools.

So, generally speaking, allowing the install would most often result in other failures that would be harder to diagnose and provide helpful error messages around. Instead, we can stop in a spot where we know the end result likely won't be what the user is wanting and point them towards a solution that is more likely to work. We also do provide the VOLTA_UNSAFE_GLOBAL environment variable as a way to say "I know you think this won't work, but I actually do mean this"

[charlespierce]

@andrewplummer When you say "global install" do you mean of ESLint? Or of other libraries that ESLint needs? ESLint itself should work through volta install eslint, and then there will be an eslint executable on your PATH, so it should still be possible to call it from vim.

[frangio]

Since Volta manages multiple Node versions simultaneously, the "global install directory" isn't on the PATH, so any globally installed tools won't be callable

This isn't necessarily so. I configure npm with a custom prefix in my home directory, so no matter what npm/node version I install with Volta global installs would always go there. (With the caveat that native dependencies would likely break.) This kind of configuration is actually very common to avoid having to use sudo for global installs. What would you suggest in this scenario?

I do agree that for users who don't configure their npm prefix the behavior of global installs could be confusing. Perhaps instead of forbidding them altogether Volta could check to see if the global npm bin directory is in PATH, and that the global npm modules directory is in NODE_PATH? And only if they aren't then show this error.

[charlespierce]

@frangio The tricky part is that the version of Node / NPM that runs with Volta can change based on where you are in your file system. For example, if you have a default version of 12.13.1 installed, then when you run node in your home directory, you'll get version 12.13.1 (and the bundled version of npm). But if you happen to be working in a project that has Node pinned to 10.17.0 in package.json, then when you run Node you'll get version 10.17.0. So there isn't one global path you can use as the Node prefix.

As for how to move forward, if you're talking about globally installed _executables_, the recommended way would be to use volta install <package> as a drop-in replacement for npm install -g <package>. If you mean globally installed libraries, the recommendation would be to avoid that, but if you need it, I would say using yarn and VOLTA_UNSAFE_GLOBAL=1 yarn global add <package> would likely work slightly better. The reason for that is, in my testing, yarn installs global packages in a separate directory, so you may be able to add that yarn directory to your NODE_PATH in the same way.

If you have a specific use-case it would probably be easier to explain, as opposed to talking only in abstractions :smile:

[frangio]

So there isn't one global path you can use as the Node prefix.

Sorry if I keep insisting, but why not? My Node prefix is always ~/.local/share/npm, regardless of the Node version. I get that this might occasionally break a few things that are installed there, if they are using newer features of Node, or if they are using a dependency with native addons. But these situations are exceptions. Is there some other incompatibility that I'm not aware of?

One semi-specific (sorry) use-case I have is when I develop CLI tools. I will in some cases recommend that users do global installs via npm install -g, and I want to test that this works correctly before making the recommendation.

If I run into a specific scenario where I want to avoid volta install I will make sure to report it here. However, I still think it's worth considering in the abstract whether it's okay to block a standard and widely used feature of the tool that Volta is installing, even if Volta provides a better alternative to this feature.

[charlespierce]

@frangio Ahh, I think I misunderstood your overall setup. If you have a set npm_config_prefix and then add _that_ to your PATH, that makes sense. For that case, you should be able to use the VOLTA_UNSAFE_GLOBAL=1 environment variable to allow npm install -g to work as expected. If that environment variable is set, then the interception of global installs won't happen.

While it's definitely worth a consideration, the core point about npm install -g not working the way you might think is still important. Without specific customization (like you're doing), it will result in difficult to understand / debug errors. We are keeping an eye on it though, and as more use-cases come in we'll continue to re-evaluate the decision.

[andrewplummer]

ESLint itself should work through volta install eslint, and then...

Ahh I see because it's a binary... ok ya that should work then... nevermind my comment.

I do agree with @frangio though this setup is quite common. I'd be fine as well with an error if the setup would lead to confusion. As long as there's some way to install and access global modules from a script, which I definitely think people are going to expect.

The tricky part is that the version of Node / NPM that runs with Volta can change based on where you are in your file system. For example, if you have a default version of 12.13.1 installed, then when you run node in your home directory, you'll get version 12.13.1 (and the bundled version of npm).

But that's totally fine... dare I say expected, as node modules themselves work in exactly the same way.

[frangio]

Without specific customization (like you're doing), it will result in difficult to understand / debug errors.

What did you think about my proposal to inspect $PATH and $NODE_PATH to detect the potential errors for the users?

[charlespierce]

@frangio I definitely think if we can come up with a heuristic to give better error messages (and in the process allow some definitely-usable situations to pass through), we can do that. One concern is that without recreating the npm config / yarn config logic (which is remarkably complicated and subtly different from each other) entirely, we won't know for sure where a global install will go. Without that half of the picture, it's difficult to know for sure if an install will result in unexpected behavior or not.

@andrewplummer There's definitely still an outstanding question about global modules in a few cases. Yours is one, another is plugin modules for global tools (like yeoman). One approach may be to provide a command (possibly even using the existing volta which) for locating global installs, then those can be added to the NODE_PATH. At that point we could support installing global packages without binaries, because there would be a way to find and use them.

[charlespierce]

Another use-case for global installs / allowing packages that don't have binaries (reported in #624): Packages that exist primarily to run a postinstall script, such as https://github.com/sindresorhus/alfred-npms

[charlespierce]

This should be resolved as of Volta 0.9.0. We now support non-executable packages (as well as using npm i -g directly)