Volta: Support alternative version pinning mechanism

[dcramer]

We love the idea of Notion (and nvm) as it makes it easier to persist a shared version of Node.js throughout many environments (local, Travis, production [Docker]). However, there's a limitation in Notion that we didn't have with nvm, and that comes from locking node/yarn within package.json itself.

This limitation realistically only comes up when you're doing layer caching, which if you're not familiar with I'm probably not a good person to learn from, but here we go:

We try to install Node and capture it outside of package.json because it means we dont have to rebuild the Node.js layer every time packages change. Dockerfile looks something like:

COPY .nvmrc /usr/src/zeus/
ENV YARN_VERSION 1.7.0
RUN set -x \
  && export NODE_VERSION=$(cat /usr/src/zeus/.nvmrc) \
  [...]

Now this worked actually extremely well. We had a single file we could pull/parse to get the node version everywhere.

When moving to Notion our only real alternative is passing a Docker build arg:

--build-arg NODE_VERSION=$(shell bin/get-node-version | tr -d '\n') \

This is fine and all, but tools like Google's Cloud Build (cloudbuild.yaml) doesn't let you do dynamic arguments like this. That means we're stuck with either rebuilding the Node.js layer every time package.json changes, or hardcoding NODE_VERSION elsewhere.

I'm not proposing a solution here, but wanted to make sure everyones aware of this problem as its a shortcoming with a lot of package management tooling.

[dcramer]

/cc @mitsuhiko who also has been playing with Notion on some of our other projects and might have additional feedback

[charlespierce]

Hi @dcramer, thanks for bringing this issue up! In our design discussions, we have always wanted to avoid adding a notion configuration file to a project, as we didn't want to add yet another required file that everyone who uses the tool would have to check in to source control.

It seems like, for this use case, you would _prefer_ a separate Notion configuration file, so that you can use that as a single source of truth that is distinct from package.json? If that's the case, it seems like a reasonable compromise would be to allow Notion to handle both cases: Configuration inside of package.json as well as configuration in a separate config file. We'll need to hash out the details of precedence and how to handle the notion pin command, but that could provide a possible solution.

Another possible option, if I'm interpreting your concern correctly, would be to have a Docker image that has Notion installed, and then let that Notion install handle fetching node the first time it is activated. Would that work for your situation as an alternative?

[dcramer]

@charlespierce the docker image wont help us (we need base images and then to layer things on top). The main concern is that we want to be able to define a version via a filesystem object that is not in package.json. The reason is that we can hash(that file) to know if Node.js needs updated or not, whereas doing that package.json will suggest Node.js needs updated almost every day based on unrelated dependency changes.

In general I think this is going to be an ongoing industry-wide concern, but I've seen it's often not been approached in a way that works well with existing systems.

The alternative is that whenever we change our Node.js or Yarn versions we have to update multiple locations. This issue is on the goal of trying to accomplish a singular canonical version reference, without compromising build caches (and thus build time/resources).

If Notion decides "package.json is the standard and we dont want to have multiple locations" I think that's totally fair. Again, to me this is a general industry-wide problem that very few package management solutions have resolved. We got someone lucky that nvm was doable in a manner that worked for us, but Notion is easier for developers to work with out of the box.

[chriskrycho]

@dcramer you're not alone in that desire; I know of at least one other team for which that would make a big difference (my old team, which is also making good use of Docker in a not-dissimilar way)!

[chriskrycho]

Related: notion-cli/rfcs#33.

[charlespierce]

FYI With the change to support extends in package.json (#755), this will be _sort of_ supported. While intended for workspaces / monorepos, Volta's logic for extends is flexible, so you could use it like this:

package.json:

{
  ...
  "volta": {
    "extends": "./volta.json"
  }
}

volta.json

{
  "volta": {
    "node": "14.0.0",
    "yarn": "1.22.0"
  }
}

So that changing your Node / npm / Yarn version wouldn't require making a change to package.json, it would keep the same contents. If you _also_ happened to be in a monorepo, you could then set extends in volta.json to point at the workspace root, and that would work as expected.

The main caveat with this is that with how volta pin works, you wouldn't be able to use that as a convenience method and would need to manually update the version in your settings file (volta.json in this example). Running volta pin would result in the version being written into package.json along side the extends directive.

[frangio]

I'm personally interested in this feature so that I can use Volta in my personal local environment without polluting the package.json of my open source projects. So the extends option doesn't work for me.

[charlespierce]

@frangio Understandable! While the approach I outlined is _a_ solution to this issue, I don't think it's _the_ solution since there are a number of problems with the usability.

Can you expand on your use case a little? Are you looking to have a completely separate file that you can add to .gitignore so that it isn't committed to the open source projects? Or are you looking to have that file completely outside of the project folder?

The former is potentially doable, though we'd need to make sure we're on top of performance and not trying to look in too many different places for files. The latter is difficult and I'm not sure really fits into the Volta data model.

Out of curiosity, what is the hesitation regarding adding volta to the package.json for the open source projects?

[frangio]

Are you looking to have a completely separate file that you can add to .gitignore so that it isn't committed to the open source projects?

Yes exactly. The performance concern is very interesting though. Maybe there could be a limitation that this separate file has to be next to a package.json file. Sounds kind of arbitrary but it would limit the effect on performance.

The hesitation is because of a sort of "separation of concerns". I view Volta as a concern of my local setup, and as much as I love it and would recommend it to everyone, other developers may choose nvm (or anything else) and it wouldn't make sense to have duplicate configuration for all of these tools. I would've thought they'd all use the engines field, but I saw there are reasons for Volta not to use that (which I haven't read).

[charlespierce]

Related proposal from Twitter: Support the .nvmrc file directly so that users of Volta and NVM can both work on the same code without changes. There's a few issues we'll need to tackle / design in order to make that work:

• Where to look for the .nvmrc? My preference here would be as a sibling of package.json iff that file exists _and_ there isn't a volta key already. We want to minimize the file I/O operations on the hot path, to make sure that our startup time is quick.
• How to map the values supported by .nvmrc into Volta's VersionSpec type? It appears that nvm supports a number of different keywords that don't necessarily match up with Volta's.
• How to handle range versions? .nvmrc supports a range / semver specifier (e.g. 4.*), how should Volta treat that? This is the biggest hurdle that I can see, because Volta has a fundamentally different model from nvm for how to manage the local cache.
  
  
  
  • We don't want to fetch the Node index and re-resolve the version every time a Node command is run, since that will add unnecessary overhead to every command.
  
  
  • At the same time, if we only check the local cache, we go against Volta's philosophy of auto-downloading the necessary versions on-demand and require the user to manually manage their "installed" versions.
  
  
  • Even a hybrid model requires the user to be aware of the internal cache, which they shouldn't need to manage directly.
  
  
  • Perhaps we only support pure versions in that file, and show a warning if a file _would_ be used but is being skipped because we don't have a specific version.
  
  

If we can work out the specifics of how to handle the .nvmrc file, the existing extends logic already supports multiple sources of truth, so it should be possible to slot .nvmrc into that logic. Similarly, the existing extends work also should support a new, separate file (perhaps .voltarc.json?) that can be loaded.

[ljharb]

Hi - nvm maintainer here.

The way nvm looks for an .nvmrc file is just like npm looks for .npmrc or eslint looks for .eslintrc - in the current directory, or in any ancestor, all the way up to /. Any other algorithm would be incorrect.

.nvmrc contains a "version-ish". This is not actually a range - it always resolves to a single version, either locally, or remotely, depending on the context/command. Asterisks are not supported whatsoever. For example, 4 means "the latest version of 4", or in semver terms, "^4.0.0". "node" means "the latest version of node", or in semver terms, "*".

I beg you not to support .nvmrc at all unless you can support 100% of its semantics, which include alias lookup, release line lookup, LTS lookup (lts/*, lts/argon, etc), major (4), minor (4.2), or exact (4.2.1).

[charlespierce]

Hi @ljharb, thanks for the clarification on the specifics of .nvmrc! I agree we shouldn't try to support .nvmrc unless we follow the same semantics, as that's just going to create undue confusion all around. And given that, ultimately I think the non-exact version-ish resolution is going to make supporting .nvmrc untenable with Volta's current model.

The difference as I understand it (and please correct me if I'm wrong) is that nvm has an explicit command (nvm use) to say "Resolve the version now and set that up" after which it stays fixed (at least until the command is run again). By contrast, Volta is always re-evaluating the context, so we would need to constantly redo that resolution. Which would ultimately cause more problem than it solves, imo.

[ljharb]

The difference as I understand it

That's correct, yes - nvm use always uses the locally installed version list, and thus, shouldn't change unless someone has ran nvm install or nvm uninstall, or changed an alias. nvm install also reads from .nvmrc to install the desired version, and reads from the remote list.

[pho3nixf1re]

There is also the .node-version file which is supposed to be more generic between projects but the semantics are the same as nvmrc I believe. As an alternative, could Volta provide a command to migrate an nvmrc file over to the volta config automatically? This would be helpful in the case of migrating lots of projects at once or even using volta beside nvm for larger teams.

[bmblb]

Pinning to package.json pollutes the repository with irrelevant info. I'd love to have smth like .voltarc config somewhere in the directory to pin versions there. I wouldn't expect it to be compliant with any other tool, like nvm, just own config. @charlespierce would you consider that?

[ljharb]

@pho3nixf1re the semantics aren’t the same, for one because there are no agreed-upon semantics, but also because nvmrc supports anything nvm considers a “version-ish”, which includes user aliases, “iojs”, LTS aliases, etc.

[pho3nixf1re]

Sorry, let me clarify. I mean "the same" as in .nvmrc == .node-version. As for Volta my suggestion was to provide a command that would resolve the nvmrc once and then run the pin command based on that. It would be a one time thing to be used as a migration to Volta or to provide a temporary path for a team that has already established nvm.

[alexilyaev]

A separate config file support would be most appreciated.
Some projects already are using .nvmrc or .node-version and don't want to add another configuration to package.json.
Ideally in these projects I could add a .voltarc file (git ignored) next to the package.json or in a parent folder and that would be used in the absence of volta settings in package.json.

The use cases are also mentioned in https://github.com/volta-cli/volta/issues/924.

I understand the performance concern, so maybe it can be an opt-in feature.
If it's opt-in, it could be even faster if we don't need to parse package.json first.

Side note:
I found Volta while reading The State of JS 2020 😄
https://2020.stateofjs.com/en-US/technologies/build-tools/

[Mischala]

I would love to be able to configure Volta for my repos on my own machine, and pin in a file i don't have to check in.

[matthewadams]

Volta is well on its way to be exactly what I'm looking for. The one feature that I'm missing is the .voltarc, searched for from the current directory on up to the root.

I _really_ like the features offered by asdf, particularly https://asdf-vm.com/#/core-manage-versions?id=set-current-version, where you can set the version at many levels: a global default version, a local version (directory-based .voltarc, as has been discussed here), a shell version (via an environment variable like VOLTA_${TOOL}_VERSION), or even a version for an individual command ($ VOLTA_NODE_VERSION=12.15.0 npm run foobar).

If volta could support these same features, it'd be just jim-dandy.

[shadowspawn]

For possible interest, collection of .node-version usage: https://github.com/shadowspawn/node-version-usage