Conda-forge.github.io: JavaScript Packing Issues and Best Practices

[scopatz]

Hello all, I currently see a few issues surrounding the creation and maintainence of JavaScript packages:

1. The current way of building JS packages (npm install -g .) no longer works with conda build v3.
   
   
   
   • For example, https://github.com/conda-forge/jstz-feedstock/pull/1 produces the following: Error: lib/node_modules/jstz is a symlink to a path that may not exist after the build is completed (../../../work/jstz-2.0.0)
   
   
   • This is apparently because of changes to npm (CC @isuruf @minrk)
   
   
   • The fix is npm pack; npm install -g pkg-version.tgz
   
   
2. Because of (1), all PRs the bot is making against JS packages are broken.
3. The maintainers of most JS packages (@sannykr @cshaley) haven't been responsive to the bots PRs or gitter (but it hasn't been that long on gitter).
4. npm seems to install a copy of all dependencies for each package that is installed. Is there any way around this so that we can have real dependency chains on the graph in conda?
5. Related to (4), do we need to use npm to install JS packages at all? If all npm install is doing is shoving some files into a known directory and adding some symlinks, it seems like we could do that with conda-build as well. It is not as though there would be any penalty or collision with npm either, since npm would copy the package again anyway.

Anyway, some feedback, guidance, and discussion would be most welcome! CC @conda-forge/core

[minrk]

If the original tarball were available as an environment variable rather than unpacking it, this would be easy, because it would be:

npm install -g $SRC_ARCHIVE

Basically, conda-build unpacking the archive isn't helpful because we have to pack it back up for npm to install it. A handle on the original tarball would allow us to ignore (or ideally skip) the unpacked directory, rather than manually undo the step.

npm seems to install a copy of all dependencies for each package that is installed. Is there any way around this so that we can have real dependency chains on the graph in conda?

It's technically possible (debian tries to do this), but I would strongly discourage any attempt to do this. It's antithetical to how node and npm work, and making conda-installed js packages not behave like npm-installed js packages seems undesirable to me. This is also part of why I don't think packaging node libraries with conda makes much sense. Only repackaging applications makes sense to me, because libraries installed globally aren't importable (dependencies are always local) unless you are running an unusual, customized environment. One of the things that's critical for npm is that you must be able to have multiple versions of one package in a single env, since this is a common occurrence due to dependencies always being local, and strict semantic-versioning being standard practice. I'm not sure conda can or should handle that, and by leaving it to npm, it doesn't have to.

do we need to use npm to install JS packages at all? If all npm install is doing is shoving some files into a known directory and adding some symlinks...

That's not all it's doing. Like pip+distutils, it's a dependency solver, build system, arbitrary code execution, config files, etc.

[scopatz]

If the original tarball were available as an environment variable

Yeah that would be nice

It's technically possible (debian tries to do this), but I would strongly discourage any attempt to do this. It's antithetical to how node and npm work,

Ok, good to know. We should avoid it then.

Like pip+distutils, it's a dependency solver, build system, arbitrary code execution, config files, etc.

Yeah, so there is no reason to try to duplicate that effort.

I don't think packaging node libraries with conda makes much sense. Only repackaging applications makes sense to me, because libraries installed globally aren't importable (dependencies are always local)

This makes sense as a general rule to me as a conda-forge developer. As a conda-forge user it is more frustrating, because now to be able to develop an application I have to be using two package managers: conda and npm. It seems like npm's model forces us into this mode where we have to use npm.

More precisely, I would like to have an environment.yml file that specifies all of the dependencies for a project so that anyone could execute conda env create -f environment.yml and be up and running. It seems like this may not be possible with JS packages.

[cshaley]

More precisely, I would like to have an environment.yml file that specifies all of the dependencies for a project so that anyone could execute conda env create -f environment.yml and be up and running.

Yes! This is exactly why we have been adding JS packages to conda forge.

I can start filtering through js packages and updating them with npm pack; npm install -g pkg-version.tgz if that seems like the right path forward. Otherwise, it may make sense to update the PR bot to fix these on its own.

[CJ-Wright]

I can code up the bot to run the migration, once we can decide:

1. how to filter for recipes which need the migration
2. what exactly to do for the migration (change x to y in the meta.yaml)

[scopatz]

@CJ-Wright:

1. I think the filter is to look for build: noarch: generic and then for build: script: npm install -g .
2. the migration seems to be to change the build script to be:

build:
  script: |
    tgz=$(npm pack)
    npm install -g $tgz

[CJ-Wright]

Attn: @justcalamari

[CJ-Wright]

xref: https://github.com/regro/cf-scripts/issues/169

[CJ-Wright]

xref: https://github.com/regro/cf-scripts/pull/171

[CJ-Wright]

@scopatz the bot has issued 38 PRs updating to the new JS syntax.
Edit: (as clarification, that is all the filters picked up)

[scopatz]

Awesome!

[minrk]

Very cool! Thanks, everyone!

[jakirkham]

As part of this process, discovered electron is being built as noarch, which seems wrong to me as IIUC it compiles code. Can someone please check on this?

xref: https://github.com/conda-forge/electron-feedstock/issues/9